file permissions

[gedankenstuecke]

https://opensnp.org/data/zip/opensnp_datadump.current.zip right now gives a 403, seems to be something off with the default permissions when created.

[gedankenstuecke]

okay, it's not a permission issue in itself. the file itself just doesn't exist :open_mouth:

[tsujigiri]

This might have something to do with it.

[gedankenstuecke]

I tried to investigate it a bit further. So the zipfulldata job dies somewhere in the zip_genotype_file thing (https://github.com/gedankenstuecke/snpr/blob/master/app/workers/zipfulldata.rb#L50), as the zip file itself is created and it contains some of the genotyping files, but the overall archive is broken and the symlink isn't redirected. I'm a bit out of ideas on why this is the case right now.

[philippbayer]

Could be a bug in rubyzip, this person has the same problem: https://www.ruby-forum.com/topic/211146

[gedankenstuecke]

We also had this problem earlier: https://github.com/rubyzip/rubyzip/issues/82 (just ctrl-f for your own name ;-)).

On 14 April 2015 at 13:28, Philipp Bayer notifications@github.com wrote:

Could be a bug in rubyzip, this person has the same problem: https://www.ruby-forum.com/topic/211146

— Reply to this email directly or view it on GitHub https://github.com/gedankenstuecke/snpr/issues/157#issuecomment-92775926 .

[philippbayer]

haha yeah :D I totally forgot about it. back then it was an issue with file size, there was a "maximum" for rubyzip that later versions upped/removed, maybe we hit the same thing again?

[philippbayer]

It could also of course be that we have some dangling phenotype/genotyping/picture_phenotype/etc. with some Nil as an attribute somewhere in the db

[gedankenstuecke]

That might be, but seems unlikely to me as the problem did arise pretty much in time with the rails update :D

[tsujigiri]

We could try using the regular release of rubyzip again. We pinned it to master. This could also be broken because I updated all the gems and hence, this one was also updated to the latest master commit.

[tsujigiri]

See #160

[philippbayer]

Let's deploy and have a look

[gedankenstuecke]

Playing with :fire: :D

[philippbayer]

Deployed - did it burn down yet? Do you have a testcase?

[tsujigiri]

We can mark it resolved in Errbit and see if it comes back.

[philippbayer]

btw, pry is still being weird on production: (as snpr)

>bundle exec rake dump:full                                                                                                              deploy 
rake aborted!
LoadError: cannot load such file -- pry
/srv/www/snpr/.rvm/gems/ruby-2.2.1-railsexpress@global/gems/bundler-1.8.4/lib/bundler/runtime.rb:76:in `require'

even though it's not in production env

[gedankenstuecke]

Run the zipfulldata-job once. :-)

On 14.04.2015, at 18:13, Philipp Bayer notifications@github.com wrote:

Deployed - did it burn down yet? Do you have a testcase?

— Reply to this email directly or view it on GitHub.

[tsujigiri]

RAILS_ENV=production?

[philippbayer]

Herpderp that fixed it. zipfulldata running right now

[gedankenstuecke]

It will take some time before you can tell. It crashed in the last steps. ;)

On 14.04.2015, at 18:24, Philipp Bayer notifications@github.com wrote:

Herpderp that fixed it. zipfulldata running right now

— Reply to this email directly or view it on GitHub.

[tsujigiri]

You also could have waited for the tests to pass... ;) https://travis-ci.org/tsujigiri/snpr/builds/58463832

[tsujigiri]

Restarted it, now it's green.

[tsujigiri]

The error is still there.

[gedankenstuecke]

Okay, maybe it really is a single file that causes the crash?

[philippbayer]

I've added a bunch of logging to the job to pinpoint a bit more which of the methods actually crashes, deployed, let's see what happens

[gedankenstuecke]

Awesome. Will be in Paris until Sunday afternoon for the Sage Assembly, with limited access to the internet. Will look into once I'm back in my office.

On 16.04.2015, at 17:07, Philipp Bayer notifications@github.com wrote:

I've added a bunch of logging to the job to pinpoint a bit more which of the methods actually crashes, deployed, let's see what happens

— Reply to this email directly or view it on GitHub.

[philippbayer]

It's a mystery:

Zip::File.open(zip_fs_path, Zip::File::CREATE) do |zipfile|
   #lots of stuff
   zip_genotype_files(genotypes, zipfile)
   log "6/6 Finished zipping genotype file"
end
log "Chmodding public zip"

The last line in the logs before the job crashes is "6/6 Finished zipping genotype file" - even though there's "just" an end there, "Chmodding public zip" does not appear.

That makes me think that this is not an issue with our data but with rubyzip.

[gedankenstuecke]

Or it really is a problem with the files we're trying to zip? What happens when you try to manually create the zip file?

On 16.04.2015, at 22:35, Philipp Bayer notifications@github.com wrote:

It's a mystery:

Zip::File.open(zip_fs_path, Zip::File::CREATE) do |zipfile|

lots of stuff

zip_genotype_files(genotypes, zipfile) log "6/6 Finished zipping genotype file" end log "Chmodding public zip" The last line in the logs before the job crashes is "6/6 Finished zipping genotype file" - even though there's "just" an end there, "Chmodding public zip" does not appear.

That makes me think that this is not an issue with our data but with rubyzip.

— Reply to this email directly or view it on GitHub.

[philippbayer]

I think I fixed it! But it's one of those fixes that make you wonder why it worked in the first place... rubyzip does not use zip64 by default since not all unzippers can handle zip64. So by default, it uses "normal" zip, which has a size limit of 4GB. For some reason in the past it seems to have used zip64 but upgrading everything seems to have "reset" it to standard zip.

I've ran the zipping job manually in the console with zip64 and unicode enabled and that worked, I've added a new file to config/initializers so hopefully in about 3-4 hours we'll know whether it's fixed completely

[gedankenstuecke]

Great. I'll keep my fingers crossed that it works out.

[philippbayer]

:neckbeard: https://opensnp.org/data/zip/opensnp_datadump.current.zip

[tsujigiri]

:clap:

[gedankenstuecke]

:+1: Very nice! I will answer all the support questions about this then. :-)