search

Sapphire-CM / Sapphire

MIT License
3 stars 0 forks source link

Zip Utils Support #518

Open PrangerStefan opened 3 years ago

PrangerStefan commented 3 years ago

We have received the following error message:


A Zip::DestinationFileExistsError occurred in background at 2021-03-20 16:56:41 +0100 :

  Destination '/tmp/g1-07-heplan/helist.xlsx' already exists
  /home/sapphire/sapphire/shared/bundle/ruby/2.6.0/gems/rubyzip-2.3.0/lib/zip/entry.rb:617:in `create_file'

downloading it and locally decompressing resulted in the following:

[stefan@ziperror] ❯ unzip g1-07-heplan.zip
Archive:  g1-07-heplan.zip
   skipping: g1-07-heplan/helist.xlsx  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/heplan.html  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/heuristics.pdf  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/logs/log-dk-mob.txt  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/logs/log-dk-pc.txt  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/logs/log-dp-mob.txt  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/logs/log-dp-pc.txt  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/logs/log-tl-mob.txt  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/logs/log-tl-pc.txt  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/logs/log-ts-mob.txt  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/logs/log-ts-pc.txt  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/presentation/heplan-slides.pdf  need PK compat. v6.3 (can do v4.6)
   skipping: g1-07-heplan/report.css  need PK compat. v6.3 (can do v4.6)
   creating: g1-07-heplan/
   creating: g1-07-heplan/logs/
   creating: g1-07-heplan/presentation/
PrangerStefan commented 3 years ago

https://unix.stackexchange.com/questions/183452/error-trying-to-unzip-file-need-pk-compat-v6-1-can-do-v4-6

PrangerStefan commented 3 years ago

Is this a simple case of enabling z7 support?

matthee commented 3 years ago

I'm not sure, if that's the error we are seeing in prod. When extracting the archive locally, you are using your local environment, unrelated to sapphire production.

The real culprit imho is Zip::DestinationFileExistsError, which happens when unextracting in the archive in SubmissionExtractionService. There might be two reasons for that:

  1. A broken ZIP file was uploaded and rubyzip chokes
  2. Another File really exists in that directory
    • This might be related to us using Dir.tmpdir as part of generating the extraction path. Dir.tmpdir returns the path to the tmp dir of the system (usually /tmp)
    • According to the documentation Dir.tmpdir does not ensure uniqueness, but simply returns the path.
    • Instead we should be using Dir.mktmpdir
    • As far as I can tell, we currently do not properly cleanup the previously created extraction dirs, could you please verify in production? (I'm expecting to see several gx-xxx directories in /tmp
    • We need to make sure to cleanup after ourselves, esp. when using Dir.mktmpdir.
matthee commented 3 years ago

I've just reread the documentation on Dir.mktmpdir. If we are passing in a block, the directory will get cleaned up once the block is completed. So no further cleanup actions would be required from our side.

PrangerStefan commented 3 years ago
  • As far as I can tell, we currently do not properly cleanup the previously created extraction dirs, could you please verify in production? (I'm expecting to see several gx-xxx directories in /tmp

Definitely the case, saw about 5 dirs in there yesterday.

(Regarding the error: I guess the SO post lead me on the wrong track. I read something about proprietary software and thought that this must be it...)

PrangerStefan commented 3 years ago

Note to self: Keith asked about adding support for different decompression methods.