search

plazi / treatmentBank

Repository devoted to house keeping of treatmentBank
0 stars 0 forks source link

upload of megataxa.8.1.1 to frankfurt fails #75

Closed myrmoteras closed 1 year ago

myrmoteras commented 1 year ago

@gsautter can you please have a look at this file and why it does not upload? the file is here - may be it id due to the size (900MB)?

$ for file in *.pdf; do curl -H "Authorization:How can there be a copyright on stating that the sky is blue?" -H "Meta-Data-Mode:Go use your templates!" -F "file=@$file; filename=$file" -F "user=plazi" -F "mimeType=application/pdf" -X PUT https://tb.plazi.org/GgServer/docUpload; done; # file <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

502 Bad Gateway

Bad Gateway

The proxy server received an invalid response from an upstream server.

gsautter commented 1 year ago

Not sure what might have caused the error, but file size might have played a role with this behemoth ... anyway, downloaded it to the server now, and triggered the import via local file intake ... the PDF is decoding as I type this.

gsautter commented 1 year ago

Looks like decoding proper went well, but when the decoder stored the resulting IMF on the RAM disk for the main server to pick it up, it ran out of disk space (on that specific 8GB RAM disk, which is otherwise empty, as we use it solely for disk based file handover between components in the main server and their sub processes, i.e., PDF decoding and server side batch) ... need to increase the size of the RAM disk and try again ...

gsautter commented 1 year ago

After increasing the RAM disk to 16GB, it went through ... batch seems to have done a decent job: https://tb.plazi.org/GgServer/summary/FF97FFB6BD52FF93FFA63955FFF1FF82 The reason this PDF is so large, and also for the RAM disk space problem, seems to be a good number of figures with crazy high resolution (600 DPI and above), as well as the sheer number of figures (almost 300) ... the IMF is 6.5GB, and that for a mere 292 pages.

myrmoteras commented 1 year ago

thanks . looks good