Closed aforsythe closed 6 months ago
How big is the Zip archive out of curiosity, I tested locally with archives ~11Gb max. It would interesting to see memory usage by using the API directly here: https://github.com/ampas/idt-calculator/blob/6a36c81184a9f8822b1b6be5f6e58260526d7513/aces/idt/prosumer_camera.py#L1629
All zips are ~450 MB
Oh that should be very much fine!
Thomas Mansencal - colour-science.org https://www.colour-science.org - thomasmansencal.com http://www.thomasmansencal.com
On Wed, 24 Apr 2024 at 8:40 AM, Alexander Forsythe @.***> wrote:
All zips are ~450 MB
— Reply to this email directly, view it on GitHub https://github.com/ampas/idt-calculator/issues/81#issuecomment-2073407410, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAYLQ64MLTTAXELC6ZFOIDY63BMZAVCNFSM6AAAAABGTBNOHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZTGQYDONBRGA . You are receiving this because you commented.Message ID: @.***>
I did a profiling pass with some images, i.e., x32 HD1080, Josh sent over and I don't see anything that would look like memory is consumed excessively:
If everything was loaded in memory at once, I would expect something along the lines of 8 bytes x 1920 x 1080 x 32 ~= 530mb but we are at 340m all inclusive!
Would it be possible to share the archive?
The archives are from Josh and may be related to production. It would probably be better if you grab them directly from him.
Right, so this looks like the ones I processed for him over the weekend, I haven't tried with docker, let me give it a crack.
I got it to run by increasing the gunicorn
timeout, I think that there is an issue with the files, they are very long to read and I noted that the other day:
This was fixed in #84.
Was trying to process a few cameras locally using docker when I ran into the following error. Memory doesn't seem to be a problem, though CPU spikes to 100%. Full resources of the Macbook Pro are available to the docker daemon, so not sure that's the issue.