Unknown Worker Error

[aforsythe]

Was trying to process a few cameras locally using docker when I ran into the following error. Memory doesn't seem to be a problem, though CPU spikes to 100%. Full resources of the Macbook Pro are available to the docker daemon, so not sure that's the issue.

web-1  | /usr/local/lib/python3.12/site-packages/colour/utilities/verbose.py:265: ColourUsageWarning:
web-1  |
web-1  | "OpenImageIO" related API features are not available, switching to "Imageio"!
web-1  |
web-1  | [2024-04-22 15:38:05 +0000] [8] [CRITICAL] WORKER TIMEOUT (pid:9)
web-1  | [2024-04-22 15:38:05 +0000] [9] [INFO] Worker exiting (pid: 9)
web-1  | ===============================================================================
web-1  | *                                                                             *
web-1  | *   Interpreter :                                                             *
web-1  | *       python : 3.12.3 (main, Apr 10 2024, 14:35:18) [GCC 12.2.0]            *
web-1  | *                                                                             *
web-1  | *   colour-science.org :                                                      *
web-1  | *       colour : 0.4.4                                                        *
web-1  | *       colour-checker-detection : 0.2.0                                      *
web-1  | *       colour-datasets : 0.2.5                                               *
web-1  | *                                                                             *
web-1  | *   Runtime :                                                                 *
web-1  | *       imageio : 2.33.1                                                      *
web-1  | *       matplotlib : 3.8.2                                                    *
web-1  | *       networkx : 2.8.8                                                      *
web-1  | *       numpy : 1.26.3                                                        *
web-1  | *       scipy : 1.12.0                                                        *
web-1  | *       tqdm : 4.66.1                                                         *
web-1  | *       opencv : 4.9.0                                                        *
web-1  | *                                                                             *
web-1  | ===============================================================================
web-1  | [2024-04-22 15:38:05 +0000] [8] [ERROR] Worker (pid:9) exited with code 1
web-1  | [2024-04-22 15:38:05 +0000] [8] [ERROR] Worker (pid:9) exited with code 1.
web-1  | [2024-04-22 15:38:05 +0000] [59] [INFO] Booting worker with pid: 59

[KelSolaar]

How big is the Zip archive out of curiosity, I tested locally with archives ~11Gb max. It would interesting to see memory usage by using the API directly here: https://github.com/ampas/idt-calculator/blob/6a36c81184a9f8822b1b6be5f6e58260526d7513/aces/idt/prosumer_camera.py#L1629

[aforsythe]

All zips are ~450 MB

[KelSolaar]

Oh that should be very much fine!

Thomas Mansencal - colour-science.org https://www.colour-science.org - thomasmansencal.com http://www.thomasmansencal.com

On Wed, 24 Apr 2024 at 8:40 AM, Alexander Forsythe @.***> wrote:

All zips are ~450 MB

— Reply to this email directly, view it on GitHub https://github.com/ampas/idt-calculator/issues/81#issuecomment-2073407410, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAYLQ64MLTTAXELC6ZFOIDY63BMZAVCNFSM6AAAAABGTBNOHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZTGQYDONBRGA . You are receiving this because you commented.Message ID: @.***>

[KelSolaar]

I did a profiling pass with some images, i.e., x32 HD1080, Josh sent over and I don't see anything that would look like memory is consumed excessively:

If everything was loaded in memory at once, I would expect something along the lines of 8 bytes x 1920 x 1080 x 32 ~= 530mb but we are at 340m all inclusive!

Would it be possible to share the archive?

[aforsythe]

The archives are from Josh and may be related to production. It would probably be better if you grab them directly from him.

[KelSolaar]

Right, so this looks like the ones I processed for him over the weekend, I haven't tried with docker, let me give it a crack.

[KelSolaar]

I got it to run by increasing the gunicorn timeout, I think that there is an issue with the files, they are very long to read and I noted that the other day:

[KelSolaar]

This was fixed in #84.