search

SkyTruth / cerulean-cloud

All cloud services including inference and database structure
Apache License 2.0
7 stars 1 forks source link

Error: Cloud Run out of memory #90

Open jonaraphael opened 1 year ago

jonaraphael commented 1 year ago

Memory limit of 5120 MiB exceeded with 5261 MiB used. Happens about 25 times / day with the orchestrator (@5GB). Happens about 10 times / day with the offset tiles (@5GB).

Can be mitigated by increasing memory value in the following locations. Note that if you increase it substantially, you will also have to increase the number of vCPUs according to https://cloud.google.com/run/docs/configuring/memory-limits

cerulean-cloud/stack/cloud_run_orchestrator.py cerulean-cloud/stack/cloud_run_offset_tile.py

jonaraphael commented 1 year ago

Reduced: Happens about 25 times / day with the orchestrator (@6GB). Happens about 0 times / day with the offset tiles (@6GB).

Examples of Scenes triggering the failure in the orchestrator: S1A_IW_GRDH_1SDV_20230809T080754_20230809T080823_049796_05FD0B_1519 S1A_IW_GRDH_1SDV_20230809T064557_20230809T064622_049795_05FD02_7AAF S1A_IW_GRDH_1SDV_20230808T234551_20230808T234620_049791_05FCD9_CEA1 S1A_IW_GRDH_1SDV_20230809T050408_20230809T050433_049794_05FCF7_3878 S1A_IW_GRDH_1SDV_20230809T050314_20230809T050343_049794_05FCF7_1027