search

openclimatefix / Satip

Satip contains the code necessary for retrieving, transforming and storing EUMETSAT data
https://satip.readthedocs.io/
MIT License
41 stars 28 forks source link

Memory management #218

Open PaulaPerezL opened 8 months ago

PaulaPerezL commented 8 months ago

I have some troubles managing memory when reading and saving images. How do you handle it? Do you store each image in its own Zarr, or do you stack them up into one big Zarr?

Thank you very much

peterdudfield commented 8 months ago

We do a bit of both, for our production service we store each image as its own zarr, and then we join then together to make a joint zarr. We seperate non hrv and hrv out.

We run the service on AWS using these memory limits - https://github.com/openclimatefix/ocf-infrastructure/blob/main/terraform/modules/services/sat/ecs.tf#L4

What memory limits did you have?

@jacobbieker should we put some of this in the readme? We could add it to the https://github.com/openclimatefix/Satip?tab=readme-ov-file#running-in-production section.

peterdudfield commented 8 months ago

Does this make sense @PaulaPerezL ?

PaulaPerezL commented 8 months ago

Yes, thank you, it makes sense. What happened to us is that the command "del object" is not working, or at least it does not release the RAM consumed by that object. However, we devised a method to launch each image in an independent process, and by doing so, we are able to clear the RAM after reading the image.