alliander-opensource / weather-provider-api

This API is intended to help you fetch weather data from different data sources in an efficient and uniform way. By just supplying a list of locations and a time window you can get data for a specific source immediately. This project is licensed under the MPL-2.0 license.
https://alliander-opensource.github.io/weather-provider-api/
Mozilla Public License 2.0
27 stars 6 forks source link

WPLA 2.x\\ BUG - Memory usage issues #119

Open rflinnenbank opened 11 months ago

rflinnenbank commented 11 months ago

Some users of (at least) the Gunicorn image encounter high memory usage when dealing with the AROME updating process and while requesting AROME data. This results in the update process requiring up to and at times over twelve gigabytes of memory, and the request parsing to even run out of memory even when only requesting a few megabytes.

Steps to reproduce the behavior:

  1. Use Gunicorn image based on version 2.47.031.
  2. Adjust lock-file to match that of the reporting party to duplicate exact container settings.
  3. Verify occurrence of memory usage problems

Expected behavior When updating AROME data, the maximum memory usage should barely go above the size of the files being processed (400-500Mb for now) . The same goes for regular requests, which shouldn't generate any kind of issue under several gigabytes of data processing.

Additional context This issue is a strange one, because even in the image we install via poetry with a .lock file in the folder. This should normally mean that there aren't that many possible differences between different installations of the image. This makes one wonder even more about the reason for such huge differences in memory usage.