search

geobeyond / Arpav-PPCV-backend

Backend di Piattaforma Proiezioni Climatiche per il Nord-Est.
Creative Commons Attribution 4.0 International
0 stars 1 forks source link

parallelize calls to the THREDDS server #163

Open ricardogsilva opened 6 days ago

ricardogsilva commented 6 days ago

The time series endpoint performs a series of requests to the THREDDS server - let's parallelize these in order to improve performance of the system

ricardogsilva commented 3 days ago

I refactored the code that calls the THREDDS NCSS service to use an async pattern and did not see any noticeable gains in performance (These changes will be submitted as part of #159). This ruled out network IO as being the bottleneck in the time series generation process.

I was then able to trace the degraded performance to the code that applies the LOESS algorithm. As an example, I found out that a certain request was taking ~3.4s to complete, with ~3s being spent on the LOESS operation while the other ~0.4s was the time it took to fetch all raw data from THREDDS NCSS (8 datasets) and applying all of the other smoothing operations.

It seems the LOESS code is not suitable for the usage pattern that we had envisaged whithin the system.