Handle request throttling when making a new schematisation

[leendertvanwolfswinkel]

I tried to make a new schematisation (when testing #406), but I get this error: 2022-11-07T15:47:05 CRITICAL 3Di Models and Simulations : Error: Request was throttled. Expected available in 6 seconds.

I tried twice, but both times I got the same error.

I have asked our backend devs if this could be a backend issue. But if it occurs, would it be possible to wait the stated number of seconds and try again? Perhaps even generally for all requests?

[ldebek]

@leendertvanwolfswinkel It is possible, but it would be good to test and merge PR https://github.com/nens/threedi-api-qgis-client/pull/407 first.

[caspervdw]

I happened to open the QGis plugin from the console on linux and I noticed these logs starting as soon as I started a schematisation download:

INFO:threedi_models_and_simulations.api_calls.threedi_calls:Starting to grab sim statuses for 822 simulations
threedi_models_and_simulations.api_calls.threedi_calls INFO Starting to grab sim statuses for 822 simulations
threedi_models_and_simulations.api_calls.threedi_calls INFO Starting to grab sim statuses for 822 simulations
DEBUG:threedi_models_and_simulations.api_calls.threedi_calls:Fetching status for simulation 102694
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102694
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102694
DEBUG:threedi_models_and_simulations.api_calls.threedi_calls:Fetching status for simulation 102692
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102692
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102692
DEBUG:threedi_models_and_simulations.api_calls.threedi_calls:Fetching status for simulation 102690
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102690
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102690
DEBUG:threedi_models_and_simulations.api_calls.threedi_calls:Fetching status for simulation 102688
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102688
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102688
DEBUG:threedi_models_and_simulations.api_calls.threedi_calls:Fetching status for simulation 102687
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102687
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102687
DEBUG:threedi_models_and_simulations.api_calls.threedi_calls:Fetching status for simulation 102685
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102685
threedi_models_and_simulations.api_calls.threedi_calls DEBUG Fetching status for simulation 102685

My guess is that keeping track of 822 simulation statuses will definitely run into throttling problems. Maybe this is a direction to look into @ldebek

[ldebek]

@caspervdw Main issue here, is that after logging in I need to get a list of all currently running simulations. Currently this is not possible to establish within a single call (at least as far as I know). If fetched simulations could have their statuses available without need of any additional calls that would solve the problem. Another solution would be to have a parameter in the /v3/simulations/ endpoint that would allow to filter by current status name.

[leendertvanwolfswinkel]

@jpprins1 please share your opinion / advice on this

[jpprins1]

Two options:

1) https://api.3di.live/v3/statuses/?name=initialized 2) wss://api.3di.live/v3/active-simulations/ starts with given the list of all active sims on connection: like so: {"type": "active-simulations", "data": {}}

And give updates of incoming new simulations like so: {"type": "active-simulation", "data": {"110398": "{\"uid\": 110398, \"name\": \"parramatta operational forecast run\", \"date_created\": \"2023-01-05T08:29:13.315375+00:00\", \"user_name\": \"parramatta.service\", \"organisation_name\": \"City of Parramatta\", \"simulation_slug\": \"parramatta-operational-forecast-run-c1c4cb86-9f2f-4ca1-b73b-22610e9cc20f\", \"progress\": 0.0, \"status\": \"initialized\", \"duration\": 52200}"}}

Progress update: {"type": "progress", "data": {"simulation_id": 110398, "progress": 1}}

Status update: {"type": "status", "data": {"status": "finished", "simulation_id": 110398}}

[leendertvanwolfswinkel]

@ldebek Everything still works, but getting a list of running simulations and getting a list of results available for download is still as slow as it used to be (for Nelen & Schuurmans Consultancy)

I have not been able to check if this reduces the risk of throttling, as this is super hard to reproduce.

I suggest to merge the changes, and have a look at optimizing the two points I mentioned above later

[ldebek]

@leendertvanwolfswinkel PR with refactored code for fetching and tracking running simulations and results listing is available for testing here: https://github.com/nens/threedi-api-qgis-client/pull/458 There is one regression with proposed approach, which is a missing simulation user name in the simulation results window, as this information is not present within SimulationStatus object. If it's crucial I can add it by quering listed finished simulations, but I think it would be much more efficient to just add that information to SimulationStatus class in the threedi-api-client itself.