search

bird-house / birdy

Birdy provides a command-line tool to work with Web Processing Services.
http://birdy.readthedocs.io/en/latest/
Apache License 2.0
8 stars 3 forks source link

Emulate sync mode in client, while always sending async requests to server #207

Open huard opened 3 years ago

huard commented 3 years ago

Description

Due to some weird interactions between dask and PyWPS multiprocessing logic, there is a lock happening when PyWPS runs a process in async mode after having run a process in sync mode. One option is to always run in async mode server side, but have the client "wait" for the server to respond, like in a sync call.

At the moment, the execution mode is set using the progress argument, which is not intuitive and mixes two concepts, the progress bar display and the execution mode. I believe we should review our user interface for this, to allow in particular the possibility to send async requests but wait for the response in the client.

huard commented 3 years ago

@cehbrecht @cjauvin

cjauvin commented 3 years ago

The problem I see with this fix is that it is making a big change to the interface in order to essentially overcome a problem with only a very specific component: writing a NetCDF file in parallel with Dask + multiprocessing.

cjauvin commented 3 years ago

The ideal would be to understand the problem and find a way to mitigate it at the core, but nobody seems to have a clue (I asked in many places/contexts).

huard commented 3 years ago

We can keep the current interface as is, and just add new thin interfaces without disrupting existing code.