Closed Dolgalad closed 2 years ago
Reading this issue makes me think about introducing a max_request_duration
parameter that could be 1 day by default. Because the best would be to have a maximum data size for requests but this is really hard to evaluate. Letting the user override this value allows to increase it for really slow datasets where a several days request would be OK and likely more efficient.
In AMDA, when a "getParameter" request time is greater than 4 minutes, the execution enter in a batch mode.
In this case, the result will look like:
{
"success": true,
"status": "in progress",
"id": "process_ucuGXR_1650348560_252656"
}
In this condition, getStatus API can be used to retrieve the status. This API can be called until the request is complete. And when it's done, the result should be:
{
"success": true,
"status": "done",
"dataFileURLs": "http://amda.irap.omp.eu/AMDA//data/WSRESULT/getparameter_mms1_dce_qual_brst_35b6786739efcdc5a74ab1dca29d3b6b_20210101T000000_20210102T000000.txt"
}
It seems that Speasy does not implement this scenario.
For information, our AMDA backend is behind a proxy with a timeout defined as 5 minutes. This is why we need to enter in a "batch mode" when the execution of a request is "too long".
@brenard-irap we can use this from REST API now?
@jeandet Yes
Ok, I propose to work on that during next week workshop.
Keep in mind that when timeout is reached the "batch mode" task is created on the server. This means that if a user interrupts speasy while its getting data, the task will keep running on the server, this is why I don't like the timeout solution. Splitting the time range into intervals means that if the user interrupts the process only a single block of data will be requested from the server. Splitting the data also provides a natural way of notifying the user of the progress of the request (functionality I find useful when dealing with long time periods).
Another simple way of dealing with this problem is to raise an Exception if a timeout is reached. The value of the timeout needs to be smaller than the 4 minutes used by AMDA.
PR #41
Description
AMDA creates background jobs to deal with requests that take too long to answer (timeout exceeded). Speasy is not notified of this fact, thus when trying to retrieve a large dataset the
get_data
method may never return, it will wait indefinitely.What I Did
Solution
Modified the
dl_parameter
function inspeasy.webservices.amda._impl
module :