search

clstoulouse / motu-client-python

Motu is a high efficient and robust Web Server which fills the gap between heterogeneous data providers to end users. Motu handles, extracts and transforms oceanographic huge volumes of data without performance collapse. This client enables to extract and download data through a python command line.
GNU Lesser General Public License v3.0
49 stars 22 forks source link

Parallel motu_api requests saving incorrect data #30

Open vbalza opened 2 years ago

vbalza commented 2 years ago

Hello! Following the guidance in this article, we implemented a Python script using the motuclient library to retrieve CMEMS data via the subsetter.

Because the data size of the download is limited, we break up a larger temporal request (e.g., 2022-08-01 through 2022-09-04) into smaller, seven-day requests. The script attempts to submit parallel requests but we are running into an error in which the data saved for a given request start and end date (e.g., 2022-08-29 to 2022-09-04) via the motu_api.execute_request() method is actually the data of another request running in parallel (e.g., 2022-08-01 to 2022-08-07).

Do you have any guidance on how to address this issue?

We have attached a Jupyter notebook illustrating the error. Thanks in advance for any help you can provide! download_cmems.pdf

clstoulouse commented 2 years ago

Hi @vbalza, As far as I understand your issue, you try to run serveral requests in parralel and motu python script seems to not isolate the dates of all your requests. We have never tried this use case, so either you can try to fix the source code as it is open source ;-) or another way could be to run Motu requests using exec method to isolate each request in a separate python process.

Regards, Motu Team