search

euroargodev / BlueCloud

Working space for BlueCloud demontrator 3 - Lops Task
0 stars 1 forks source link

Memory limit #1

Open AndreaGarciaJuan opened 4 years ago

AndreaGarciaJuan commented 4 years ago

We try to use a list of all argo floats deployed in mediterranean sea from 2010 to 2020 but there is not enough RAM memory to load all data using argopy (even in Datarmor with a job of 60GB)

gmaze commented 4 years ago

hi @AndreaGarciaJuan How many floats/profiles that is ?

I'm working on a parallel version of the fetcher that should be able to address this.

Can you try to fetch these data with an argopy installed from the "parallel-requests" branch ?

pip install git+http://github.com/euroargodev/argopy.git@parallel-requests

then you can create a loader like:

loader = ArgoDataFetcher(src='erddap', parallel=True, chunks={'wmo': 2}).float(random_wmo)
loader.to_xarray()
AndreaGarciaJuan commented 4 years ago

I pushed 2 lists: -MedFloats_2010-2020.csv: 316 floats, a lot of profiles (not working) -MedFloats_2017-2020.csv: 107 floats, 13838 profiles (working now in the notebook, in datarmor) I am trying the parallel-requests now

gmaze commented 4 years ago

Note that the "parallel-requests" branch will only work with the "erddap", so can't use on datarmor.

gmaze commented 4 years ago

Can you send the "MedFloats_2010-2020.csv" file ?