bilelomrani1
commented
3 years ago Is the downloading of just 4 files and then trying to covert to L3 normal behaviour? Or should it first download all the files first before converting to L3?
It is not a normal behaviour. As of right now, all L2 should be downloaded before starting the resampling. Looking at the traces, it looks like the script proceeds to access L2 files that failed to download. Can you post your query so that I can try to reproduce your issue?
3zero2
My query requires a download of 160Gb and 1556 products. After downloading the first 4 files, then it tries to covert to L3, during which it crashes with the following error:
` Traceback (most recent call last):
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info
file = self._cache[self._key]
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/lru_cache.py", line 53, in getitem
value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/wrfchem/Desktop/s5p-tools/L2_data/L2__AER_AI/S5P_NRTI_L2__AER_AI_20210703T152705_20210703T153205_19284_01_010400_20210703T160645.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
for filename in L2_files_urls
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/api.py", line 500, in opendataset
**kwargs,
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/netCDF4.py", line 558, in opendataset
autoclose=autoclose,
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/netCDF4.py", line 380, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 328, in init
self.format = self.ds.datamodel
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/netCDF4.py", line 389, in ds
return self.acquire()
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/netCDF4.py", line 383, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/contextlib.py", line 112, in enter
return next(self.gen)
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
File "netCDF4/_netCDF4.pyx", line 2291, in netCDF4._netCDF4.Dataset.init
File "netCDF4/_netCDF4.pyx", line 1855, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'/home/wrfchem/Desktop/s5p-tools/L2_data/L2__AER_AI/S5P_NRTI_L2__AER_AI_20210703T152705_20210703T153205_19284_01_010400_20210703T160645.nc'
`
File "s5p-request.py", line 411, in
num_workers=args.num_workers,
File "s5p-request.py", line 228, in main for filename in L2_files_urls File "s5p-request.py", line 228, in
Re-running the command then downloads another 4 files and then crashes again. Any idea what is wrong here please @bilelomrani1 ? I can't seem to be able to follow what's happening.
Is the downloading of just 4 files and then trying to covert to L3 normal behaviour? Or should it first download all the files first before converting to L3?
Thank you.