Open dpfinch opened 2 years ago
First, for the units, it seems that in the conversion part after line 150 of get_satellite_data.py, the case of « v/v » is not considered to be converted in molec cm-2. I was in this case and got the error message « unable to converts o3 units of v/v… » Then, I converted in my test nc file, my O3 profile to ppb to see if it goes further. It does (see next question). Still on units, I wonder if the conversion ppb —> molec cm-2 is correct. I would expect to have something more complicated for the conversion such as vmr(dP/g)(Na/mair). What do you think?
Second, independently of the units issue, I continue using ppb units. I succeed to download CAMS outputs. But the script stops without doing anything. I tracked it is quitting the code at line 432 of the utils.py script. I don’t know what’s going wrong. Below the last lines written on my screen.
--> Converting ozone from units of ppb to molec/cm2 --> Converting prior from units of ppb to molec/cm2 --> Cannot find units for level. Assuming hPa. 2022-09-30 09:10:48,027 INFO Welcome to the CDS 2022-09-30 09:10:48,028 INFO Sending request to https://ads.atmosphere.copernicus.eu/api/v2/resources/cams-global-reanalysis-eac4 2022-09-30 09:10:48,103 INFO Request is queued 2022-09-30 09:10:49,133 INFO Request is running 2022-09-30 09:10:52,944 INFO Request is completed 2022-09-30 09:10:52,945 INFO Downloading https://download-0003-ads-clone.copernicus-climate.eu/cache-compute-0003/cache/data1/adaptor.mars.internal-1664521848.1761296-22594-7-b957cfe8-358c-46db-b2fa-8ccc22e6671f.grib to /DATA/SPECAT_2/gaelle/TOAR2/toar_sample/OUT/model_download/1664521847_CAMS_model.grib (755.9K) 2022-09-30 09:10:53,333 INFO Download rate 1.9M/s Warning: ecCodes 2.21.0 or higher is recommended. You are running version 2.18.0 Approximately 0% of profiles in this file complete. ** Quiting.
Using the CAMS model
Retreving satellite file list
--> Found 11445 satellite files
--> Satellite records start from 2019-09-01.
Changing start date to match
--> Sampling model at satellite observations coordinates.
Sampling file: /home/users/blatter/Data/projects/ral_ozone_scheme/omi_aura/v003/omi_aura-solopt7-rat1-cclev2-saap-lip-samp1a2-samp2b2-af0-cmacc-pcbira-ecm-ec5-sfg2s-rsfw3-ng37-pol4-sc10-noxfr-ch2o-nosw-retpoly-s5xs-sckxs-ring4-smcl2-mplap-g2lk-nomr-cci-newfm2b-b2leak2-dxs2-0p006-superg/nc-rnd/2019/09/01/ESACCI-OZONE-L2P-NP-RAL_OMI_AURA-20190901005000_20190901005000-fv0214.nc
--> Converting ozone from units of DU to molec/cm2
--> Converting prior from units of DU to molec/cm2
Traceback (most recent call last):
File "sample_model.py", line 295, in
In running toar_sample, the first error I met arises from reading "Time" because the variables are saved into each group.
I think, I well followed your instruction in read.me file. I defined the variable name as a group_name/var_name so that time_var_name is given as 'HDFEOS/SWATHS/OMI Vertical Ozone Profile/Geolocation Fields/Time' in config.py.
The error message is follows as
Using the CAMS model
Retreving satellite file list
--> Found 1 satellite file
Traceback (most recent call last):
File "sample_model.py", line 291, in
satellite_info = get_satellite_data.meta_data(config_vars)
File "/data/tempo1/Shared/jbak/toar_sample/get_satellite_data.py", line 21, in init
self.start_date, self.end_date = self.get_satellite_date_range(config_vars)
File "/data/tempo1/Shared/jbak/toar_sample/get_satellite_data.py", line 58, in get_satellite_date_range
pass_time = nc_dataset.variables[config_vars.time_var_name]
KeyError: 'HDFEOS/SWATHS/OMI Vertical Ozone Profile/Geolocation Fields/Time'
Thank you for the clarification. Now I am facing a new problem with the sample_model
As a first test I wanted to read some S5p / tropomi data (https://s5phub.copernicus.eu/dhus/#/home).
However the reading routine (get_satellite_data.py)
Asumes the file to be flat (unstructured):
class extract_data(): def init(self,config_vars,sat_file):
nc_dataset = nc.Dataset(sat_file,'r')
lat_var = config_vars.lat_var_name ... prior_var = config_vars.prior_var_name time_var = config_vars.time_var_name self.latitude = nc_dataset.variables[lat_var][:] the data in the S5P format are structured:
lat_var= nc_dataset.groups['PRODUCT].variables['latitude'][:]
or
ak_var= nc_dataset.groups['PRODUCT'].groups['SUPPORT_DATA'].groups['DETAILED_RESULTS'].variables['averaging_kernel']
My suggeted sollution would be to allow the user GROUP/SUBGROUP/Variable as input in config.py and split the input according to
input_array=input_variable.split('/')
if len(input_array)==1:
_var= nc_datatset.variable[input_variable][:]
else:
readgroud=nc_dataset
for sub_group in range(len(input_array))-1:
readrgoup=readgroup.goups[input_array[sub_group]]
_var= readgroup.variable[input_array[-1]][:]
As an alternative we might use h5py.file instead of nc.Dataset
here we can directly use nc_datatset =h5py.file(sat_file)
nc_dataset[input_variable][:]
as long as the input is given as 'group/variable' it can read it
I will test the later version locally