Log of email issues

[dpfinch]

• I’ve tried to run the code commenting the else blocks around lines 433 and 462 in utils.py. With that, I could reach the end of the code but the results look strange.
• In adding to my previous comment on the unit conversion, I’d like to draw your attention to a point when applying the AK. I didn’t go to all your pieces of code in detail but as far as I quickly go through, I have understood you convert the model in molecule/cm2 before applying the AK. I’m not sure it will work for all the products. For example, for our product we retrieve vmr and then our ak correspond to o3 profiles in vmr. One can apply them directly to vmr O3 profile but I don’t think it is possible to apply them directly to molecule/cm2 columns without some conversions (we did that a long time ago at the beginning of IASI, but I don’t remember exactly what the issue was). In general, I would recommend to transform the model in the units of the o3 product, apply the ak and at the end convert to molecule/cm2. It should prevent some issues.

[dpfinch]

First, for the units, it seems that in the conversion part after line 150 of get_satellite_data.py, the case of « v/v » is not considered to be converted in molec cm-2. I was in this case and got the error message « unable to converts o3 units of v/v… » Then, I converted in my test nc file, my O3 profile to ppb to see if it goes further. It does (see next question). Still on units, I wonder if the conversion ppb —> molec cm-2 is correct. I would expect to have something more complicated for the conversion such as vmr(dP/g)(Na/mair). What do you think?

Second, independently of the units issue, I continue using ppb units. I succeed to download CAMS outputs. But the script stops without doing anything. I tracked it is quitting the code at line 432 of the utils.py script. I don’t know what’s going wrong. Below the last lines written on my screen.

--> Converting ozone from units of ppb to molec/cm2 --> Converting prior from units of ppb to molec/cm2 --> Cannot find units for level. Assuming hPa. 2022-09-30 09:10:48,027 INFO Welcome to the CDS 2022-09-30 09:10:48,028 INFO Sending request to https://ads.atmosphere.copernicus.eu/api/v2/resources/cams-global-reanalysis-eac4 2022-09-30 09:10:48,103 INFO Request is queued 2022-09-30 09:10:49,133 INFO Request is running 2022-09-30 09:10:52,944 INFO Request is completed 2022-09-30 09:10:52,945 INFO Downloading https://download-0003-ads-clone.copernicus-climate.eu/cache-compute-0003/cache/data1/adaptor.mars.internal-1664521848.1761296-22594-7-b957cfe8-358c-46db-b2fa-8ccc22e6671f.grib to /DATA/SPECAT_2/gaelle/TOAR2/toar_sample/OUT/model_download/1664521847_CAMS_model.grib (755.9K) 2022-09-30 09:10:53,333 INFO Download rate 1.9M/s Warning: ecCodes 2.21.0 or higher is recommended. You are running version 2.18.0 Approximately 0% of profiles in this file complete. ** Quiting.

[dpfinch]

SAMPLING MODEL FOR OMI-RAL

Using the CAMS model Retreving satellite file list --> Found 11445 satellite files --> Satellite records start from 2019-09-01. Changing start date to match --> Sampling model at satellite observations coordinates. Sampling file: /home/users/blatter/Data/projects/ral_ozone_scheme/omi_aura/v003/omi_aura-solopt7-rat1-cclev2-saap-lip-samp1a2-samp2b2-af0-cmacc-pcbira-ecm-ec5-sfg2s-rsfw3-ng37-pol4-sc10-noxfr-ch2o-nosw-retpoly-s5xs-sckxs-ring4-smcl2-mplap-g2lk-nomr-cci-newfm2b-b2leak2-dxs2-0p006-superg/nc-rnd/2019/09/01/ESACCI-OZONE-L2P-NP-RAL_OMI_AURA-20190901005000_20190901005000-fv0214.nc --> Converting ozone from units of DU to molec/cm2 --> Converting prior from units of DU to molec/cm2 Traceback (most recent call last): File "sample_model.py", line 295, in sat_data = sample_model(config_vars, satellite_info) File "sample_model.py", line 187, in sample_model model_data = get_CAMS_data(config_vars, sat_data) File "/home/users/blatter/python/toar/toar_sample/CAMS_model_data.py", line 32, in init self.download_model_data(config_vars) File "/home/users/blatter/python/toar/toar_sample/CAMS_model_data.py", line 53, in download_model_data c = cdsapi.Client() File "/home/users/blatter/python_venv/toar_sample/lib/python3.8/site-packages/cdsapi/api.py", line 301, in init raise Exception("Missing/incomplete configuration file: %s" % (dotrc)) Exception: Missing/incomplete configuration file: /home/users/blatter/.cdsapirc

[dpfinch]

In running toar_sample, the first error I met arises from reading "Time" because the variables are saved into each group.

I think, I well followed your instruction in read.me file. I defined the variable name as a group_name/var_name so that time_var_name is given as 'HDFEOS/SWATHS/OMI Vertical Ozone Profile/Geolocation Fields/Time' in config.py.

The error message is follows as

SAMPLING MODEL FOR OMI-Aura

Using the CAMS model

Retreving satellite file list

--> Found 1 satellite file

Traceback (most recent call last):

File "sample_model.py", line 291, in

satellite_info = get_satellite_data.meta_data(config_vars)

File "/data/tempo1/Shared/jbak/toar_sample/get_satellite_data.py", line 21, in init

self.start_date, self.end_date = self.get_satellite_date_range(config_vars)

File "/data/tempo1/Shared/jbak/toar_sample/get_satellite_data.py", line 58, in get_satellite_date_range

pass_time = nc_dataset.variables[config_vars.time_var_name]

KeyError: 'HDFEOS/SWATHS/OMI Vertical Ozone Profile/Geolocation Fields/Time'

[dpfinch]

Thank you for the clarification. Now I am facing a new problem with the sample_model

As a first test I wanted to read some S5p / tropomi data (https://s5phub.copernicus.eu/dhus/#/home).

However the reading routine (get_satellite_data.py)

Asumes the file to be flat (unstructured):

class extract_data(): def init(self,config_vars,sat_file):

TODO: Change to a 'with' statement

TODO: Need to make this configurable

nc_dataset = nc.Dataset(sat_file,'r')

Try to get coord variables either as full names or shortened

lat_var = config_vars.lat_var_name ... prior_var = config_vars.prior_var_name time_var = config_vars.time_var_name self.latitude = nc_dataset.variables[lat_var][:] the data in the S5P format are structured:

lat_var= nc_dataset.groups['PRODUCT].variables['latitude'][:]

or

ak_var= nc_dataset.groups['PRODUCT'].groups['SUPPORT_DATA'].groups['DETAILED_RESULTS'].variables['averaging_kernel']

My suggeted sollution would be to allow the user GROUP/SUBGROUP/Variable as input in config.py and split the input according to

input_array=input_variable.split('/')

if len(input_array)==1:

_var= nc_datatset.variable[input_variable][:]

else:

readgroud=nc_dataset

for sub_group in range(len(input_array))-1:

readrgoup=readgroup.goups[input_array[sub_group]]

_var= readgroup.variable[input_array[-1]][:]

As an alternative we might use h5py.file instead of nc.Dataset

here we can directly use nc_datatset =h5py.file(sat_file)

nc_dataset[input_variable][:]

as long as the input is given as 'group/variable' it can read it

I will test the later version locally