search

umr-lops / grdwindinversion

python library to compute wind speed from GRD SAR images
https://cerweb.ifremer.fr/datarmor/doc_sphinx/grdwindinversion/index.html
MIT License
9 stars 1 forks source link

RCM processing are in error #22

Closed agrouaze closed 3 months ago

agrouaze commented 1 year ago 
python main.py --input_file /home/datawork-cersat-public/provider/asc-csa/satellite/l1/rcm/rcm3/SCLNC/GRD/2023/137/RCM3_OK2554779_PK2560509_1_SCLNC_20230517_132957_VV_VH_GRD/ --outputdir /tmp/

traceback:

10/07/2023 12:00:44 INFO schemas.py(1234) Resource 'XMLSchema.xsd' is already loaded                                                                                                                          
/opt/conda-envs/dev/lib/python3.10/site-packages/safe_rcm/product/reader.py:59: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be rela
xed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to 
the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  {"timeStamp": ds["timeStamp"].astype("datetime64")}                                                                                                                                                         
/opt/conda-envs/dev/lib/python3.10/site-packages/safe_rcm/product/reader.py:68: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be rela
xed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to 
the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.                                              
  {"timeStamp": ds["timeStamp"].astype("datetime64")}                                                                                                                                                         
10/07/2023 12:00:48 ERROR io.py(224) FilePath is currently a read-only interface.                                                                                                                             
10/07/2023 12:00:48 ERROR io.py(224) FilePath is currently a read-only interface.                                                                                                                             
10/07/2023 12:00:48 ERROR io.py(224) File-like object not found in virtual filesystem: b'4c0b12dc-4cb1-44b9-9279-d1232838c090/4c0b12dc-4cb1-44b9-9279-d1232838c090.aux'
10/07/2023 12:00:48 ERROR io.py(224) File-like object not found in virtual filesystem: b'4c0b12dc-4cb1-44b9-9279-d1232838c090/4c0b12dc-4cb1-44b9-9279-d1232838c090.AUX'
10/07/2023 12:00:48 ERROR io.py(224) File-like object not found in virtual filesystem: b'4c0b12dc-4cb1-44b9-9279-d1232838c090/4c0b12dc-4cb1-44b9-9279-d1232838c090.aux'
10/07/2023 12:00:48 ERROR io.py(224) File-like object not found in virtual filesystem: b'4c0b12dc-4cb1-44b9-9279-d1232838c090/4c0b12dc-4cb1-44b9-9279-d1232838c090.AUX'
10/07/2023 12:00:48 ERROR io.py(224) FilePath is currently a read-only interface.
10/07/2023 12:00:48 ERROR io.py(224) FilePath is currently a read-only interface.
10/07/2023 12:00:48 ERROR io.py(224) File-like object not found in virtual filesystem: b'f8fbcc5c-b210-4bd0-9597-7bd55b111071/f8fbcc5c-b210-4bd0-9597-7bd55b111071.aux'
10/07/2023 12:00:48 ERROR io.py(224) File-like object not found in virtual filesystem: b'f8fbcc5c-b210-4bd0-9597-7bd55b111071/f8fbcc5c-b210-4bd0-9597-7bd55b111071.AUX'
10/07/2023 12:00:48 ERROR io.py(224) File-like object not found in virtual filesystem: b'f8fbcc5c-b210-4bd0-9597-7bd55b111071/f8fbcc5c-b210-4bd0-9597-7bd55b111071.aux'
10/07/2023 12:00:48 ERROR io.py(224) File-like object not found in virtual filesystem: b'f8fbcc5c-b210-4bd0-9597-7bd55b111071/f8fbcc5c-b210-4bd0-9597-7bd55b111071.AUX'
Traceback (most recent call last):
  File "/home1/datahome/agrouaze/sources/git/grdwindinversion/grdwindinversion/main.py", line 48, in <module>
    out_file = makeL2(input_file, out_folder, config_file,overwrite=args.overwrite)
  File "/home1/datahome/agrouaze/sources/git/grdwindinversion/grdwindinversion/inversion.py", line 120, in makeL2
    meta = fct_meta(filename)
  File "/home1/datahome/agrouaze/sources/git/xsar/src/xsar/utils.py", line 94, in wrapper
    result = f(*args, **kwargs)
  File "/home1/datahome/agrouaze/sources/git/xsar/src/xsar/rcm_meta.py", line 93, in __init__
    self.manifest_attrs = self._create_manifest_attrs()
  File "/home1/datahome/agrouaze/sources/git/xsar/src/xsar/rcm_meta.py", line 196, in _create_manifest_attrs
    dic["swath_type"] = os.path.basename(self.path).split('_')[4]
IndexError: list index out of range
agrouaze commented 1 year ago

another issue?

python main.py --input_file /home/datawork-cersat-public/provider/asc-csa/satellite/l1/rcm/rcm3/SC50MC/GRD/2023/162/RCM3_OK2597128_PK2600375_1_SC50MC_20230611_134458_HH_HV_GRD/ --outputdir /home1/scratch/agrouaze/ --resolution 2 --overwrite
161/ 162/ 
the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  {"timeStamp": ds["timeStamp"].astype("datetime64")}
/opt/conda-envs/dev/lib/python3.10/site-packages/safe_rcm/product/reader.py:68: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  {"timeStamp": ds["timeStamp"].astype("datetime64")}

10/07/2023 13:58:12 ERROR io.py(224) FilePath is currently a read-only interface.
10/07/2023 13:58:12 ERROR io.py(224) FilePath is currently a read-only interface.
10/07/2023 13:58:12 ERROR io.py(224) File-like object not found in virtual filesystem: b'246e050c-dbd6-4b3f-acce-32f5cd35a37b/246e050c-dbd6-4b3f-acce-32f5cd35a37b.aux'
10/07/2023 13:58:12 ERROR io.py(224) File-like object not found in virtual filesystem: b'246e050c-dbd6-4b3f-acce-32f5cd35a37b/246e050c-dbd6-4b3f-acce-32f5cd35a37b.AUX'
10/07/2023 13:58:12 ERROR io.py(224) File-like object not found in virtual filesystem: b'246e050c-dbd6-4b3f-acce-32f5cd35a37b/246e050c-dbd6-4b3f-acce-32f5cd35a37b.aux'
10/07/2023 13:58:12 ERROR io.py(224) File-like object not found in virtual filesystem: b'246e050c-dbd6-4b3f-acce-32f5cd35a37b/246e050c-dbd6-4b3f-acce-32f5cd35a37b.AUX'
10/07/2023 13:58:12 ERROR io.py(224) FilePath is currently a read-only interface.
10/07/2023 13:58:12 ERROR io.py(224) FilePath is currently a read-only interface.
10/07/2023 13:58:12 ERROR io.py(224) File-like object not found in virtual filesystem: b'b71745a7-5471-4ba2-90bb-ae9b6a248ce2/b71745a7-5471-4ba2-90bb-ae9b6a248ce2.aux'
10/07/2023 13:58:12 ERROR io.py(224) File-like object not found in virtual filesystem: b'b71745a7-5471-4ba2-90bb-ae9b6a248ce2/b71745a7-5471-4ba2-90bb-ae9b6a248ce2.AUX'
10/07/2023 13:58:12 ERROR io.py(224) File-like object not found in virtual filesystem: b'b71745a7-5471-4ba2-90bb-ae9b6a248ce2/b71745a7-5471-4ba2-90bb-ae9b6a248ce2.aux'
10/07/2023 13:58:12 ERROR io.py(224) File-like object not found in virtual filesystem: b'b71745a7-5471-4ba2-90bb-ae9b6a248ce2/b71745a7-5471-4ba2-90bb-ae9b6a248ce2.AUX'
10/07/2023 13:58:58 WARNING base_dataset.py(813) Raster variable are experimental
Traceback (most recent call last):
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3652, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'VV'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/indexes.py", line 486, in sel
    indexer = self.index.get_loc(label_value)
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3654, in get_loc
    raise KeyError(key) from err
KeyError: 'VV'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home1/datahome/agrouaze/sources/git/grdwindinversion/grdwindinversion/main.py", line 50, in <module>
    out_file = makeL2(input_file, out_folder, config_file,overwrite=args.overwrite)
  File "/home1/datahome/agrouaze/sources/git/grdwindinversion/grdwindinversion/inversion.py", line 241, in makeL2
    dataset_1000m['owiNrcs'] = dataset_1000m['sigma0_ocean'].sel(pol='VV')
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/dataarray.py", line 1550, in sel
    ds = self._to_temp_dataset().sel(
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/dataset.py", line 2653, in sel
    query_results = map_index_queries(
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/indexing.py", line 190, in map_index_queries
    results.append(index.sel(labels, **options))
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/indexes.py", line 488, in sel
    raise KeyError(
KeyError: "not all values found in index 'pol'. Try setting the `method` keyword argument (example: method='nearest')."
vincelhx commented 1 year ago

For the first issue, try /home/datawork-cersat-public/provider/asc-csa/satellite/l1/rcm/rcm3/SCLNC/GRD/2023/137/RCM3_OK2554779_PK2560509_1_SCLNC_20230517_132957_VV_VH_GRD instead of /home/datawork-cersat-public/provider/asc-csa/satellite/l1/rcm/rcm3/SCLNC/GRD/2023/137/RCM3_OK2554779_PK2560509_1_SCLNC_20230517_132957_VV_VH_GRD/

agrouaze commented 1 year ago

Ok the right trailing slash issue as already been fixed. The problem is in fact that your algorithm only applied for dual pol acquisition in VV for now. Question is do I let the algo also process:

agrouaze commented 1 year ago

for now I will add a secuity on the main.py script stating that :

this processor only handle dual polarization acquisitions VV+VH for now.
vincelhx commented 3 months ago

closed; can handle single or dualpol can handle RCM