search

innosat-mats / MATS-L1-processing

Python code for calibrating MATS images
MIT License
0 stars 1 forks source link

function dataframe_to_ccd_items does not seems to work if one send in a calibrated df #213

Closed lindamegner closed 7 months ago

lindamegner commented 7 months ago

🐛 Description

function dataframe_to_ccd_items in read_parquet_functions does not seem to work if one sends in a l1b dataframe. It crashes.

🔢 To reproduce

start_time = DT.datetime(2023, 5, 5, 20, 10)
stop_time = DT.datetime(2023, 5, 5, 20, 15)
df = read_MATS_data(start_time,stop_time,version='0.6',level='1b',dev=False)
CCDitems = dataframe_to_ccd_items(df)

*ERROR output : *****

KeyError                                  Traceback (most recent call last)
File [~/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/core/indexes/base.py:3653](https://file+.vscode-resource.vscode-cdn.net/Users/lindamegner/MATS/MATS-retrieval/~/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/core/indexes/base.py:3653), in Index.get_loc(self, key)
   [3652](file:///Users/lindamegner/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/core/indexes/base.py?line=3651) try:
-> [3653](file:///Users/lindamegner/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/core/indexes/base.py?line=3652)     return self._engine.get_loc(casted_key)
   [3654](file:///Users/lindamegner/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/core/indexes/base.py?line=3653) except KeyError as err:

File [~/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/_libs/index.pyx:147](https://file+.vscode-resource.vscode-cdn.net/Users/lindamegner/MATS/MATS-retrieval/~/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/_libs/index.pyx:147), in pandas._libs.index.IndexEngine.get_loc()

File [~/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/_libs/index.pyx:176](https://file+.vscode-resource.vscode-cdn.net/Users/lindamegner/MATS/MATS-retrieval/~/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/_libs/index.pyx:176), in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:7080, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:7088, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'ImageData'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
[/Users/lindamegner/MATS/MATS-retrieval/MATS-analysis/Linda/Calibration/find_error_script.py](https://file+.vscode-resource.vscode-cdn.net/Users/lindamegner/MATS/MATS-retrieval/MATS-analysis/Linda/Calibration/find_error_script.py) in line 6
      [74](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-analysis/Linda/Calibration/find_error_script.py?line=73) stop_time = DT.datetime(2023, 5, 5, 20, 15)
      [75](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-analysis/Linda/Calibration/find_error_script.py?line=74) df = read_MATS_data(start_time,stop_time,version='0.6',level='1b',dev=False)
----> [76](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-analysis/Linda/Calibration/find_error_script.py?line=75) CCDitems = dataframe_to_ccd_items(df)
      [77](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-analysis/Linda/Calibration/find_error_script.py?line=76) pickle.dump(CCDitems, open('testdata/CCD_items_in_orbit_l1b.pkl', 'wb'))
File [~/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py:188](https://file+.vscode-resource.vscode-cdn.net/Users/lindamegner/MATS/MATS-retrieval/~/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py:188), in dataframe_to_ccd_items(ccd_data, remove_empty, remove_errors, remove_warnings, legacy)
    [186](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=185) if legacy:
    [187](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=186)     add_ccd_item_attributes(data)
--> [188](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=187) convert_image_data(data)
    [189](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=188) rename_ccd_item_attributes(data)
    [191](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=190) return remove_faulty_rows(
    [192](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=191)     data,
    [193](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=192)     remove_empty,
    [194](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=193)     remove_errors,
    [195](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=194)     remove_warnings,
    [196](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=195) ).to_dict("records")

File [~/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py:81](https://file+.vscode-resource.vscode-cdn.net/Users/lindamegner/MATS/MATS-retrieval/~/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py:81), in convert_image_data(ccd_data)
     [71](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=70) """Convert image data from PNG data to float representation.
     [72](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=71) 
     [73](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=72) Args:
   (...)
     [77](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=76)     None:   Operation is performed in place.
     [78](file:///Users/lindamegner/MATS/MATS-retrieval/MATS-L1-processing/src/mats_l1_processing/read_parquet_functions.py?line=77) """
...
   [3658](file:///Users/lindamegner/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/core/indexes/base.py?line=3657)     #  InvalidIndexError. Otherwise we fall through and re-raise
   [3659](file:///Users/lindamegner/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/core/indexes/base.py?line=3658)     #  the TypeError.
   [3660](file:///Users/lindamegner/miniconda3/envs/MatsAnalysis/lib/python3.11/site-packages/pandas/core/indexes/base.py?line=3659)     self._check_indexing_error(key)
KeyError: 'ImageData'

✌🏽 Expected behaviour

return datafram mad into a CCDitems list

📷 Screenshots

image

💻 Traceback

Paste logs here...
lindamegner commented 7 months ago

I think it is you that has written this function right Andreas? Could you update it so that it can take in a calibrated df? Or, if that is tricky make another that takes in a calibrated df and spits out a list of CCDitems in a similar way?