search

NOAA-EMC / WW3-tools

20 stars 16 forks source link

Fix type error when processing satellite data #43

Closed Ghazal-Mohammadpour closed 9 months ago

Ghazal-Mohammadpour commented 9 months ago

Pull Request Summary

This PR addresses a TypeError encountered when processing satellite data from JASON2. The issue was due to performing comparison operations on lists, which need to be supported for the involved data types.

Description

While running a script to process satellite altimetry data, we faced a type error that prevented the script from reading data from JASON2. The error message was as follows: TypeError: '>=' not supported between instances of 'list' and 'float' This error was triggered during logical operations within a np.where function, specifically when comparing elements of lists (shs, swsp, swdepth, swdistcoast) with float values from the configuration.

Issue(s) addressed

The solution implemented involves converting the affected lists (shs, swsp, swdepth, swdistcoast) to numpy arrays. By applying this fix, the script now successfully processes data from a variety of satellites without encountering the previously observed type error.

    swdepth = np.array(swdepth)
    swdistcoast = np.array(swdistcoast)
    shs = np.array(shs)
    swsp = np.array(swsp)

Testing

The fix was verified by re-running the script with the same dataset that initially triggered the error. Post-fix, the script executed successfully, processing data from multiple satellite sources without any type errors, confirming the effectiveness of the solution.

I request a review from the maintainers to ensure the changes align with project standards.

JessicaMeixner-NOAA commented 9 months ago

Please add to the PR description that this fixes issue #39

JessicaMeixner-NOAA commented 9 months ago

@Ghazal-Mohammadpour I found this issues when trying to run JASON2 for a period that I did not realize there was no JASON2 data. So I tried again to run for a period of time in which there was no JASON2 data and this time I got a different error:

For: python ProcSat_Altimeter.py JASON2 2020091400 2020091500 60.0 

wread.aodn_altimeter  successfully completed in 429 seconds. JASON2
Traceback (most recent call last):
  File "/work2/noaa/marine/jmeixner/TestPR43/ww3-tools/ww3tools/ProcSat_Altimeter.py", line 435, in <module>
    savesat(AODN_ALONGTRACK,wconfig,altsel)
  File "/work2/noaa/marine/jmeixner/TestPR43/ww3-tools/ww3tools/ProcSat_Altimeter.py", line 354, in savesat
    datein = datetime.utcfromtimestamp(AODN['TIME'].iloc[0]).strftime('%Y%m%d%H')
  File "/work2/noaa/marine/jmeixner/general/pythonenv/ww3tools/lib/python3.10/site-packages/pandas/core/indexing.py", line 1153, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/work2/noaa/marine/jmeixner/general/pythonenv/ww3tools/lib/python3.10/site-packages/pandas/core/indexing.py", line 1714, in _getitem_axis
    self._validate_integer(key, axis)
  File "/work2/noaa/marine/jmeixner/general/pythonenv/ww3tools/lib/python3.10/site-packages/pandas/core/indexing.py", line 1647, in _validate_integer
    raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds

I do not get an error when I choose a time period of which JASON2 has satelite info, for example: python ProcSat_Altimeter.py JASON2 2018091400 2018091500 60.0

If others agree we can merge this code, but then I think we need to also have a more graceful exit to the code if there is no data. Do you also get the error I'm seeing above for a period in which we do not have JASON2 data or is there something on orion/my environment?

JessicaMeixner-NOAA commented 9 months ago

@Ghazal-Mohammadpour Can you share what a good commit message for this PR will be? As soon as we have that I will merge.

Ghazal-Mohammadpour commented 9 months ago

Updates for JASON2 and to exit with message when there is no data

On Thu, Feb 8, 2024 at 7:05 AM Jessica Meixner @.***> wrote:

@Ghazal-Mohammadpour https://github.com/Ghazal-Mohammadpour Can you share what a good commit message for this PR will be? As soon as we have that I will merge.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/WW3-tools/pull/43#issuecomment-1934312556, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5W2POEV6EKCMF44D7RSFTDYSTSUPAVCNFSM6AAAAABC5GT6FSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZUGMYTENJVGY . You are receiving this because you were mentioned.Message ID: @.***>