Unable to clean downloaded data

[wxguy]

Please Describe the bug Sounderpy is unable to clean the data properly resulting in following error:-

## ---------------------------------- SOUNDERPY ----------------------------------- ##
##          Vertical Profile Data Retrieval and Analysis Tool For Python            ##
##                     v3.0.4 | June 2024 | (C) Kyle J Gillett                      ##
##                 Docs: https://kylejgillett.github.io/sounderpy/                  ##
## --------------------- THANK YOU FOR USING THIS PACKAGE! ------------------------ ##

> OBSERVED DATA ACCESS FUNCTION --
-----------------------------------
> PROFILE FOUND: VOMM on 09/13/2024 at 00z | From UW
> COMPLETE --------
> RUNTIME: 00:00:03
> SUMMARY: 00Z Launch for VOMM, MADRAS/MINAMBAKKAM at 09-13-2024-00Z

> THERMODYNAMICS --------------------------------------------- 
--- SBCAPE: 3464.3 | MUCAPE: 3464.3 | MLCAPE: 2380.6 | MUECAPE: 250.5
--- MU 0-3: 201.8 | MU 0-6: 669.1 | SB 0-3: 201.8 | SB 0-6: 669.1

> KINEMATICS ------------------------------------------------- 
--- 0-500 SRW: 19.5 knot | 0-500 SWV: 0.015 | 0-500 SHEAR: 13.3 | 0-500 SRH: 87.2
--- 1-3km SRW: 12.5 knot | 1-3km SWV: 0.044 | 1-3km SHEAR: 1.3 | | 1-3km SRH: 5.5
============================================================== 

> SOUNDING PLOTTER FUNCTION --
---------------------------------
Traceback (most recent call last):
  File "/mnt/34B2CD28B2CCEF82/Data/myprojects/tephigram.py", line 4, in <module>
    spy.build_sounding(data, save=True, filename="sample.png")
  File "/home/sundar/miniconda3/envs/vatavar/lib/python3.11/site-packages/sounderpy/sounderpy.py", line 1505, in build_sounding
    __full_sounding(clean_data, color_blind, dark_mode, storm_motion, special_parcels, show_radar, radar_time, map_zoom, modify_sfc).savefig(filename, bbox_inches='tight')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sundar/miniconda3/envs/vatavar/lib/python3.11/site-packages/sounderpy/plot.py", line 466, in __full_sounding
    trace = calc_ecape_parcel(p, z, T, Td, u, v, 
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sundar/miniconda3/envs/vatavar/lib/python3.11/site-packages/metpy/units.py", line 333, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/sundar/miniconda3/envs/vatavar/lib/python3.11/site-packages/ecape_parcel/calc.py", line 234, in calc_ecape_parcel
    ecape, ncape = calc_ecape_ncape(height, pressure, temperature, specific_humidity, u_wind, v_wind, cape_type, cape, inflow_bottom=inflow_layer_bottom, inflow_top=inflow_layer_top, storm_motion=storm_motion_type, lfc=lfc, el=el, u_sm=storm_motion_u, v_sm=storm_motion_v)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sundar/miniconda3/envs/vatavar/lib/python3.11/site-packages/metpy/units.py", line 333, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/sundar/miniconda3/envs/vatavar/lib/python3.11/site-packages/ecape_parcel/ecape_calc.py", line 657, in calc_ecape_ncape
    el_idx = np.where(height_msl > el)[0][0]
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: index 0 is out of bounds for axis 0 with size 0

To Reproduce Steps to reproduce the behavior:

import sounderpy as spy

data = spy.get_obs_data("vomm", "2024", "09", "13", "00")
spy.build_sounding(data, save=True, filename="sample.png")

Expected behavior Process and plot the data after downloading without an error.

Screenshots No applicable. See the error message above.

What are you running?

• OS: Linux (Arch Linux)
• SounderPy Version - Latest version from Pypi

[kylejgillett]

Hi @wxguy,

Thanks for submitting this issue.

Unfortunately at this time I do not have a good fix for this. The error you have found is coming from the ecape-parcel library which stems from an issue with the data itself. (i.e. not the fault of the data cleaning process, or ecape-parcel)

For some reason, unknown to me at this time, a number of profiles from the University of Wyoming archive come back with incorrect height values somewhere in the profiles. See the example below. These height data errors don't appear on the University of Wyoming's web-based archive viewer, so the issue may be traced back to the siphon library which does the loading/reading/outputting of the data from the University of Wyoming. I have been looking into this issue for some time now. Hopefully I have a real fix soon.

Workaround options to fix this are limited at this time, but using the IRGAv2 data source is the best option right now. SounderPy can also retrieve data from the IRGAv2 archive the same way as data from UW, just by using the IGRAv2 profile ID instead. The only issues with IGRAv2 access are that it's a bit slower (x2 the time) and the data doesn't populate in the archive as quickly as UW.

Ex: Your station in RAOB form: "vomm", or "43279" and your station in IRGAv2 form: "INM00043279"

data = spy.get_obs_data("INM00043279", "2024", "09", "13", "00")

Example height data errors on another profile from UW:

For now I will leave this issue open until I have this height data error issue resolved as a whole.

[kylejgillett]

Update on this issue:

I believe I have an "in-house" fix for this. In this fix, the data still downloads incorrectly, but during the "cleaning process" SounderPy can identify if the height data is erroneous and attempt to correct it.

This will be implemented with the next release of SounderPy. However, for now, until this is tested enough to be proved as the correct fix, I will leave this issue open.