processAltimeterData cannot save CSV and fails

[npucino]

I am trying to extract wave data from Port Philip Bay (VIC).

Not all the satellites are there, so I used a subset of sats as list (ppb_sats, see below code). However, the function fails when is trying to save a CSV with Pandas .to_csv method.

I suspect there is a problem with the previous steps of extracting information from NetCDFs but I cannot understand where. Here is what happens:

# Satellites subset in Port Philip Bay
ppb_sats=['SENTINEL-3A','SENTINEL-3B','CRYOSAT-2','JASON-2','SARAL','ERS-1']

# Wave analysis works fine
ppb= rwave.waveAnalysis(altimeterURL=url_file,bbox=[144.69,145.04,-38.04,-37.89],stime=[1995,1,1], etime=[2020,2,19])

# This would fail
ppb.processAltimeterData(altimeter_pick='all', saveCSV = 'altimeterData_ppb2.csv')

and the error is the following:

Processing Altimeter Dataset 

   +  name JASON-2     / number of tracks                               1   
   +  name SARAL       / number of tracks                               1   
   +  name SENTINEL-3A / number of tracks                               1   
   +  name CRYOSAT-2   / number of tracks                               2   
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<timed eval> in <module>

~\AppData\Local\Continuum\anaconda3\envs\wave_analysis\lib\site-packages\RADWave\altiwave.py in processAltimeterData(self, altimeter_pick, saveCSV)
    405                 sep=" ",
    406                 index=False,
--> 407                 header=1
    408             )
    409             altiData = pd.read_csv(

~\AppData\Local\Continuum\anaconda3\envs\wave_analysis\lib\site-packages\pandas\core\generic.py in to_csv(self, path_or_buf, sep, na_rep, float_format, columns, header, index, index_label, mode, encoding, compression, quoting, quotechar, line_terminator, chunksize, date_format, doublequote, escapechar, decimal)
   3200             doublequote=doublequote,
   3201             escapechar=escapechar,
-> 3202             decimal=decimal,
   3203         )
   3204         formatter.save()

~\AppData\Local\Continuum\anaconda3\envs\wave_analysis\lib\site-packages\pandas\io\formats\csvs.py in __init__(self, obj, path_or_buf, sep, na_rep, float_format, cols, header, index, index_label, mode, encoding, compression, quoting, line_terminator, chunksize, quotechar, date_format, doublequote, escapechar, decimal)
    113             else:
    114                 cols = list(cols)
--> 115             self.obj = self.obj.loc[:, cols]
    116 
    117         # update columns to include possible multiplicity of dupes

~\AppData\Local\Continuum\anaconda3\envs\wave_analysis\lib\site-packages\pandas\core\indexing.py in __getitem__(self, key)
   1760                 except (KeyError, IndexError, AttributeError):
   1761                     pass
-> 1762             return self._getitem_tuple(key)
   1763         else:
   1764             # we by definition only have the 0th axis

~\AppData\Local\Continuum\anaconda3\envs\wave_analysis\lib\site-packages\pandas\core\indexing.py in _getitem_tuple(self, tup)
   1287                 continue
   1288 
-> 1289             retval = getattr(retval, self.name)._getitem_axis(key, axis=i)
   1290 
   1291         return retval

~\AppData\Local\Continuum\anaconda3\envs\wave_analysis\lib\site-packages\pandas\core\indexing.py in _getitem_axis(self, key, axis)
   1952                     raise ValueError("Cannot index with multidimensional key")
   1953 
-> 1954                 return self._getitem_iterable(key, axis=axis)
   1955 
   1956             # nested tuple slicing

~\AppData\Local\Continuum\anaconda3\envs\wave_analysis\lib\site-packages\pandas\core\indexing.py in _getitem_iterable(self, key, axis)
   1593         else:
   1594             # A collection of keys
-> 1595             keyarr, indexer = self._get_listlike_indexer(key, axis, raise_missing=False)
   1596             return self.obj._reindex_with_indexers(
   1597                 {axis: [keyarr, indexer]}, copy=True, allow_dups=True

~\AppData\Local\Continuum\anaconda3\envs\wave_analysis\lib\site-packages\pandas\core\indexing.py in _get_listlike_indexer(self, key, axis, raise_missing)
   1551 
   1552         self._validate_read_indexer(
-> 1553             keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing
   1554         )
   1555         return keyarr, indexer

~\AppData\Local\Continuum\anaconda3\envs\wave_analysis\lib\site-packages\pandas\core\indexing.py in _validate_read_indexer(self, key, indexer, axis, raise_missing)
   1638             if missing == len(indexer):
   1639                 axis_name = self.obj._get_axis_name(axis)
-> 1640                 raise KeyError(f"None of [{key}] are in the [{axis_name}]")
   1641 
   1642             # We (temporarily) allow for some missing keys with .loc, except in

KeyError: "None of [Index(['lat', 'lon', 'wh', 'time', 'ws'], dtype='object')] are in the [columns]"

If I add a list of sats (ppb_sats) in the altimeter_pick parameter, it prints: "No altimeter data found...".

Any idea where is the error?

[tristan-salles]

Hi Nicolas,

Could you send over your url_file so I can try to replicate the error?

Cheers T

[tristan-salles]

Hi again @npucino,

I have updated the code and created a new version 1.0.2 which fix the problem. You will need to use the following command: wa.processAltimeterData(max_qc=5, altimeter_pick='all', saveCSV = 'altimeterData.csv')

where max_qc is the quality flag number. By default it was set to 1 and in your cases the flags where always over it. In this case the result was an empty data frame. The significant wave height control flag is defined in the netcdf files as the SWH_KA_quality_control or SWH_KU_quality_control depending on the different satellites. I have also "SENTINEL-3B" in the list of all as I didn't see it before.

It should fix your issues...

Cheers T

[npucino]

Perfect, thx Tristan-salles!

It is good that you kept the highest quality flag default. I found some sig.wh in Port Phillip Bay to be 8m high! That would be a disaster! Cheers