SnowEx / snowexsql

A project to access the SnowEx database holding data from the NASA SnowEx campaign
https://snowexsql.readthedocs.io/en/latest/readme.html
GNU General Public License v3.0
25 stars 6 forks source link

Metadata doesn't properly read SMP header #35

Closed micahjohnson150 closed 3 years ago

micahjohnson150 commented 3 years ago

The SMP CSVs are not being read correctly which leads to a missing location information. This has never happened before because when we first uploaded the SMP data it was with the SMP log file to account for updates to the time issue in the files. Thats since been fixed and were no longer using the SMP log file.

What I did:

What happened:

snowexsql.batch INFO Accessing Database localhost/snowex
snowexsql.batch INFO Preparing to upload 958 files...
snowexsql.metadata INFO Interpreting metadata in /home/ubuntu/snowexsql/scripts/download/data/SNOWEX/SNEX20_SMP.001/csv_resampled/SNEX20_SMP_S06M0655_1N6_20200128.CSV
snowexsql.metadata DEBUG Found end of header at line 8...
snowexsql.metadata INFO Names to be uploaded as main data are: force
snowexsql.metadata DEBUG Column Data found to be 3 columns based on Line 7
snowexsql.metadata DEBUG Discovered 0 lines of valid header info.
Traceback (most recent call last):
  File "add_smp.py", line 45, in <module>
    main()
  File "add_smp.py", line 38, in main
    b.push()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/snowexsql-0.1.0-py3.8.egg/snowexsql/batch.py", line 223, in push
    self._push_one(f, **meta)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/snowexsql-0.1.0-py3.8.egg/snowexsql/batch.py", line 127, in _push_one
    d = self.UploaderClass(f, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/snowexsql-0.1.0-py3.8.egg/snowexsql/upload.py", line 34, in __init__
    self.hdr = DataHeader(profile_filename, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/snowexsql-0.1.0-py3.8.egg/snowexsql/metadata.py", line 377, in __init__
    self.info = self.interpret_data(info)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/snowexsql-0.1.0-py3.8.egg/snowexsql/metadata.py", line 717, in interpret_data
    info = add_geom(info, self.epsg)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/snowexsql-0.1.0-py3.8.egg/snowexsql/projection.py", line 72, in add_geom
    epsg, info['easting'], info['northing']),
KeyError: 'easting'

In testing I added a test and I was able to reproduce locally.