Regression: current parser fails on v2 SST data which parser v1.12 could succeed on

[tcj]

It seems (maybe) the concatenation phase of the SST data is outputting blank lines for legacy Spotter. cc: @pieterbartsmit

parser_v1.12.py —

[...]
Concatenating all SST files:
- 0533_SST.CSV (File 1 out of 109)
- 0534_SST.CSV (File 2 out of 109)
[...]
- 0640_SST.CSV (File 108 out of 109)
- 0641_SST.CSV (File 109 out of 109)
Processing spotter displacement output
% 

sst.csv contents:

% wc -l sst.csv 
  142311 sst.csv
% sort -u sst.csv
[...]
2022,3,9,9,6,26,0,13.88
2022,3,9,9,7,26,0,13.88
2022,3,9,9,8,26,0,13.88
2022,3,9,9,9,27,0,13.88

current parser —

Processing Spotter displacement output - SST
Traceback (most recent call last):
  File "/Users/tcj/src/spotter-sd-parser/sd_file_parser.py", line 1738, in <module>
    main(**arguments)
  File "/Users/tcj/src/spotter-sd-parser/sd_file_parser.py", line 533, in main
    parseLocationFiles(inputFileName = fileName,kind=suffix,
  File "/Users/tcj/src/spotter-sd-parser/sd_file_parser.py", line 639, in parseLocationFiles
    data = pd.read_csv(inputFileName,
  File "/Users/tcj/Library/Python/3.9/lib/python/site-packages/pandas/util/_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/Users/tcj/Library/Python/3.9/lib/python/site-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/Users/tcj/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/Users/tcj/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/readers.py", line 605, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/tcj/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/Users/tcj/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/readers.py", line 1753, in _make_engine
    return mapping[engine](f, **self.options)
  File "/Users/tcj/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 79, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 554, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file

sst.csv contents 284622 blank rows (!) (double the count of the v1.12 parser):

% wc -l sst.csv  
  284622 sst.csv
% sort -u sst.csv

% 

[tcj]

also worth noting that this dataset has two "corrupt" SST files:

- ERROR:, file [...]0554_SST.CSV is corrupt
- ERROR:, file [...]0586_SST.CSV is corrupt

[tcj]

(removing the two "corrupt" SST CSV files does not make a difference)

[tcj]

fixed; closing