[Bug]: A dvas error occures during first run of recipe UAII2022

msommer24 commented 2 hours ago

Is there already an issue for this?

[X] I found no existing issue for this bug.

Describe the bug

Following dvas error occurs during first run of a fresh installed dvas with downloaded data package of UAII2022: Usecols do not match columns, columns expected but not found: ['temp[ËšC]']

Steps to reproduce

Install (correctly) dvas
Download and unpack UAII2022 data packed (WMO's UAII 2022 Supplementary Material)
Setup of the dvas processing arena (e.g. paths)
Try to run dvas recipe using dvas_run_recipe uaii2022.rcp uaii2022_fid-eid-rid-edt_day.csv -s '00' -e '00' -f F01

Relevant log output

Traceback (most recent call last):
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\python_parser.py", line 607, in _handle_usecols
    col_indices.append(usecols_key.index(col))
                       ^^^^^^^^^^^^^^^^^^^^^^
ValueError: 'temp[ËšC]' is not in list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\temp\dvas\dvas-1.1.0\src\dvas\data\linker.py", line 932, in eval
    expr_out = expr_out.interpret()
               ^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\dvas\dvas-1.1.0\src\dvas\data\linker.py", line 1063, in interpret
    out = self._FCT(self._expression, *self._ARGS, **self._KWARGS)  # noqa pylint: disable=E1102
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\dvas\dvas-1.1.0\src\dvas\data\linker.py", line 615, in get_data
    data = pd.read_csv(data_file_path, **original_csv_read_args).squeeze("columns")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py", line 1898, in _make_engine
    return mapping[engine](f, **self.options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\python_parser.py", line 133, in __init__
    ) = self._infer_columns()
        ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\python_parser.py", line 542, in _infer_columns
    columns = self._handle_usecols(
              ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\python_parser.py", line 609, in _handle_usecols
    self._validate_usecols_names(self.usecols, usecols_key)
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\base_parser.py", line 979, in _validate_usecols_names
    raise ValueError(
ValueError: Usecols do not match columns, columns expected but not found: ['temp[ËšC]']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\msommer\AppData\Local\Programs\Python\Python312\Scripts\dvas_run_recipe.exe\__main__.py", line 7, in <module>
  File "D:\temp\dvas\dvas-1.1.0\src\dvas_recipes\__main__.py", line 125, in dvas_run_recipe
    run_recipe(args.rcp_fn, args.fid_log_fn, fid_to_treat=args.f,
  File "D:\temp\dvas\dvas-1.1.0\src\dvas_recipes\hl_commands.py", line 298, in run_recipe
    rcp.execute(from_step_id=from_step_id, until_step_id=until_step_id)
  File "D:\temp\dvas\dvas-1.1.0\src\dvas_recipes\recipe.py", line 422, in execute
    self.init_db(reset=self._reset_db, data_in_db=self._data_in_db)
  File "D:\temp\dvas\dvas-1.1.0\src\dvas_recipes\recipe.py", line 368, in init_db
    DB.fetch_original_data([rcp_dyn.INDEXES[PRF_TDT]] +  # The reference times
  File "D:\temp\dvas\dvas-1.1.0\src\dvas\dvas.py", line 81, in fetch_original_data
    update_db(search_prm, strict=strict)
  File "D:\temp\dvas\dvas-1.1.0\src\dvas\data\io.py", line 118, in update_db
    result = handler.handle(file_path, prm_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\dvas\dvas-1.1.0\src\dvas\data\linker.py", line 256, in handle
    return self._get_main(data_file_path, prm_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\dvas\dvas-1.1.0\src\dvas\data\linker.py", line 281, in _get_main
    data = LoadExprInterpreter.eval(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\temp\dvas\dvas-1.1.0\src\dvas\data\linker.py", line 937, in eval
    raise DvasError(exp)
dvas.errors.DvasError: Usecols do not match columns, columns expected but not found: ['temp[ËšC]']

Environment

- OS: Windows 10
- Python: 3.12
- dvas: 1.1.0

Code of Conduct

[X] I agree to follow the dvas Code of Conduct.

msommer24 commented 2 hours ago

Maybe this issue is related to file dvas_demo\config\data_config\PS-B3_origdata.yml with following lines:

_temp:
    value: get('temp[˚C]', 'c2k')
So, maybe it is an encoding issue.

fpavogt commented 1 hour ago

Yes, seeing as this error appears on Windows, this would be my first guess. I'll need to track down a Windows machine to try to reproduce this.

It has also been a while that the last automated tests (using Github actions) were run on a Windows machine - I might try to re-trigger them to see if they still pass.

MeteoSwiss / dvas