search

pik-primap / primap2

The next generation of the PRIMAP climate policy analysis suite
https://primap2.readthedocs.io
Apache License 2.0
8 stars 2 forks source link

CSV reading with non-comma separators #210

Open JGuetschow opened 3 months ago

JGuetschow commented 3 months ago

Is your feature request related to a problem? Please describe.

When reading data from csv files, only files with comma as separator can be read using the read...csv_file_if functions as the sep parameter in pd.read_csv can not be specified

Describe the solution you'd like

Add the sep parameter to primap2 csv reading routines

Describe alternatives you've considered

Read csv using pandas function and convert in a second thep

mikapfl commented 3 months ago

Read csv using pandas function and convert in a second thep

Might actually be worth, the API surface of pd.read_csv is huge and next thing we probably want to read from database or something, which pandas can do, but we don't have a function. So, I'd say: just use pandas' reading functions, then convert_long_dataframe_if.

JGuetschow commented 3 months ago

I do that all the time. At some point we might remove the csv reading functions as they can basically never be used

mikapfl commented 3 months ago

Yeah, I think reading the interchange format with a specific function has value, but in general so-called CSVs are super diverse. Probably it was not realistic to expect them to be sufficiently well-formed to read them with one function. /-: