calliope-project / calliope

A multi-scale energy systems modelling framework
https://www.callio.pe
Apache License 2.0
277 stars 89 forks source link

Update data source dtype coercion #560

Closed brynpickering closed 4 months ago

brynpickering commented 4 months ago

Fixes issue where data sources contain numeric data with separators.

E.g., 25,000 would be loaded by stored in CSV as "25,000" and pandas can't natively coerce that to a numeric dtype.

Summary of changes in this pull request:

Reviewer checklist:

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (6db0657) 95.76% compared to head (9625db9) 95.77%. Report is 1 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #560 +/- ## ======================================= Coverage 95.76% 95.77% ======================================= Files 24 24 Lines 3566 3573 +7 Branches 777 713 -64 ======================================= + Hits 3415 3422 +7 Misses 86 86 Partials 65 65 ``` | [Files](https://app.codecov.io/gh/calliope-project/calliope/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project) | Coverage Δ | | |---|---|---| | [src/calliope/preprocess/data\_sources.py](https://app.codecov.io/gh/calliope-project/calliope/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project#diff-c3JjL2NhbGxpb3BlL3ByZXByb2Nlc3MvZGF0YV9zb3VyY2VzLnB5) | `100.00% <100.00%> (ø)` | |
brynpickering commented 4 months ago

Closing as, after some testing, locale setting turns out to not be that powerful. E.g., swiss locale can't handle 1'000.0 and german locale can't handle 1.000,0. Users will just need to sanitise their Excel data before loading it into Calliope.