IAMconsortium / pyam

Analysis & visualization of energy & climate scenarios
https://pyam-iamc.readthedocs.io/
Apache License 2.0
221 stars 115 forks source link

Different approach to unstack in `timeseries()` #720

Closed danielhuppmann closed 1 year ago

danielhuppmann commented 1 year ago

Please confirm that this PR has done the following:

Adding to RELEASE_NOTES.md (remove section after adding to RELEASE_NOTES.md)

Please add a single line in the release notes similar to the following:

- (#XX)[http://link-to-pr.com] Added feature which does something

Description of PR

I ran into a MemoryError yesterday when writing a large IamDataFrame to file so I tried to implement a more performant approach for pvioting to the wide IAMC format.

danielhuppmann commented 1 year ago

Re questions by @phackstock:

I did not yet run any profiling and this change will (unfortunately) not change the performance issues when reading data.

phackstock commented 1 year ago

Ah ok thanks for the clarifications @danielhuppmann. Still very good to improve data writing performance.

codecov[bot] commented 1 year ago

Codecov Report

Merging #720 (840d738) into main (a39fdca) will increase coverage by 0.0%. The diff coverage is 100.0%.

@@          Coverage Diff          @@
##            main    #720   +/-   ##
=====================================
  Coverage   95.0%   95.0%           
=====================================
  Files         59      59           
  Lines       5998    6004    +6     
=====================================
+ Hits        5702    5708    +6     
  Misses       296     296           
Impacted Files Coverage Δ
pyam/core.py 95.4% <100.0%> (+<0.1%) :arrow_up:
tests/test_core.py 100.0% <100.0%> (ø)
tests/test_time.py 100.0% <100.0%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

danielhuppmann commented 1 year ago

Performance tests indicate that this was a very bad idea... But I pulled out some minor improvements into #723