search

NREL / flasc

A rich floris-driven suite for SCADA analysis
https://nrel.github.io/flasc/
BSD 3-Clause "New" or "Revised" License
31 stars 18 forks source link

Feature/port to polars #81

Closed paulf81 closed 1 year ago

paulf81 commented 1 year ago

Not ready to be merged

Feature or improvement description This pull request, currently very much just a function-by-function work in progress to replace pandas code with polars to facilitate faster computations, allow multiprocessing and improve memory usage.

Since this is a going through, also taking the opportunity to make some side-changes to sqlalchemy linked code, and move some os -> pathlib, but these are mainly minor changes.

Related issue, if one exists Issue #80

Impacted areas of the software Will be a fairly pervasive change.

paulf81 commented 1 year ago

hi @Bartdoekemeijer , @ejsimley and @misi9170 , just wanted to share I merged develop back into port_to_polars, and then I added a notebook where I'm comparing some of the in-development polars methods to the pandas methods (examples_smarteole/06b_wake_steering_compare_with_polars.ipynb)

https://github.com/NREL/flasc/blob/feature/port_to_polars/examples_smarteole/06b_wake_steering_compare_with_polars.ipynb

I think good news is the polars code is running much faster, and is mostly giving the same answer for energy ratio, but some of the bootstrapping answers have changed a bit and so I'll dig in to it to check but, wanted to share what I have so far in case you'd like to have a look

Bartdoekemeijer commented 1 year ago

hi @Bartdoekemeijer , @ejsimley and @misi9170 , just wanted to share I merged develop back into port_to_polars, and then I added a notebook where I'm comparing some of the in-development polars methods to the pandas methods (examples_smarteole/06b_wake_steering_compare_with_polars.ipynb)

https://github.com/NREL/flasc/blob/feature/port_to_polars/examples_smarteole/06b_wake_steering_compare_with_polars.ipynb

I think good news is the polars code is running much faster, and is mostly giving the same answer for energy ratio, but some of the bootstrapping answers have changed a bit and so I'll dig in to it to check but, wanted to share what I have so far in case you'd like to have a look

Thanks for doing this! The speed-up sounds great. It would be good to understand why the non-bootstrapped energy ratios differ between polars and pandas, even if it's only by a small number. Any idea?

paulf81 commented 1 year ago

I agree, I'll try to track this down soon

paulf81 commented 1 year ago

Going to close this pull request in favor of the one built off of latest develop branch (#97 )