search

ezmsg-org / ezmsg

Pure-Python DAG-based high-performance SHM-backed pub-sub and multi-processing pattern
https://ezmsg.readthedocs.io/en/latest/
MIT License
9 stars 5 forks source link

Add AdaptiveStandardScaler #65

Closed cboulay closed 7 months ago

cboulay commented 7 months ago

This branch uses the Generator pattern outlined in #55.

The implementation is based on River's AdaptiveStandardScaler. In fact, 2 implementations are provided and tested. The first, scaler, uses river directly (import is inside scaler function). The second, scaler_np, uses only numpy.

The first is about twice as fast for single-channel data but scaler_np is at least twice as faster for multi-channel data.

Either implementation is faster than the current EWM (at least in a 62-channel 500 Hz pipeline I was testing) and requires less memory.

cboulay commented 7 months ago

I think I would like to see @consumer moved out of the example pattern and directly into the core ezmsg before this is merged, then I can clean that out of the files in this PR.

griffinmilsap commented 7 months ago

At this point, I think I'm happy to add the generator utils to core.

cboulay commented 7 months ago

Thanks for the changes to the generator code! I'll rebase and recreate this PR. I don't have time at the moment but I will get to it before Monday.