search

actigraph / agcounts

Code for the technical report on the ActiLife counts algorithm.
GNU General Public License v3.0
32 stars 13 forks source link

Run data in chunks? #10

Closed muschellij2 closed 1 year ago

muschellij2 commented 2 years ago

So I have 8 days worth of 80Hz data, and it would be useful for memory-saving to be able to run some of this in chunks, but it's unclear how to break up the data to get the same result.

The recursive nature of the filtering doesn't make it easy for me to determine how to do it, including how the epoch/sample rate combination would affect how to chunk the data.

Is there any clear way to perform this operation, even if you need to pass in some of the previous data into the next chunk?

rouzbeh commented 2 years ago

Have you seen this https://stackoverflow.com/questions/40483518/how-to-real-time-filter-with-scipy-and-lfilter ?

You can get the internal state of the filter when calling lfilter and pass it back when running the next chunk.

muschellij2 commented 2 years ago

I have not, but that seems promising, but I'd likely only want to read chunks of the CSV in at a time and then run that chunk and then aggregate the chunk.