Read data in chunks

[dominik-weber-92]

Hey there, obviously it's not possible to read the BIOPAC file in chunks. Or if yes, how to do it? The code is not really self explanatory. Thanks!

[njvack]

It is... what are you looking to do?

An example of incrementally reading these files is here:

https://github.com/uwmadison-chm/bioread/blob/master/bioread/runners/acq2hdf5.py#L160

Or:

from bioread import reader as br
r = br.Reader.read_headers(acq_file)
streamer = r.stream()
for chunk in streamer:
    for data_buffer in chunk:
        print(buf.channel_slice)
        # Data is in buf.buffer, buf.channel_slice indexes you into an output slice

Basically, you can create a reader object, call stream() on it, and then use the result as an iterator to run through your file a chunk at a time.

Note that if your Biopac file is compressed, you can't do the same thing, you currently need to read a whole channel in at once as I haven't implemented streaming gzip reading.

[dominik-weber-92]

Hey Nate,

this is very helpful, I’ll try it. Thanks!

Best, Dominik

Von: Nate Vack notifications@github.com Gesendet: Dienstag, 14. April 2020 00:32 An: uwmadison-chm/bioread bioread@noreply.github.com Cc: Weber, Dominik dominik.weber@iis.fraunhofer.de; Author author@noreply.github.com Betreff: Re: [uwmadison-chm/bioread] Read data in chunks (#28)

It is... what are you looking to do?

An example of incrementally reading these files is here:

https://github.com/uwmadison-chm/bioread/blob/master/bioread/runners/acq2hdf5.py#L160

Or:

from bioread import reader as br

r = br.Reader.read_headers(acq_file)

streamer = r.stream()

for chunk in streamer:

for data_buffer in chunk:

    print(buf.channel_slice)

    # Data is in buf.buffer, buf.channel_slice indexes you into an output slice

Basically, you can create a reader object, call stream() on it, and then use the result as an iterator to run through your file a chunk at a time.

Note that if your Biopac file is compressed, you can't do the same thing, you currently need to read a whole channel in at once as I haven't implemented streaming gzip reading.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/uwmadison-chm/bioread/issues/28#issuecomment-613129271, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APDY2KBREEZ4LHQZGHMNYA3RMOHGBANCNFSM4MEYGSFA.

[njvack]

I'm closing this for now, feel free to reopen it if you keep having trouble.