search

slaclab / pysmurf

Other
2 stars 9 forks source link

Add set/get for MaxFileSize to allow splitting long streaming acquisitions over multiple files based on file size. #760

Closed swh76 closed 1 year ago

swh76 commented 1 year ago

Description

Adds new set_max_file_size and get_max_file_size to restore support for splitting streamed data over multiple files of a fixed size. If set to zero, streams to the same file indefinitely. Otherwise if nonzero, streams to the set value in number of bytes. E.g. set_max_file_size(2000000) will set the file partition size to 20MB per file. Files are split by appending an incrementing integer on the end of the usual ctime taggedfile name ; e.g. .1, .2, .3, etc.. Addresses issue #759.

Also removes datafile glob in read_stream_data and added logic to find the mask for rolling data files (which only gets written to disk once for MaxFileSize!=0).

swh76 commented 1 year ago

@jlashner right now not doing anything smart with rotating files, just adding the handles to do it. Your comment reminds me though there's a bug in reading rolling data because of the mask lookup. Just committed a fix for that.