simonsobs / sisock

Sisock ('saɪsɒk): streaming of Simons Obs. data over websockets for quicklook
Other
2 stars 0 forks source link

Use so3g getdata for reading g3 files #29

Closed ahincks closed 4 years ago

ahincks commented 5 years ago

I recommend that we use the so3g library in the g3_reader data node server.

ahincks commented 5 years ago

So g3_reader and so3g.hk.getdata have some implementation differences that I think could be streamlined. I actually think many of the changes should happen in so3g, so I'd be interested to hear what @mhasself thinks of the following.

In general terms I think so3g.hk.getdata should be capable of optionally reading from a database of the same architecture as that created by g3_file_scanner. One way to do this would be to pass a DB cursor object when initialising HKArchive instead of a list of _ChannelGroup objects. Then the HKArchive._get_groups could query the DB appropriately; of course if a list of _ChannelGroup objects is passed, it would work exactly as it is now. If I am understanding Matthew's code, with this implemented get_fields() and get_data() could work essentially as they are currently written.

(For the above solution, though, we would have to add byte_offset information to Brian's DB architecture.)

Then, we could simply get g3_reader to call get_fields() and get_data() from so3g.hk.getdata and pass on the results. (However, we might want to consider having the g3_reader cache data as is currently done.)

Other points to consider:

ahincks commented 5 years ago

Having thought about this some more, I think an even better solution is one that also tackles the question of creating and keeping track of downsampled files. See Issue https://github.com/simonsobs/sisock/issues/37#issuecomment-495954785.

BrianJKoopman commented 4 years ago

This was resolved by #46.