Closed ahincks closed 4 years ago
So g3_reader
and so3g.hk.getdata
have some implementation differences that I think could be streamlined. I actually think many of the changes should happen in so3g
, so I'd be interested to hear what @mhasself thinks of the following.
In general terms I think so3g.hk.getdata
should be capable of optionally reading from a database of the same architecture as that created by g3_file_scanner
. One way to do this would be to pass a DB cursor object when initialising HKArchive
instead of a list of _ChannelGroup
objects. Then the HKArchive._get_groups
could query the DB appropriately; of course if a list of _ChannelGroup
objects is passed, it would work exactly as it is now. If I am understanding Matthew's code, with this implemented get_fields()
and get_data()
could work essentially as they are currently written.
(For the above solution, though, we would have to add byte_offset
information to Brian's DB architecture.)
Then, we could simply get g3_reader
to call get_fields()
and get_data()
from so3g.hk.getdata
and pass on the results. (However, we might want to consider having the g3_reader
cache data as is currently done.)
Other points to consider:
MAX_POINTS
could be implemented by having an optional argument in so3g.hk.getdata.get_data()`min_stride
? Probably by having so3g.hk.getdata._get_groups()
know how to query the DB properly for the right downsampled data file. (If the user is relying lists of _ChannelGroup
, on the other hand, it would be up to the user to supply the downsampled data files when using an HKArchiveScanner
.Having thought about this some more, I think an even better solution is one that also tackles the question of creating and keeping track of downsampled files. See Issue https://github.com/simonsobs/sisock/issues/37#issuecomment-495954785.
This was resolved by #46.
I recommend that we use the so3g library in the
g3_reader
data node server.