B612-Asteroid-Institute / precovery

Fast precovery of small body observations at scale
BSD 3-Clause "New" or "Revised" License
5 stars 2 forks source link

Rewrite sourcecatalog ingestion for flexibility #68

Closed spenczar closed 1 year ago

spenczar commented 1 year ago

Deleted precovery.sourcecatalog.SourceExposure. Its functionality was purely internal to the sourcecatalog module, and it can be accomplished using a dictionary instead, which is less code weight.

Rewrote most of sourcecatalog. Instead of primarily operating on CSV files, it breaks that step into two phases. One phase is loading in a stream of SourceObservations. This could be from a CSV file, but could also be from a DataFrame, for example. The second phase is organizing the SourceObservations into chunks grouped by HEALPixel ("SourceFrames"), suitable for loading into the database. Those two phases are now contained in separate functions.