ooni / data

OONI Data CLI and Pipeline v5
https://docs.ooni.org/data
8 stars 4 forks source link

Come up with better pattern for handling composed/nested attributes of observations probe_meta, measurement_meta and processing_meta #77

Closed hellais closed 2 weeks ago

hellais commented 2 months ago

We need to do this in order to obtain the correct offsets into the queried columns. Basically probe_meta, measurement_meta and processing_meta are class attributes that are composed into the dataclass, however in the database they need to be stored flat, since nesting is not desirable.

What we are doing here is figuring out how to construct the nested class in order by manually recomputing the offsets of the returned queries. If we had an ORM this might be all avoided and even without it there is probably a better pattern.

hellais commented 2 weeks ago

This was done in #74