hellais
commented
4 months ago Leaving here some links following our discussion.
You will have to adjust the models to make use of the new table_model decorator and use composition instead of inheritance for the meta fields. See: https://github.com/ooni/data/blob/main/oonidata/src/oonidata/models/observations.py#L344 as an example.
What was already missing and needs to be added is support for your test inside of the transforms: https://github.com/ooni/data/blob/main/oonipipeline/src/oonipipeline/transforms/observations.py#L27
To see how this is actually used, check the temporal activity: https://github.com/ooni/data/blob/main/oonipipeline/src/oonipipeline/temporal/activities/observations.py
It's also worth reading the docs of temporal to understand what you should do and not do inside of an activity and workflow.
If you want your tables to be created automatically via the --create-tables flag (you probably do), you should also add your table to this list: https://github.com/ooni/data/blob/main/oonipipeline/src/oonipipeline/db/create_tables.py#L146
Finally, you mentioned interest in making use of scheduled operations. You can see an example of how to set this up here: https://github.com/ooni/data/blob/main/oonipipeline/src/oonipipeline/temporal/workflows.py#L208, though note that this is currently not used in production and I haven't tested it heavily.
work in progress; pushing for discussion on the data modelling and needed granularity.