Integration Testing

[troyraen]

Initial thoughts towards automated integration testing :

• GitHub Action
• Build a testing instance of the broker (run the setup script)
• ~Push some alerts through using the consumer simulator~
• Load some alerts from file using broker_utils.data_utils.decode_alert(filename)
• Publish the alerts to the instance's alerts topic using broker_utils.gcp_utils.publish_pubsub(topic, alert_dict)
• Check the results using tools in either broker-utils or pgb-utils. Query BigQuery tables, pull Pub/Sub streams, download avro files (plot cutouts).

~If it's helpful, there is a VM called stream-looper that is always on and already has the consumer simulator installed. (It publishes the ztf-loop Pub/Sub stream). It can easily be utilized to run consumer simulators for testing broker instances. I sometimes do this when I run these tests manually.~

As discussed offline with @djperrefort it makes more sense to select a specific set of alerts and load them from file, rather than using the consumer simulator. I have updated the process above to reflect this.

[troyraen]

@djperrefort Here is the python code to load a single alert from file and publish it to pubsub

from broker_utils import data_utils, gcp_utils, schema_maps

# broker instance keywords
survey = 'ztf'
testid = 'integrationtest'

# load alert from file to dict
# here's a convenient test file that's in the repo
fname = 'version_tracking/v0.6.0/ZTF18acegotq.1680242110915010008.ztf_20210808_programid1.avro'
# avoid a bug; Issue #88
drop_cutouts = True
schema_map = schema_maps.load_schema_map(survey, testid)  # needed for drop_cutouts=True
alert_dict = data_utils.decode_alert(
    fname, return_as='dict', drop_cutouts=drop_cutouts, schema_map=schema_map
)

# publish to Pub/Sub
# topic = f'{survey}-alerts-{testid}'  # use this for integration testing
topic = 'test'  # I use this topic for random tests
gcp_utils.publish_pubsub(topic, alert_dict)

If you need to load a schema_map but haven't set up a testing instance yet, just set testid=False in that call. That will load the schema map from the production instance.

Tagging #88 since the above code has to avoid it.

[troyraen]

I will collect some alerts that I know will make it through all the filters/classifiers/fitters, plus some that will not, so that we have a good set to test from.

@djperrefort should I add them somewhere in the issues/integration_workflow branch you created?

Also, if you haven't seen it yet and just because you'll probably be interested: the new metadata table in BigQuery makes it easy to find alerts that do/do not pass through different components/filters/classifiers/etc. The GCS bucket and file names are right there as well, which makes it easy to download the files from storage. Tools in broker_utils.gcp_utils facilitate the db query and file downloads.

[troyraen]

I pulled 50 alerts and stashed them here. The README gives a little info.