search

askap-vast / vast-pipeline

This repository holds the code of the Radio Transient detection pipeline for the VAST project.
https://vast-survey.org/vast-pipeline/
MIT License
7 stars 3 forks source link

make Image and Measurement creation atomic together #648

Closed marxide closed 2 years ago

marxide commented 2 years ago

When ingesting inputs for a pipeline run, an Image object is created and saved to the database, then the catalogue is read, its contents are saved to the database as Measurement objects, and a copy of the catalogue is saved to the filesystem in a parquet file.

This PR puts the Image and Measurement object creation in an atomic transaction, i.e. if one fails, all operations conducted during the transaction are undone. If the catalogue read fails, the Image will be deleted from the database.

Fixes #647

ajstewart commented 2 years ago

Is the sky region, if it was a new one, also dumped if the image fails? Can't quite see it from that changed code, though maybe it's not even created there.

Is this something that should be included in testing?

marxide commented 2 years ago

Is the sky region, if it was a new one, also dumped if the image fails? Can't quite see it from that changed code, though maybe it's not even created there.

The sky regions are retrieved/created along with the image, so it should be within the same atomic transaction, just buried elsewhere.

Is this something that should be included in testing?

Good point. I've added one in 05adcf7. While I was there I also added a tearDown for RunPipelineTest that cleans up the run products so the pipeline-run folders are back in their original state after the tests are complete.