Two behaviors of the preservation service sets up a race condition that can make an update to an AIP cause data to be lost:
When the preservation service has finished serializing its output bags, it copies the bag files to a storage directory for long-term storage and the NERDm record is sent to the repository ingest service. The storage directory in practice is a staging area where files eventually get replicated to AWS-S3; this replication can take 15-30 minutes.
When the preservation service is directed to make an update to a dataset, the service will try to pull in the last published headbag as the seed for the update; to do this, it queries the public distribution service, which in turn looks for the bag in AWS-S3 storage.
If the update comes before the previously saved bags have migrated to S3, then the preservation will either fail to pull over a head bag, or pull over the wrong one (i.e. not the latest). This means that the data (or updated metadata) from that bag will not be represented in the update and will be effectively lost. When this has occurred in the past, all of the data was lost. Further, the output version and sequence number would be wrong, and the new output bags would overwrite previously saved ones.
(In actuality, the data has been recoverable, thanks to versioning being turned on in the S3 bucket.)
This PR addresses this bug with three major changes:
The code for retrieving the previous bag and setting it up as a seed was reworked a bit; in particular, it will first look for the latest bag from the storage directory used for staging to S3. Thus, even if the data has not been migrated, yet, we still get the latest version.
A check is done on the output bag (pre-splitting) that ensures that all distributions listed in the NERDm metadata with a distribution service download URL are available--either stored in that bag itself or is locate-able in a previous bag.
The preservation service cannot overwrite a bag in the storage directory with the same name.
Two behaviors of the preservation service sets up a race condition that can make an update to an AIP cause data to be lost:
If the update comes before the previously saved bags have migrated to S3, then the preservation will either fail to pull over a head bag, or pull over the wrong one (i.e. not the latest). This means that the data (or updated metadata) from that bag will not be represented in the update and will be effectively lost. When this has occurred in the past, all of the data was lost. Further, the output version and sequence number would be wrong, and the new output bags would overwrite previously saved ones.
(In actuality, the data has been recoverable, thanks to versioning being turned on in the S3 bucket.)
This PR addresses this bug with three major changes: