DART runner writes incorrect tag values in workflow batch mode

[diamondap]

From Josh Westguard at UMD:

I have been troubleshooting some issues I noticed with the tags for some of the bags submitted with dart-runner's batch mode, and I think I have discovered a bug. I suspect that concurrency may play a role. I've been able to reproduce the error with test data, and so I'm attaching that.

The effect is that dart-runner creates bags with tags from the wrong lines of the spreadsheet. It appears to consistently create correctly named bag.tar files, containing the correct payload, so the error seems to be limited to the metadata.

I hope this is helpful. Feel free to reach out if you have questions. I'm reasonably certain that this is a bug and not human error, though one should never rule out the latter.

Josh

---

Steps to reproduce -- using the attached bug-test.tar.gz and a workflow configured to bag locally with no upload, run these commands:

(1) remove the files from bug-test/output

find bug-test/output -type f -delete

(2) re-create the bags using the attached tag file

./dart-runner --workflow=workflows/prod-no-upload.json --batch=bug-test/input/tags.csv --delete=false --concurrency=2 --output-dir=bug-test/output/bags

(3) extract the tags from the tar files into the directory bug-test/output/tags

for p in $(ls bug-test/output/bags/*); do f=$(basename $p); tar --extract --verbose --file=$p --directory=bug-test/output/tags --wildcards --no-wildcards-match-slash ${f%.tar}/*.txt; done

(4) observe that the tags in aptrust-info and bag-info are not from the correct spreadsheet rows.

See attachments. We have reproduced this, and it is a bug.

[diamondap]

Files to reproduce:

bug-test.tar.gz prod-no-upload.json.zip

[diamondap]

Summary of the effect of this bug:

• You run a workflow against a batch of directories as listed in a CSV file.
• Each entry in the CSV file includes tag values that should be inserted into the bag described on that line.
• Let's assume your CSV file defines bags A, B, and C.
• A mishandled data race causes tag values from bags B and C to be written into A. Bag B may also contain tag values that should have been written into C.
• The bags are valid, and you won't know they contain the wrong tag values until you open them up and look inside.

[diamondap]

This bug is fixed in release 0.96-beta. To download the latest version of DART Runner, go to https://aptrust.github.io/dart-docs/users/dart_runner/#downloads