Archive Cleanup

[guynir42]

This PR makes two different setups for communicating with the archive during tests.

In the local config method, we use local_read_dir and local_write_dir to define direct folder access to the archive, as you'd do when you were using a local (or mounted) folder to contain the archive storage.

In the dockerized config method, we use the archive as a server, where archive_url: http://archive:8080/ is used to communicate with the archive, as you would when, e.g., launching the pipeline on a cloud server or on NERSC. In this case we will re-introduce the environmental variable SEECHANGE_TEST_ARCHIVE_DIR to the docker compose file, so that tests can go to the actual storage and see what is going on in the archive when stuff gets added/deleted there.

In addition, I would also like to verify that

• [x] the delete_from_disk_and_database(remove_downstream_data=True) actually removes the downstream data from the archive, not just from local storage, and
• [x] that the archive is empty at the end of the test run, just like the temp folder should be.

[guynir42]

@rknop There's a test failure that is related to an issue that I've solved in PR #190 and everything else is passing.

[guynir42]

@rknop These files used to exist on NERSC. They were also hard-coded but that didn't help.

Here's the folder: https://portal.nersc.gov/cfs/m2218/decat/decat/templatecache/

It contains several other files, but the specific one we used as a reference for DECam is gone.

Please check if you can re-upload it there. The files are in the tens-of-megabytes and I don't see a reason to add them to the repo when we are already downloading so many other things on the fly.