e2e tests shared asset_directory can cause test failure.

[busma13]

Adding functionality to store assets in an external storage like S3 instead of in Elasticsearch (see: #3563) has surfaced a race condition in e2e tests.

• simple-spec saves assets to the teraslice.assets_directory cache.
• recovery-spec gets the array of asset IDs needed to create a jobSpec from the list of IDs in the teraslice.assets_directory cache.
• Since jest runs the two spec files in parallel, simple-spec can be adding an asset while recovery-spec is reading all the assets in the directory.
• The additional time required to save an asset in S3 slows the post('/assets') route enough to make obvious that an asset can be added to the local cache in teraslice.assets_directory, but not yet be stored in Elasticsearch.
• If recovery-spec gets an ID that has just been added to the cache, it will add it to jobSpec.assets and then run validation on jobSpec. If the validation check for the presence of the asset happens before it has been saved to ES by simple-spec the test will fail with an error: Failure to get assets, caused by TSError: No assets found for "2909ec5fd38466cf6276cc14ede25096f1f34ee9"

[busma13]

There are a few ways that we may be able to resolve this issue:

• Use jests test-sequencer to ensure that recovery-spec runs first.
• Change how the jobSpec in recovery-spec gets the list of IDs for assets it needs.
• Add a function to the teraslice-harness that will create a jobSpec that does not need to be modified by the test?

[busma13]

I decided the best approach was to add a function to the teraslice-harness that returns an array of IDs belonging to the "base assets" that the e2e tests autoload. recover-spec calls this function and assigns the return value to jobSpec.assets, ensuring that job validation only checks the assets that were autoloaded.