terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

Fix recovery-spec test to use only the autoloaded assets in the jobSpec #3568

Closed busma13 closed 5 months ago

busma13 commented 5 months ago

This PR makes the following changes:

These changes prevent the recovery-spec from trying to use assets placed in the teraslice.assets_directory by other tests running in parallel, that have not finished uploading to ES. Ref: #3567

busma13 commented 5 months ago

This solution works, but I want to look into why we can't just use the array of asset names for jobSpec.assets. The initializeTestExecution calls JobValidator.validateConfig(). The JobValidator uses the OperationLoader which expects asset IDs, not assets names. It seems like jobConfig and ExecutionConfig types might be getting muddled.

busma13 commented 5 months ago

I dug into this and here's what I found out:

I think the changes I have made in the PR solve the problem sufficiently.