terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

move s3 asset storage related fields to teraslice section of config #3601

Closed busma13 closed 4 months ago

busma13 commented 5 months ago

When the external asset storage feature was implemented, all the fields in the config were put in the terafoundation section, but they are specific to teraslice so should be in that section.

To move them to teraslice we need to move the validation function. Unfortunately it requires access to terafoundation.connectors so the inputs to validator functions will need to allow for the whole config to be passed in.

godber commented 5 months ago

Lets deprecate the asset_storage_* keys in terafoundation. at the same time as adding them to teraslice. That way we're not making a breaking change to the terafoundation.yml. Also make sure to add comments that allow us to find and remove the asset_storage keys that need to be removed in 2.0.