terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

Add support for storing assets in S3 #3563

Closed busma13 closed 7 months ago

busma13 commented 7 months ago

This PR makes the following changes:

Ref: #3561

godber commented 7 months ago

Is assetSearch called once on job startup or once per worker?

https://github.com/terascope/teraslice/blob/dad488eedd7230805abcb11289463e196834341f/packages/teraslice/src/lib/cluster/services/assets.ts#L219

sotojn commented 7 months ago

Is assetSearch called once on job startup or once per worker?

https://github.com/terascope/teraslice/blob/dad488eedd7230805abcb11289463e196834341f/packages/teraslice/src/lib/cluster/services/assets.ts#L219

assetsearch is called whenever you hit the v1/assets or the txt/assets endpoints. The only other code I see using those endpoints are inisde the teraslice-client-js

Also the only thing I see that utilizes the teraslice-client-js package is teraslice-cli.