Open Veldhoen opened 4 months ago
Instead of tarring the worker's output before uploading to S3, files are added separately with prefixes to maintain the directory structure. This mostly relies on a proposed update in DANE core; https://github.com/CLARIAH/DANE/tree/23-update-s3_util-to-allow-for-untarred-uploading
poetry install
./config/config.yl
.
BASE_MOUNT
PATHS
S3_endpoint_URL
TRANSFER_ON_COMPLETION
poetry run python worker.py --run-test-file
docker build -t dane-video-segmentation-worker .
docker-compose up
TAR_OUTPUT: false
TAR_OUTPUT: true
Bonus points: check out the DANE PR too (https://github.com/CLARIAH/DANE/tree/23-update-s3_util-to-allow-for-untarred-uploading)
Instead of tarring the worker's output before uploading to S3, files are added separately with prefixes to maintain the directory structure. This mostly relies on a proposed update in DANE core; https://github.com/CLARIAH/DANE/tree/23-update-s3_util-to-allow-for-untarred-uploading
poetry install./config/config.ylto the root of the repo (.) and adjust the necessary settings:BASE_MOUNTandPATHS,S3_endpoint_URL,TRANSFER_ON_COMPLETIONpoetry run python worker.py --run-test-filedocker build -t dane-video-segmentation-worker ../config/config.yl(setS3_endpoint_URLandTRANSFER_ON_COMPLETION)docker-compose upTAR_OUTPUT: falseandTAR_OUTPUT: trueand confirm that the output is transferred in the appropriate formatBonus points: check out the DANE PR too (https://github.com/CLARIAH/DANE/tree/23-update-s3_util-to-allow-for-untarred-uploading)