data-yaml / vivos

Versioned Interoperability to Velocitize Open Science

Apache License 2.0

0 stars 0 forks source link

pipe edp #34

Closed drernie closed 8 months ago

drernie commented 8 months ago

Add EDP to QuiltPipe https://github.com/QuiltSimon/edp-sdk

1. Add Constants

https://github.com/QuiltSimon/edp-sdk/blob/master/app.py

quilt_vpc_name = "sales-production"
edp_sns_arn = "arn:aws:sns:us-east-1:850787717197:sales-production-edp-EventBusTopic-jEVFPz6zu3ou"
image=ecs.ContainerImage.from_registry("850787717197.dkr.ecr.us-east-1.amazonaws.com/edp-container-registry:latest"),

Incorporate Trigger into Pipe

https://github.com/QuiltSimon/edp-sdk/blob/master/src/queue_trigger.py

Translate BatchFargateStack into TypeScript

https://github.com/QuiltSimon/edp-sdk/blob/master/app.py

use constants
use event_bus events
write sentinel file
test container access

drernie commented 8 months ago

Already added:

raw_bucket_arn = "arn:aws:s3:::quilt-sales-raw"
staging_bucket_arn = "arn:aws:s3:::quilt-sales-staging"

drernie commented 8 months ago

Also: read quilt.pipe.json -> write .pipe.json

drernie commented 8 months ago

Seems to work, modulo fixes required for https://github.com/QuiltSimon/edp-sdk/issues/3