Replacing pipeline/notebook containers to Batch/sageMaker in AWS module(Transition to AWS environment)

[swiri021]

Currently, we are using docker instance to deploy notebook and it could be replaced with AWS sageMaker, possibly, pipeline container can be deployed in ECS.

[swiri021]

For ECS deployment(Using AWS Fargate): https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html#ECS_CLI_tutorial_fargate_iam_role

[swiri021]

• Pipeline controller need S3 update function
• https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html

[swiri021]

• Updated connection between S3 and snakemake, but has an issue in feature extaction pipeline.
• SageMaker is fully connected with S3 buckets and github.

[swiri021]

• Pipelines do not need to specify output folder if we go to s3 bucket, think about updating for this point.
• Currently pipeline output folder should not have directory path, just use the name only

[swiri021]

• Need to implement AWS batch for pipeline

[swiri021]

• Need CloudFormation management for AWS resources

[swiri021]

• Shared volume(EFS) + array job type in AWS batch could be used for sequential task execution for data pipeline.