Implement copy data/metadata functionality

[JoCaudwell]

User Story

So that the data is stored in the Data Lake, as a Data Maintainer, I want the data lake to copy data and metadata from it's current S3 source to the Data Lake at the end of the import process.

Acceptance Criteria

• [ ] Given any list of dataset and metadata, when import is triggered, then the data lake copies the data and metadata across to data/metadata lake s3 bucket.

Additional context

Potential tasks from planning discussion

• [x] Create lambda function to use S3 Batch Operations with list of files to be transferred, as last step in Step function
• [x] Add this step in Step function to infrastructure using CDK
• [x] Store result (specify) of S3 Batch Operation (PUT operation) in target S3 bucket, but don't do anything with results (yet).
• [x] Add required S3 Batch Operations permissions
• [x] implement logging in the task
• [x] write functional and logging tests for import dataset task

Definition of Ready

• [x] This story is ready to work on, according to the team's definition

Definition of Done

• [ ] This story is done, according to the team's definition

Subtasks

• Test AWS Batch Copy and confirm we want to use it

[imincik]

@MitchellPaff , see

S3 batch copy IAM role policy template and IAM trust policy example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectTagging",
                "s3:PutObjectLegalHold",
                "s3:PutObjectRetention",
                "s3:GetBucketObjectLockConfiguration"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::imincik-test2/*"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::{{SourceBucket}}/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::imincik-test2/manifest.csv"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::imincik-test2/aaa/*"
            ]
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "batchoperations.s3.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

[billgeo]

Tested by importing multiple files and works as expected. It would be good to make sure there is at least one asset listed and I believe there is a ticket already open for that.