S3 Bucket and AWS credentials for vault file upload to s3

[dylanmartin]

Task Description

The coinstac api server needs to be able to pipe file uploads to S3. For this we'll need:

• A secure S3 bucket for the files to go in
• AWS credentials for the AWS SDK to use to connect
• • AWS_ACCESS_KEY_ID
• • AWS_SECRET_ACCESS_KEY.

[segomath]

@dylanmartin I need some additional information. s3-bucket

• s3 bucket name ( suggest based on the usage )
• do you need the content to be encrypted?
• do you need versioning enabled?
• any other configurations needed? AWS credentials
• is this IAM user non-human? mainly for the application usage with out console access?
• what services the user need access to?
• preferred name for the IAM user

[praeducer]

@segomath I'd like to own this task if you can provide me any policies I need to follow. Dylan and I are on a call now.

[praeducer]

I'd like to think of an alternate architecture then the one proposed. As Dylan works on this, I'll think through some more modern patterns we can start following. With the little I know so far, these services and patterns come to mind:

• https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-api-gateway.html
• https://docs.aws.amazon.com/step-functions/latest/dg/connect-lambda.html
• https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html

Dylan agrees this sounds like a better direction. In general, we need to get away from managing all of these servers and building software like modern cloud solutions don't exist. The kind of services above will save an enormous amount of developer time and be easier/more fun to use.

[segomath]

@praeducer Sure. We don't have TReNDS level IAM policies defined yet. As I mentioned in the chat, COINS team is mainly using AWS resources as of now. So most of the policies are customized for COINS team. You may want to define new policies for COINSTAC and add the users accordingly. You can refer COINS related resources for naming convention etc.

[praeducer]

ty!

[praeducer]

As we continue to containerize our services, we may also want to consider some pattern like this https://aws.amazon.com/blogs/containers/introducing-aws-step-functions-integration-with-amazon-eks/

[praeducer]

Currently writing a bucket policy for the COINSTAC API service (a Node package). It will allow this custom application service, which runs on a virtual machine, to talk to S3, a cloud-native web service.

It will be allowed to:

• Access one specific bucket.
• Upload objects to the bucket.
• Download objects from the bucket.

[praeducer]

Need to verify SSL encryption end-to-end.

[praeducer]

@segomath Do you know what regulations we need to follow in the cloud for COINSTAC? Trying to decide what data to encrypt and when to encrypt it.

[praeducer]

Server-side encryption is on for the S3 bucket vis S3-managed keys.

[praeducer]

Strongly consider using the key management service or the secrets manager instead of env variables on the VM.

[praeducer]

When complete, we want to do some kind of security audit. There is likely some kind of penetration testing we can do.

[praeducer]

Need to secure the results data from users too.

[segomath]

@segomath Do you know what regulations we need to follow in the cloud for COINSTAC? Trying to decide what data to encrypt and when to encrypt it.

@praeducer In case of S3, It is recommended to encrypt "Data at rest" if the data has PHI. It would be good to check the HIPAA compliance requirements while deciding on any design/changes.