search

DistributedScience / Distributed-CellProfiler

Run encapsulated docker containers with CellProfiler in the Amazon Web Services infrastructure.
https://distributedscience.github.io/Distributed-CellProfiler/
Other
36 stars 24 forks source link

Fix credential handling #145

Closed ErinWeisbart closed 1 week ago

ErinWeisbart commented 1 year ago

Still a bit of a mess handling both roles and users. I believe what is currently pulled to run-worker.sh in master handles roles nicely but doesn't play well with user credentials.

gareth-rogers-healx commented 3 months ago

@ErinWeisbart I have built the latest version into a Docker container locally and run using an IAM role which is assigned to the task definition. That worked no problem, the only change I have made is

stdbuf -o0 s3fs $AWS_BUCKET /home/ubuntu/bucket -o passwd_file=/credentials.txt -o ecs

where I added the -o ecs option to s3fs. I found -o iam_role also worked.

I'm not sure what combination is the most correct, possibly both.

If you have a preference I can create a PR.

ErinWeisbart commented 3 months ago

Thanks @gareth-rogers-healx! Always happy to have a PR! Can you clarify, did you check that -o ecs and -o iam_role both work with IAM roles and user credentials?

gareth-rogers-healx commented 3 months ago

Hi Erin, no I haven't tested using user credentials. I can test that too. To be more compatible with how we deploy code to AWS in Healx I've converted a lot of resource deployment to Terraform and need to stabilise my implementation first :)

Hopefully I'll get a chance to submit this and a couple of other things I think are fixes. I've got a data bucket and log bucket and I think there are cases where AWS_BUCKET and SOURCE_BUCKET aren't used correctly. However, it could be me not understanding how they should be used.