search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

Read s3 data without IAM role #44

Closed mrocklin closed 4 years ago

mrocklin commented 4 years ago

I'm on a machine with a AWS credentials and the quickstart isn't working for me. I get the following error

OSError: s3://nyc-tlc/trip data/yellow_tripdata_2019-*.csv resolved to no files

When I add the keyword storage_options={"anon": True} to the read_csv call then things work ok. Historically this was because our worker tasks didn't have sufficient permissions to read public data. Is this still the case? cc @necaris

necaris commented 4 years ago

@mrocklin can you point me at the task / cluster in question? On beta or sandbox?

mrocklin commented 4 years ago

mrocklin/win-play in beta

On Mon, Aug 10, 2020 at 2:24 PM Rami Chowdhury notifications@github.com wrote:

@mrocklin https://github.com/mrocklin can you point me at the task / cluster in question? On beta or sandbox?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coiled/coiled-issues/issues/44#issuecomment-671597254, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTDMC2RRSXW4QBXRM63SABQPJANCNFSM4P2LT2IA .

necaris commented 4 years ago

The task role assigned to those workers (and the scheduler) is https://console.aws.amazon.com/iam/home#/roles/mrocklin which has the S3 policy attached :-( so I'm not sure what's going on.

dantheman39 commented 4 years ago

It seems like this has been resolved, closing. Feel free to reopen.