search

ministryofjustice / analytics-platform

Parent repository for the MOJ Analytics Platform
MIT License
14 stars 1 forks source link

s3tools laggy/returns timeout errors #51

Closed RobinL closed 6 years ago

RobinL commented 6 years ago

s3tools::get_credentials() hangs. This is the core method within s3tools which retrieves aws creds as environment variables, and it is repeatedly called by most other public-facing methods in the s3tools API. It is therefore highly likely to be the root cause of problems with other parts of s3tools

Katherine B: s3tools::get_credentials() is taking a very long time to run

Vicky H: "I'm having trouble reading data from S3: running s3_tools::read_using() in R and it just hangs ...ah sorry ignore me. restarting the platform fixed it."

Ben F: "Yep, my S3 requests all seem to hang (although I can now connect to RStudio from dom1)."

RobinL (2018-08-28) : s3tools::get_credentials() hangs forever

RobinL commented 6 years ago

This seems to have been down to one of our kube2iam pods being in a crash loop. Calls to s3tools::get_credentials() were timing out in both R Studio as well as apps. This was intermittent because it was just one of the kube2iam servers that was affected.