tensorboard not working with S3

[gourav-sg]

Environment information (required)

Mac 10.14.6 Tensorboard 2.0.0 botocore 1.19.1 boto3 1.16.0 tensorflow 2.0.0 mkl_py37hda344b4_0
tensorflow-base 2.0.0 mkl_py37h66b1bf0_0
tensorflow-estimator 2.0.0 pyh2649769_0

AWS credentials are available in the ~/.aws/credentails file in profile default

Issue description

AWS_LOG_LEVEL=1 AWS_DEFAULT_PROFILE=default S3_REGION=eu-west-1 tensorboard --logdir s3://gourav-bucket/gourav-data/tensorboard/

gives the following errors:

Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.0.0 at http://localhost:6006/ (Press CTRL+C to quit)
2020-10-24 23:13:51.056759: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
2020-10-24 23:13:51.056861: E tensorflow/core/platform/s3/aws_logging.cc:60] Http request to retrieve credentials failed
2020-10-24 23:13:51.056910: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 0 ms before attempting again.
2020-10-24 23:13:52.057796: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
2020-10-24 23:13:52.057882: E tensorflow/core/platform/s3/aws_logging.cc:60] Http request to retrieve credentials failed
2020-10-24 23:13:52.057923: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 2000 ms before attempting again.
2020-10-24 23:13:55.062484: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
2020-10-24 23:13:55.062540: E tensorflow/core/platform/s3/aws_logging.cc:60] Http request to retrieve credentials failed
2020-10-24 23:13:55.062570: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 4000 ms before attempting again.

[nfelt]

Thanks for the report. TensorBoard's ability to read from S3 is a community supported contribution (we don't ourselves test it or verify it works). If you'd like to debug further, I'd recommend first confirming that you can access the bucket using plain python and the boto3 module, e.g. the following

import boto3

client = boto3.client("s3")
print(client.list_objects(Bucket="gourav-bucket", Prefix="gourav-data/tensorboard/", Delimiter="/")

[gourav-sg]

Hi, I tested and tried this again, and boto3 and s3 cli all work, except for tensorboard.

I am using SAML and there is aws credentials file, and am not explicitly entering my credentials using environment variables, but I think that should be fine.

Regards, Gourav

[jcreinhold]

@gourav-sg I use SAML as well and was able to get TensorBoard to work by setting AWS_DEFAULT_PROFILE=saml. FYI for other people who see this issue or if this is still an issue for you.

[gourav-sg]

Hi, it does not read data from S3 for sure. I am reading the data from S3 via an EC2 instance so no need of SAML I guess.

Let me try this once again and see what happens.

Regards, Gourav Sengupta

On Tue, Aug 10, 2021 at 3:29 PM Jacob Reinhold @.***> wrote:

@gourav-sg https://github.com/gourav-sg I use SAML to log into AWS as well and was able to get TensorBoard to work by setting AWS_DEFAULT_PROFILE=saml. FYI for other people who see this issue or if this is still an issue for you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tensorflow/tensorboard/issues/4255#issuecomment-896078763, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJZLQ576GDKW3WBHVD4KDTT4EZUHANCNFSM4S54EI4A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[omrishiv]

Don't mean to resurrect an old thread, but since I spent hours on this today; I'll chime in:

Had to install tensorflow-io to get S3 to work, but it didn't look like it could see any files. It wasn't until I added 2 envvars: AWS_REGION=us-east-1 and S3_ENDPOINT=https://s3.amazonaws.com that I was able to finally use the ec2 node role to pull S3 files.

Hoping this may help someone (probably myself) in the future.

[ernygati]

It wasn't until I added 2 envvars: AWS_REGION=us-east-1 and S3_ENDPOINT=https://s3.amazonaws.com that I was able to finally use the ec2 node role to pull Can you please give me more details of where to initialized env variables?

[gourav-sg]

Hi, Not quite sure, but I think that you can do that as part of bootstrap.

Regards, Gourav

On Fri, 28 Oct 2022, 07:56 Ernest Gatyatullin, @.***> wrote:

It wasn't until I added 2 envvars: AWS_REGION=us-east-1 and S3_ENDPOINT= https://s3.amazonaws.com that I was able to finally use the ec2 node role to pull Can you please give me more details of where to initialized env variables?

— Reply to this email directly, view it on GitHub https://github.com/tensorflow/tensorboard/issues/4255#issuecomment-1294544312, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJZLQ5ID2ZA2GNWCTWW56LWFN2IBANCNFSM4S54EI4A . You are receiving this because you were mentioned.Message ID: @.***>