notebook 6a: SummaryWriter Access Denied

[murphyk]

I cannot write to writer = SummaryWriter(f's3://pytorchtraining/pytorch_bigbatch/learning_worker{worker_rank}'), which is not surprising... However, it is not clear what I should change to write to my own bucket. (The credentials are stored in my laptop, but not in this jupyter instance.)

Following the demo.py at tensorboardX, I created the script below, which fails. I suggest adding a block that explains how to setup S3 writing, and how to modify the code (with a flag?) if the user cannot write to S3.

# demo.py

import torch
import torchvision.utils as vutils
import numpy as np
import torchvision.models as models
from torchvision import datasets
from tensorboardX import SummaryWriter

resnet18 = models.resnet18(False)
#writer = SummaryWriter()
#writer = SummaryWriter(f's3://pytorchtraining/pytorch_bigbatch/learning_worker1')
writer = SummaryWriter(f's3://murphyk/murphyk-bucket')
sample_rate = 44100
freqs = [262, 294, 330, 349, 392, 440, 440, 440, 440, 440, 440]

for n_iter in range(10):

    dummy_s1 = torch.rand(1)
    dummy_s2 = torch.rand(1)
    # data grouping by `slash`
    writer.add_scalar('data/scalar1', dummy_s1[0], n_iter)
    writer.add_scalar('data/scalar2', dummy_s2[0], n_iter)

    writer.add_scalars('data/scalar_group', {'xsinx': n_iter * np.sin(n_iter),
                                             'xcosx': n_iter * np.cos(n_iter),
                                             'arctanx': np.arctan(n_iter)}, n_iter)

    dummy_img = torch.rand(32, 3, 64, 64)  # output from network
    if n_iter % 10 == 0:
        x = vutils.make_grid(dummy_img, normalize=True, scale_each=True)
        writer.add_image('Image', x, n_iter)

        writer.add_text('Text', 'text logged at step:' + str(n_iter), n_iter)

        for name, param in resnet18.named_parameters():
            writer.add_histogram(name, param.clone().cpu().data.numpy(), n_iter)

        # needs tensorboard 0.4RC or later
        writer.add_pr_curve('xoxo', np.random.randint(2, size=100), np.random.rand(100), n_iter)

dataset = datasets.MNIST('mnist', train=False, download=True)
#images = dataset.test_data[:100].float()
#label = dataset.test_labels[:100]

images = dataset.data[:100].float()
label = dataset.targets[:100]

#features = images.view(100, 784)
#writer.add_embedding(features, metadata=label, label_img=images.unsqueeze(1))

# export scalar data to JSON for external processing
writer.export_scalars_to_json("./all_scalars.json")
writer.close()

[murphyk]

Note that notebook 6a says "If you don't have access to an S3 bucket, but would still like to train your model and review results, please visit Notebook 6b and Notebook 7 to see detailed examples of how you can do that." However, 6b just refers t itself, and does not explain how to skip S3, and 7 has nothing to do with S3.

[skirmer]

Notebook 6b doesn't include anything to do with S3, intentionally. If you don't have an AWS account with an S3 bucket, you'll need to speak to AWS about that - if you run Notebooks 6b and 7 you'll bypass the need to use any S3 resources. We don't assume users have AWS accounts, so we created 6b and 7 to help them still visualize the output.

If you do have your own S3 bucket, then visit the Credentials section of the platform and put in your creds as environment variables. Then you can access them in the notebook, and write to your own bucket. (Obviously, change the path.)