S3 Media Browser

[nelsonic]

Can we view all the images in a given S3 Bucket as a list of thumbnails?

Background / Story

In a client app, we had a report of "missing" images ... We know that the images are all on S3 but attempting to view them through the AWS S3 console is useless because all it shows is the name+type of file but not the actual image. So we have have a bucket full of images but no way to view them all in order to determine if the image is in fact in the bucket ... 😞

Todo

• [ ] Find an open source script that displays the images in a given S3 Bucket

We had a look at https://github.com/awslabs/aws-js-s3-explorer which looks good, But it doesn't show the images/thumbnails ... So we may need to write a lambda function or Elixir program to do this ...

This might be a good starting point: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html

[yousefvand]

If the client app is a desktop app one solution is mounding s3 bucket as local drive. It has it pros and cons. Let me know if this could be considered as an option.

[nelsonic]

Hi @yousefvand mounting the S3 bucket as a local drive could be a good solution. 👍 We are hoping for something web-based so that the person viewing the contents (media/photos) of the S3 bucket does not need to have the AWS secret/keys and can simply scroll through the thumbnails of the photos that have been uploaded. ☁️

[yousefvand]

If you want anyone can access bucket content either solution (API usage or mounting) needs AllowPublicRead to bucket. This is a sample JS solution and this is a bash script to automate mounting on a Debian machine (did not test).

[RobStallion]

@nelsonic It does seem like that example would be exactly what you described in this issue based on this SO question. I actually came across the same example before I knew it was in this issue.

However, so far my progress has been a little slow. My AWS knowledge is not fantastic so I have had to read a fair amount of the documentation on how to do certain things whilst following the example.

I did wonder if it would be possible for us to make something like this ourselves. If there was a way that the client app could list all the items in the bucket, you could have a page that renders all images. This SO question discusses listing all items in a bucket so it should be doable.

If the client app can list each item, then looping through them to render the images should be fairly straightforward. Trying to render every image at once may be a little 'expensive' so it could be a good idea to sort the list into some sort of order and allow the user to pick that they want to see. For example, sort the list into alphabetical order and let the user look at all images where the filename starts with a.

[RobStallion]

Elixir has a package ExAws.S3 with the function list objects. Based on the aws docs, this looks like the function that would list all images in the bucket.

Going to test this function out myself and see if this is a viable solution for you.

[RobStallion]

Following the docs, I am calling the function like so...

    "AWS_S3_BUCKET"
    |> System.get_env()
    |> ExAws.S3.list_objects()
    |> ExAws.request()

Currently getting the following error...

This error appears to be coming from the ExAws.request function. I have added a log before this function to see what argument is being passed into it and this is what was logged...

%ExAws.Operation.S3{
  body: "",
  bucket: "club-soda-staging",
  headers: %{},
  http_method: :get,
  params: %{},
  parser: &ExAws.S3.Parsers.parse_list_objects/1,
  path: "/",
  resource: "",
  service: :s3,
  stream_builder: #Function<2.32038611/1 in ExAws.S3.list_objects/2>
}

The error says missing xml parser which leads me to believe that the problem is with this line...

parser: &ExAws.S3.Parsers.parse_list_objects/1

[RobStallion]

The issue above was because you need to add sweet_xml as a dep to the project as mentioned in this in this issue. Once I did this the error went away.

I have now managed to get a list of all the items in the bucket. With this list we can create a screen with a bunch of images like so...

This currently doesn't have any functionality other than displaying the images but we should be able to extend this quite easily. For example, you can filter the bucket list (it's actually called the bucket list 🤣) by folder, name, etc.

We could also label the list on the front end (not just displaying the images like I have above). This should allow for easier debugging if someone says that an image has "gone missing".

@nelsonic let me know if you have any thoughts on the above 👍

[nelsonic]

@RobStallion this is a great starting point! 🎉 We if you can push your code up on a branch I can take a look at extending / customising the CSS. 💄