bag-database not reading from AWS S3

[kai-wei-mo]

The bag-database does not show any contents of an S3 bucket, as seen in the attached image.

To reproduce this error:

• do exactly as the documentation outlines
• in settings.yml, replace the last few lines with the following

storageId: 'anything-you-want'
accessKey: 'your-aws-access-key-with-s3-read-permissions'
secretKey: 'your-aws-secret-key-with-s3-read-permissions'
# remove the endPoint key-value pair
region: 'your-aws-bucket-region'
bucket: 'your-aws-bucket-name'

• in docker-compose.yml, replace bagdb's build . with image: swrirobotics/bag-database:latest
• run docker-compose up

[danthony06]

@pjreed I don't have the capability to test S3 buckets. Any idea what's going on?

[pjreed]

I'm pretty busy at the moment, but I'll try to test this when I have the chance. For what it's worth, you could use MinIO to start up a local S3 service and test using that, although of course, that doesn't guarantee it'd work with Amazon.

@kai-wei-mo Could you share the contents of the log file that is printed after you run docker-compose up?

[kai-wei-mo]

Sure thing, here's the log file @pjreed https://gist.github.com/kai-wei-mo/3a37dc75da9d73a725b5cf73e82a3f34

[pjreed]

Hm... this exception makes me think it's having an error parsing your settings.yml file:

bagdb_1     | 2022-03-08 20:32:29.992 [main] ERROR c.g.s.config.ConfigService - Error parsing YAML:
bagdb_1     | com.esotericsoftware.yamlbeans.YamlReader$YamlReaderException: Line 19, column 22: Unable to find property 'storageConfigurations' on class: com.github.swrirobotics.support.web.Configuration

Could you also share your entire settings.yml file?

[kai-wei-mo]

Sure thing, I also updated the URL for the previous link I sent so here is both: https://gist.github.com/kai-wei-mo/3a37dc75da9d73a725b5cf73e82a3f34 (logs) https://gist.github.com/kai-wei-mo/86d25833c0a9b55599f66b8df32f2dba (settings.yml)

[pjreed]

Aha, I think the problem is that the deployed swrirobotics/bag-database:latest Docker image is an old version that doesn't support S3 storage. That's actually been deprecated for a while now in favor of using images deployed to the GitHub container repository, so @danthony06 might want to skim through the documentation and find any references to that and replace them. Try using ghcr.io/swri-robotics/bag-database/bag-database:latest as the image in your docker-compose.yml instead and see if it loads your config file.

[kai-wei-mo]

Ah yes! That fixes everything for us. Thanks for your help @pjreed :D