search

swri-robotics / bag-database

A server that catalogs bag files and provides a web-based UI for accessing them.
Other
344 stars 71 forks source link

Refactor storage for multiple backends and add S3 support #152

Closed pjreed closed 3 years ago

pjreed commented 3 years ago

This refactors how the Bag Database manages its bag storage. There is now a BagStorage interface that different mechanisms can implement to handle different types of storage, and an arbitrary number of different storage backends can be configured in the settings.yml file. Every BagStorage implementation has a storageId that is used to uniquely identify that backend and also a number of implementation-specific configuration parameters.

This also adds a FilesystemBagStorageImpl that provides filesystem-based storage that is backwards-compatible with previous versions of the Bag Database and an S3BagStorageImpl that support S3-compatible systems such as Amazon AWS or Backblaze.

As stated, this is backwards-compatible, and upgrading should not require any special actions. If a Bag Database does not have any storage backends configured, that is functionally equivalent to creating a FilesystemBagStorageImpl that has a bagPath of /bags and a storageId of default. See the updated documentation for more information on how to configure backends.

Fixes #69. (better late than never!)

danthony06 commented 3 years ago

@pjreed Looks like a good and expansive change. Do you think it's worth dropping a new version number for it, given how fundamental this change is?

pjreed commented 3 years ago

While there is a lot of new and changed code, it should all be backwards-compatible; you can upgrade in place on top of older Bag Databases and none of its APIs have changed (note: I actually just pushed a change to fix one call I had accidentally made incompatible), so if we're going by Semantic Versioning, only a minor version bump is appropriate. Up to you if you think it's worth a major version bump anyway, though. :wink:

pjreed commented 3 years ago

Sorry for another last-minute commit, but I think this is all ready now. If @daniel-dsouza is going to test this on your local BagDB, I would still recommend making a backup of the database first; this should be safe (in fact, you could probably even roll back to the previous version without any issue if the new version has problems, the only schema change is a new column in the bags table), but if something really bad happens, it could take over a week to rebuild your current database.

danthony06 commented 3 years ago

@pjreed Are you ready for us to test this now?

pjreed commented 3 years ago

Sure, it should be ready.

On Mon, Aug 23, 2021, 10:32 PM David Anthony @.***> wrote:

@pjreed https://github.com/pjreed Are you ready for us to test this now?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/swri-robotics/bag-database/pull/152#issuecomment-904294786, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC7XAVWJL7JXXCLBWJ6CAPTT6MHDBANCNFSM5CB4OTLQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .