search

pocketbase / pocketbase

Open Source realtime backend in 1 file
https://pocketbase.io
MIT License
39.21k stars 1.81k forks source link

Litestream integration for Admin #17

Closed gedw99 closed 5 months ago

gedw99 commented 2 years ago

It would be good to have Litestream integration. Litestream does not need integration as it can run as its own binary, but it would be cool to have some CLI and GUI integration for the Admin ? Is this something that is useful ?

I use litestream and can help with it if needed.

Also for scaling there are ways to deploy to Cloud run with the FS being remote. This allows the pocket base to scale out with a single FS for the DB and FS.

So Admin can easily do a restore etc.

bscott commented 2 years ago

i'm working on two Docker Images and one which includes Litestream.

https://github.com/bscott/pocketbase-docker https://github.com/bscott/pocketbase-litestream

sbusso commented 2 years ago

@bscott the litestream repo is empty. it would be good to submit your dockerfiles to the pocketbase repo when you have them

ganigeorgiev commented 2 years ago

For now I'll consider this as nice-to-have/low priority feature.

Although I agree that it will be a lot easier if the admin UI had Litestream integration, this will need to be considered more thoroughly because:

gedw99 commented 2 years ago

I opened a question issue about cgo-less SQLite on litestream to track this.

https://github.com/benbjohnson/litestream/issues/397

bscott commented 2 years ago

@bscott the litestream repo is empty. it would be good to submit your dockerfiles to the pocketbase repo when you have them

I'm waiting for another issue to be resolved as dockerizing pocket base is a bit difficult when the CLI requires to enter username and password via standard input.

ganigeorgiev commented 2 years ago

@bscott This was already resolved in v0.2.0. Now we are serving a web installer page when the user access the admin for the first time.

bscott commented 2 years ago

Thanks @ganigeorgiev I'll finish up my Litestream integration repo

bscott commented 2 years ago

Updated my Litestream docker repo. Litestream is spun up as the main process, Pocketbase as a sub-process. README is updated, but need to fix the formatting of the Markdown.

Litestream is configured to restore the pb_database if it existing in the Object Storage but not in the container at start.

https://github.com/bscott/pocketbase-litestream/ https://hub.docker.com/repository/docker/bscott/pocketbase-litestream

TylerSustare commented 2 years ago

I really appreaciated @bscott's work to get Litestream and PocketBase working. But I wanted to use PocketBase as a Go framework with a Litestream replicated DB.

I created this repo that can deploy to Fly.io and will restore the DB whenever we redeploy the Go code.

https://github.com/TylerSustare/pocketbase-framework-litestream

gedw99 commented 2 years ago

hey @TylerSustare you can also run Pocketbase on Fly.io Machines and go serverless. Saves money when no users. Startup time is .3 seconds :)

blog and code here: https://wundergraph.com/blog/wunderbase_serverless_graphql_database_on_top_of_sqlite_firecracker_and_prisma

Fly.io Machines is till pretty raw

benallfree commented 2 years ago

@TylerSustare i’m curious, why do you not use a persistent volume for the SQLite storage?

TylerSustare commented 2 years ago

@benallfree thats a good point. I only didn't have persistent volumes because I thought I would always want to be restoring from the backup. That being said, I'm not against it.

I also didn't know how easy that would be to point out how to set up those persistent volumes in the 3 providers I was trying to fit in the repo. I tried to make it easy to fit for Fly.io, GCP Cloud Run, and AWS App Runner.

But I would love the help if anyone would like to make a PR 😃

maxmcd commented 1 year ago

@TylerSustare I'm fairly certain you need to be very careful with this setup to avoid database corruption. Would be sure that your deploy strategy is rolling and max-per-region is set to 1 when deploying with fly.io. Otherwise you can have two databases replicating with litestream, which will lead to corruption.

Would definitely suggest configuring a disk with fly.io to avoid downloading a large db on startup (and to incidentally help with the deploy overlap issue): https://github.com/benbjohnson/litestream-read-replica-example/blob/dd2c9e50bac387ef3fbe6e3941cabbf590a2937b/fly.toml#L10-L12

valentindeaconu commented 5 months ago

If anybody needs it, I decided to open-source the image I'm currently using in production based on PocketBase and Litestream.

https://github.com/valentindeaconu/pocketbase-litestream-docker

ganigeorgiev commented 5 months ago

Old issues cleanup: I'm closing this feature request as I have no plans to embed Litestream into the default PocketBase binary (or integrating it with the Admin UI out of the box).

saikoml commented 5 months ago

:confused: please reconsider @ganigeorgiev see how many upvotes the issue has....

ganigeorgiev commented 5 months ago

@saikoml No sorry, the problems mentioned in https://github.com/pocketbase/pocketbase/issues/17#issuecomment-1179704321 are still existing and in general I don't think this is necessary as there too many caveats for the sake of a small convenience that I personally may never use.

Note that having a dedicated UI is neither a blocker nor a requirement for Litestream (or other db replication app) to work with PocketBase.

Additionally, the Admin UI will be refactored in the near future to allow some customization and ideally users will be able to create their own Litestream "plugin" if there is a need for it.