benbjohnson / litestream

Streaming replication for SQLite.
https://litestream.io
Apache License 2.0
11.1k stars 256 forks source link

refactor main to allow cli as high-level lib API #608

Closed lherman-cs closed 1 month ago

lherman-cs commented 1 month ago

Motivation: use Litestream as a library

Related Issues:

  1. https://github.com/benbjohnson/litestream/issues/440
  2. https://github.com/benbjohnson/litestream/issues/215

Current Workaround: https://github.com/benbjohnson/litestream-library-example

The current workaround is complex, and no guarantee on the API backward compatibility (https://github.com/benbjohnson/litestream/issues/215#issuecomment-855426990).

Solution: refactor out main package into cmd package

  1. Backward Compatibility

CLI interface is more stable than calling Litestream directly via function calls. Most users already use Litestream through CLI (including service wrappers), so backward compatibility at this interface level is desired.

  1. Simple

Using Litestream as a library is now as simple as the following:

cmd.Run([]string{"litestream", "replicate", "/path/to/db", "s3://BUCKETNAME/PATHNAME"}) cmd.Run([]string{"litestream", "restore", "-o", "/path/to/db", "s3://BUCKETNAME/PATHNAME"})

Koeng101 commented 1 month ago

main.go is mostly just copied, and the commit history should be maintained. Move it first, commit, then commit updates.

I wish there was a better way to do this, but this does seem like an alright way to run it. However, I don't see any documentation updates or new examples. I'd like to see a documentation update somewhere, as well as a Go example of how to use it. Especially around proper closing and error handling (I imagine you'll have to load as a goroutine, but then how about context returns? Or error returns upon failure to replicate? How do you handle?)

lherman-cs commented 1 month ago

@Koeng101 I mainly wanted to use this to start a conversation before investing too early in examples. From https://github.com/benbjohnson/litestream/issues/42#issuecomment-2387601816, this is very likely not going to get merged. So, I will cancel the PR.

There are 2 ways to handle closing and error handling:

  1. Via signals, https://pkg.go.dev/os/signal#Notify

Litestream listens to SIGINT. signal.Notify allows multiple signal handlers. So, the main program can also listen to SIGINT and wait for Litestream to end gracefully.

  1. Via context cancellation, similar to 1 but more generic. Although, I just saw that Litestream doesn't seem to handle context cancellation for replicate subcommands. So, it needs to be patched.

So, the workflow requires a "supervisor" goroutine that manages the error and closing for each replicate goroutine. This is far from ideal, but it works with low effort.

Something like this (WARNING: haven't been tested):

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
g, ctx := errgroup.WithContext(ctx)

gCtx := ctx
g.Go(func() error {
  return cmd.Run(gCtx, []string{"litestream", "replicate", "/path/to/db", "s3://BUCKETNAME/PATHNAME"})
})

// proceed with main program
g.Go(...) <-- main goroutine

err := g.Wait()
// handle main program or litestream errors here