tursodatabase / libsql

libSQL is a fork of SQLite that is both Open Source, and Open Contributions.
https://turso.tech/libsql
MIT License
9.54k stars 252 forks source link

implement create and copy bottomless-cli commands #1700

Closed sivukhin closed 1 month ago

sivukhin commented 1 month ago

Context

This PR implements two simple commands for bottomless-cli:

  1. bottomless-cli -n "ns-xxx:yyy" copy --to-dir local-directory -g generation - copy specified (or latest if omitted) generation S3 content to the local directory
  2. bottomless-cli -n "ns-xxx:yyy" -d temp create --source-db-path dbs/xxx/data - create new generation (with fresh timestamp) in the S3 from database file and upload single snapshot to the S3 with all necessary metadata for bottomless (.changecounter)

Create command opens DB at the path provided by --source-db-path in read-only mode and execute VACUUM INTO ? query to initialize fresh DB at the path provided by -d. After that DB without any changes in WAL will be sent as a single snapshot to the S3.

Also, CLI arguments structure a bit reworked in order to make it more easy to use:

  1. Now more command can be run without -d argument: ls, copy, verify do not require this param
  2. Now you can provide --db-name argument which will be used to form whole namespace name if LIBSQL_BOTTOMLESS_DATABASE_ID env var is set (if not - then there is a misuse of bottomless-cli and command will return an error). See an example:
    $> bottomless-cli --db-name "b7698553-676c-4e75-a449-791764e0f0e5" ls
    2024-08-26T13:27:08.054032Z  INFO bottomless_cli: LIBSQL_BOTTOMLESS_DATABASE_ID env var were updated: '47a3d201-c415-48b5-83c3-a456a617adf4' -> 'ns-47a3d201-c415-48b5-83c3-a456a617adf4:b7698553-676c-4e75-a449-791764e0f0e5'
    2024-08-26T13:27:08.832453Z  INFO bottomless::replicator: Bucket bottomless exists and is accessible
    ...
    # but if no env var set
    $> bottomless-cli --db-name "b7698553-676c-4e75-a449-791764e0f0e5" ls
    Error: db_name can be set only if LIBSQL_BOTTOMLESS_DATABASE_ID env var has namespace ID

Plus, bottomless-cli now shipped as a part of docker image (it's 15M - so I think it's ok to include it by default)