search

databacker / mysql-backup

image to enable automated backups of mysql databases in containers
636 stars 178 forks source link

How to set DB_DUMP_FILENAME_PATTERN #303

Closed maitrungduc1410 closed 2 weeks ago

maitrungduc1410 commented 4 months ago

I'm running this as docker container and trying to custom dump filename but it's not working no matter using env or command flag:

db_backup:
  image: databack/mysql-backup
  command: dump --filename-pattern=my_db_backup.tgz

db_backup:
  image: databack/mysql-backup
  environment:
    - DB_DUMP_FILENAME_PATTERN=my_db_backup.tgz

dump file name is always like this db_backup_2024-04-26T15:56:28Z.tgz

Please help, Thanks

deitch commented 4 months ago

Assuming that your config above is accurate, you are using the latest tag, which is the somewhat older script-based version (completely revamped is due GA as 1.0.0 really shortly). You can run it with debug via env var DB_DUMP_DEBUG=true. Then paste your output but please check that you don't leak any secrets in the logs.

The newer version is much better about that.

skywalkerisnull commented 4 weeks ago

I have the same issue, this is my config

  mysql-backup:
    image: databack/mysql-backup:1.0.0-rc5
    restart: always
    environment:
      DB_SERVER: db
      DB_USER: ********
      DB_PASS: ************
      DB_NAMES: ************
      DB_DUMP_TARGET: s3://backups-******
      DB_DUMP_BEGIN:
      DB_DUMP_FREQ: 360
      AWS_ACCESS_KEY_ID: ****
      AWS_SECRET_ACCESS_KEY: ******
      AWS_ENDPOINT_URL: https://s3.ap-southeast-2.amazonaws.com
      AWS_DEFAULT_REGION: ap-southeast-2
      AWS_REGION: ap-southeast-2
      COMPRESSION: gzip
      DB_DUMP_FILENAME_PATTERN: replica-{{.now}}.tgz
      DB_DUMP_DEBUG: true
      TZ: Australia/Sydney
      NICE: true
    command: dump

And the only log I get in console is: mysql-backup-1 | time="2024-08-15T23:21:07Z" level=info msg="beginning dump 2024-08-15T23:21:07Z" run=185f342c-6328-4c69-8071-f23b466a66c6

I can see the tgz file appear in the S3 bucket with the filename of: db_backup_2024-08-15T23:21:07Z.tgz

deitch commented 3 weeks ago

@skywalkerisnull (ok, you really have to explain that handle! 😄 ), DB_DUMP_DEBUG is the older one. If you look in configuration.md, it shows it as available both as a CLI flag --debug and as an env var DEBUG=true. Try that and see?

DB_DUMP_FILENAME_PATTERN should work in the newer versions. So let's get those debug logs and figure out what is going on.

skywalkerisnull commented 3 weeks ago

@deitch haha, I have had a consultancy for a long time called Skywalker Consulting (my last name is Walker, and some old colleagues used to call me Vader) and when I was looking for a unique online handle that wasn't created when I was like 12, I came up with this when I was trying to post about Null bug on a forum, so I kinda just blended them, And it is unique enough that I can get it on every platform I need to.

Anyway, something is not quiet working using either the command: dump --debug or the env var of DEBUG: true but I did get more logs with command: dump -v=1 

Attaching to mysql-backup-1
mysql-backup-1  | time="2024-08-22T00:30:28Z" level=debug msg="starting dump"
mysql-backup-1  | time="2024-08-22T00:30:28Z" level=info msg="beginning dump 2024-08-22T00:30:28Z" run=7369575a-b44b-415b-9c43-5f0b5e5b2c36
mysql-backup-1  | time="2024-08-22T00:31:28Z" level=debug msg="uploading via protocol s3 from db_backup_2024-08-22T00:30:28Z.tgz" run=7369575a-b44b-415b-9c43-5f0b5e5b2c36
mysql-backup-1  | time="2024-08-22T00:31:29Z" level=debug msg="completed copying 0 bytes" run=7369575a-b44b-415b-9c43-5f0b5e5b2c36
^CGracefully stopping... (press Ctrl+C again to force)

and command: dump -v=2

Attaching to mysql-backup-1
mysql-backup-1  | time="2024-08-22T00:35:57Z" level=debug msg="starting dump"
mysql-backup-1  | time="2024-08-22T00:35:57Z" level=info msg="beginning dump 2024-08-22T00:35:57Z" run=39d8de23-da93-4c29-a163-0d12e368ecf0
mysql-backup-1  | time="2024-08-22T00:36:55Z" level=debug msg="uploading via protocol s3 from db_backup_2024-08-22T00:35:57Z.tgz" run=39d8de23-da93-4c29-a163-0d12e368ecf0
mysql-backup-1  | time="2024-08-22T00:36:55Z" level=debug msg="completed copying 0 bytes" run=39d8de23-da93-4c29-a163-0d12e368ecf0

Running it interactively by shelling into the container:

20d214e712b7:/$ printenv

DB_DUMP_TARGET=s3://backups-******
DB_NAMES=**********
HOSTNAME=20d214e712b7
DUMP=true
DB_DUMP_PRE_RESTORE_SCRIPTS=/scripts.d/pre-restore/
AWS_DEFAULT_REGION=ap-southeast-2
AWS_REGION=ap-southeast-2
PWD=/
TZ=Australia/Sydney
DB_USER=*****
HOME=/home/appuser
AWS_SECRET_ACCESS_KEY=********
DB_DUMP_FREQ=360
DB_DUMP_POST_RESTORE_SCRIPTS=/scripts.d/post-restore/
TERM=xterm
COMPRESSION=gzip
NICE=true
SHLVL=1
AWS_ACCESS_KEY_ID=********
DB_DUMP_POST_BACKUP_SCRIPTS=/scripts.d/post-backup/
AWS_ENDPOINT_URL=https://s3.ap-southeast-2.amazonaws.com
DB_SERVER=******
DB_DUMP_PRE_BACKUP_SCRIPTS=/scripts.d/pre-backup/
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG=true
DB_PASS=*********
DB_DUMP_FILENAME_PATTERN=replica-{{.now}}.tgz
_=/bin/printenv

30cdc8eee356:/$ ./mysql-backup dump
INFO[0000] beginning dump 2024-08-22T00:46:40Z           run=4418db64-cac0-4311-8653-3de20a6bc31b
deitch commented 3 weeks ago

Anyway, something is not quiet working using either the command: dump --debug or the env var of DEBUG: true but I did get more logs with command: dump -v=1

Worth its own issue, if you don't mind opening one. --v=<level> works, but env var DEBUG and --debug should default to a level (2, I think). Just show the issue replicated.

deitch commented 3 weeks ago

have had a consultancy for a long time called Skywalker Consulting (my last name is Walker, and some old colleagues used to call me Vader) and when I was looking for a unique online handle that wasn't created when I was like 12, I came up with this when I was trying to post about Null bug on a forum, so I kinda just blended them, And it is unique enough that I can get it on every platform I need to.

That is a fun origin story. I read Ben Thompson's stratechery, every Thursday it includes an interview. There usually are fun founder stories. I like this one.

deitch commented 3 weeks ago

This line:

mysql-backup-1  | time="2024-08-22T00:36:55Z" level=debug msg="uploading via protocol s3 from db_backup_2024-08-22T00:35:57Z.tgz" run=39d8de23-da93-4c29-a163-0d12e368ecf0

So it wants the remote filename to be db_backup_2024-08-22T00:35:57Z.tgz and that is where you set it to DB_DUMP_FILENAME_PATTERN: replica-{{.now}}.tgz, is that it?

I have a suspicion that it got lost in the conversion to v1.0.0-rc*. Digging into it.

deitch commented 2 weeks ago

Open PR #340 to fix it. Would you be able to test it @skywalkerisnull ?

skywalkerisnull commented 2 weeks ago

I will test it this morning

skywalkerisnull commented 2 weeks ago

We have success on #340 

mysql-backup-1  | time="2024-08-25T21:52:21Z" level=debug msg="starting dump"
mysql-backup-1  | time="2024-08-25T21:52:21Z" level=info msg="beginning dump 2024-08-25T21:52:21Z" run=1252a048-a1ca-4495-bd40-bd7c25722b9f
mysql-backup-1  | time="2024-08-25T21:53:20Z" level=debug msg="uploading via protocol s3 from db_backup_2024-08-25T21:52:21Z.tgz to replica-2024-08-25T21:52:21Z.tgz" run=1252a048-a1ca-4495-bd40-bd7c25722b9f
mysql-backup-1  | time="2024-08-25T21:53:20Z" level=debug msg="completed copying 0 bytes" run=1252a048-a1ca-4495-bd40-bd7c25722b9f

I will open another issue for the Debug env var, and I think there is another issue with the DB_DUMP_FREQ also not working correctly, I will check through existing posts before opening a new one for that one.