search

ttionya / vaultwarden-backup

Backup vaultwarden (formerly known as bitwarden_rs) SQLite3/PostgreSQL/MySQL/MariaDB database by rclone. (Docker)
MIT License
998 stars 115 forks source link

Container crashes if cannot authenticate to S3 #79

Closed danielporto closed 1 year ago

danielporto commented 2 years ago

When I have the correct AWS ACCESS_KEY_ID and SECRET_ACCESS_KEY the backup is done correctly and I get an email to confirm.

However, if I set these two variables values to invalid ones, to simulate a problem I was expected to get an email informing that the backup has failed. However, the container crashes.

(note that the failure to find the rclone.conf is ok because im using envs to config and works when the secrets are set correctly)

Here is the output:

today at 6:38:43 PM========================================
today at 6:38:43 PM2022/08/23 17:38:43 NOTICE: Config file "/config/rclone/rclone.conf" not found - using defaults
today at 6:38:44 PM2022/08/23 17:38:44 ERROR : Attempt 1/3 failed with 1 errors and: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
today at 6:38:44 PM status code: 403, request id: 08RN2SGYYD7SMB4B, host id: 6px/vxAZm+2n1jwhN5kT2wtNhWWbNOJAzfv+lE2W3Mq4IEPSywQYQ0bA5WOJSC8R+jZbaHUkJxs=
today at 6:38:44 PM2022/08/23 17:38:44 ERROR : Attempt 2/3 failed with 1 errors and: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
today at 6:38:44 PM status code: 403, request id: 08RK2GQS5FEC8W76, host id: Nl037/OV5NB3INcJMJoqNiTlKUb5JNbfDbKA4Dwh5yQbO3jMcwungsFaqoUK40VbOOv43ypIAD4=
today at 6:38:44 PM2022/08/23 17:38:44 ERROR : Attempt 3/3 failed with 1 errors and: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
today at 6:38:44 PM status code: 403, request id: 08RMA24B37XPHKP0, host id: Z+4yoIPcv6WqRKDgJFRlhGn0ZIIbHzUwrVlECZ6Ln5zYN1Bdv47zn2GdV8QDMm73tlaX427k4qk=
today at 6:38:44 PM2022/08/23 17:38:44 Failed to mkdir: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
today at 6:38:44 PM status code: 403, request id: 08RMA24B37XPHKP0, host id: Z+4yoIPcv6WqRKDgJFRlhGn0ZIIbHzUwrVlECZ6Ln5zYN1Bdv47zn2GdV8QDMm73tlaX427k4qk=
today at 6:38:44 PMstorage system connection failure [mys3:/mys3-keychain-backup]
01/01/1970, 1:00 AM
today at 6:38:52 PMContainer stopped
ttionya commented 2 years ago

I have never used s3 before, can you provide a configuration of the wrong s3?

ttionya commented 2 years ago

@danielporto ,

I see, this is the container startup verification, the container will try to connect to the remote storage and create the folder when it starts, if this operation fails, then the container will exit automatically.

What error are you trying to simulate?

danielporto commented 2 years ago

Hello @ttionya thanks for your quick reply! The thing I'm trying to check is the email with the backup failure notification.

Despite the backup has been completed locally, the specific execution failed to upload it to a safe storage. In practice this failure can happen for a number of reasons such as wrong/updated keys, permissions etc..

What I expected to get is an email stating the failure (ideally the root cause, but Im ok with just a general failure). Thanks

ttionya commented 2 years ago

The email will only notify you that the backup failed, without any error reason. At this point you should check the docker logs to determine the exact cause of the error. I want to keep the notification system simple.

danielporto commented 2 years ago

The email will only notify you that the backup failed, without any error reason. At this point you should check the docker logs to determine the exact cause of the error. I want to keep the notification system simple.

sure thing. The problem is that right now I'm not getting any email at all. The system tries to backup but the container crashes without sending the failure notification via email. That's the point. I wish to get a notification because indeed the backup process failed.

ttionya commented 2 years ago

It is because it does not pass the check at startup, and the container will start correctly only if the rclone configuration used can read and write to the remote storage correctly.

At this point it has not entered the backup process because it fails at startup. The working state of the container can be confirmed by docker ps or docker-compose ps or docker manager. And looking at the logs, you can see the red error message storage system connection failure [mys3:/mys3-keychain-backup].

danielporto commented 2 years ago

I see. Basically there are two potentially different situations: 1 - the container starts-up with an invalid rclone config. The only way to overcome is to update the entrypoint to send the notification. (assuming the smtp configuration is also correct) 2 - the container starts up with a valid configuration but the connection is dropped somehow. So rclone will start but when the connection drop is detected some error will come out. This I did not test. Perhaps in this situation the notification is sent. That can be tested easily. I will do and report back.

ttionya commented 2 years ago

Yes

1 - the container starts-up with an invalid rclone config.

The container will fail and exit (set to restart: always and it will restart automatically) and the tool will not send any notifications. It does not enter the backup process at all, but uses the wrong configuration (and perhaps network errors, etc.) to detect that it cannot operate the remote storage properly, and it tells the user that the container has an exception by exiting the container with an error code (exit code 1).

2 - the container starts up with a valid configuration but the connection is dropped somehow.

As long as the container is started properly, it will perform backups at regular intervals according to the cron configuration and will also notify the backup results (MAIL_X or PING_URL is configured properly).