querylab / lazywarden

Automatic Bitwarden Backup
https://lazywarden.com
MIT License
606 stars 15 forks source link

vaultwarden and using secrets manager or random variable #12

Open danktankk opened 1 month ago

danktankk commented 1 month ago

I have been trying to figure this out, but I havent seen anything with regard to vaultwarden on this page other than it saying:

"This program is compatible with both self-hosted Bitwarden and Vaultwarden instances." and above that it says: "Configuring the Bitwarden Secret Manager is required for the program to function correctly. Even if you don't have an account, you must set a random variable to ensure proper operation."

I have not found a way to either get secrets manager working with vaultwarden, nor adding a random variable as described. This is the error i get: ERROR - Error authenticating Bitwarden client: Access token is not in a valid format: Doesn't contain a decryption key

Could you please explain the process for getting this to work in vaultwarden with regard to the secrets manager or how to create this random variable?

Many thanks!

BJReplay commented 1 month ago

I am not the author, but I think that you have to do two (well, three) separate things:

I haven't yet tried this.

danktankk commented 1 month ago

thanks for the info. I will try it and see what happens. much appreciated!

querylab commented 1 month ago

I am not the author, but I think that you have to do two (well, three) separate things:

* set up secrets manager on a (free) bitwarden account to hold all the secrets

* fill in unused accounts like mega with random secrets as the program (currently) expects secrets

* provide credentials for your vaultwarden instance so that the program can connect

I haven't yet tried this.

BJReplay, Thank you so much for taking the time to help with this problem. I really appreciate it

danktankk commented 1 month ago

I was able to get it to work. thank you for your help! this project is great, but not explained well. your insight was perfect!

querylab commented 1 month ago

hello danktankk, as far as I see this problem

ERROR - Error authenticating Bitwarden client: Access token is not in a valid format: Doesn't contain a decryption key

it is rather that these variables are not set correctly , every time you use a new machine you have to change a new ACCESS_TOKEN ORGANIZATION_ID= ACCESS_TOKEN=

querylab commented 1 month ago

I was able to get it to work. thank you for your help! this project is great, but not explained well. your insight was perfect!

Yes I have to improve the README 🀣 there are some confusing things I agree, glad it worked for you.

danktankk commented 1 month ago

I was able to get it to work. thank you for your help! this project is great, but not explained well. your insight was perfect!

Yes I have to improve the README 🀣 there are some confusing things I agree, glad it worked for you.

Well I spoke too soon, I have all the variables filled in, but I get littered with errors about it not being able to send to these non-existent accounts. It took me an hour or so just to get all these added and then editing the secrets.py file. A great update would be to omit the need to add all of these accounts. lol

Here is what I am seeing:

2024-10-12 17:49:59,844 - INFO - Bitwarden login successful, attempting to unlock the vault.
2024-10-12 17:49:59,845 - INFO - Unlocking the Bitwarden vault.
2024-10-12 17:50:05,683 - INFO - Vault unlocked successfully.
Syncing complete.
2024-10-12 17:50:11,075 - INFO - Logged in and synced
Bitwarden Backup:  15%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š                                     | 20/130 [00:20<01:54,  1.04s/it]2024-10-12 17:50:32,226 - INFO - ZIP file created at /root/lazywarden/backup-drive/bw-backup_2024_10_12_21_49_45.zip
2024-10-12 17:50:32,233 - INFO - Hash for ZIP file: 99adbcd23f7472cef4891cf7bf833553c7842327a61c41160ea605cd9673609
Bitwarden Backup:  31%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ                              | 40/130 [00:21<00:35,  2.55it/s]2024-10-12 17:50:32,234 - WARNING - Telegram is not configured. Notifications to Telegram will be skipped.
2024-10-12 17:50:32,675 - INFO - Notification sent to Discord
2024-10-12 17:50:32,676 - WARNING - Slack is not configured. Notifications to Slack will be skipped.
Bitwarden Backup:  38%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‰                           | 50/130 [00:21<00:22,  3.52it/s]2024-10-12 17:50:33,015 - ERROR - Error refreshing Dropbox token: 400 Client Error: Bad Request for url: https://api.dropboxapi.com/oauth2/token
2024-10-12 17:50:33,016 - ERROR - Error uploading to Dropbox: 400 Client Error: Bad Request for url: https://api.dropboxapi.com/oauth2/token
2024-10-12 17:50:33,017 - WARNING - Google Drive is not configured. Uploads to Google Drive will be skipped.
Bitwarden Backup:  46%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž                       | 60/130 [00:21<00:14,  4.81it/s]2024-10-12 17:50:33,019 - pcloud - INFO - Using pCloud API endpoint: https://api.pcloud.com/
2024-10-12 17:50:33,019 - INFO - Using pCloud API endpoint: https://api.pcloud.com/
2024-10-12 17:50:33,020 - pcloud - INFO - Using username/password authentication method.
2024-10-12 17:50:33,020 - INFO - Using username/password authentication method.
2024-10-12 17:50:33,377 - ERROR - Error uploading to pCloud: {'result': 2000, 'error': 'Log in failed.'}
2024-10-12 17:50:33,377 - ERROR - Error uploading to pCloud: {'result': 2000, 'error': 'Log in failed.'}
2024-10-12 17:50:39,333 - ERROR - Error uploading to Mega: 'int' object is not subscriptable
2024-10-12 17:50:39,333 - ERROR - Error uploading to Mega: 'int' object is not subscriptable
2024-10-12 17:50:39,644 - ERROR - Error checking the directory: HTTP error: 401
2024-10-12 17:50:39,644 - ERROR - HTTP error: 401
2024-10-12 17:50:39,645 - ERROR - Error uploading to Nextcloud: HTTP error: 401
2024-10-12 17:50:39,648 - ERROR - Error uploading the file to Seafile: Invalid URL 'I do not have an account/api2/auth-token/': No scheme supplied. Perhaps you meant https://I do not have an account/api2/auth-token/?
2024-10-12 17:50:39,649 - ERROR - Error uploading to Seafile: Invalid URL 'I do not have an account/api2/auth-token/': No scheme supplied. Perhaps you meant https://I do not have an account/api2/auth-token/?
2024-10-12 17:50:41,097 - ERROR - Error checking bucket: An error occurred (403) when calling the HeadBucket operation: Forbidden
2024-10-12 17:50:41,098 - ERROR - Error uploading file to Filebase: An error occurred (403) when calling the HeadBucket operation: Forbidden
2024-10-12 17:50:41,098 - ERROR - Error uploading to Filebase: An error occurred (403) when calling the HeadBucket operation: Forbidden
2024-10-12 17:50:41,420 - ERROR - Failed to fetch Todoist projects
2024-10-12 17:50:41,421 - ERROR - Error creating Todoist task: Failed to fetch Todoist projects
2024-10-12 17:50:41,440 - ERROR - Connection error: Invalid URL 'I do not have an account': No scheme supplied. Perhaps you meant https://I do not have an account?
2024-10-12 17:50:41,441 - ERROR - Error creating CalDAV event: Invalid URL 'I do not have an account': No scheme supplied. Perhaps you meant https://I do not have an account?
2024-10-12 17:50:41,441 - WARNING - SMTP is not configured. Sending emails will be skipped.
Bitwarden Backup:  54%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹                    | 70/130 [00:30<00:26,  2.31it/s]
2024-10-12 17:50:41,446 - INFO - Bitwarden backup completed successfully
2024-10-12 17:50:46,837 - INFO - Vault is unlocked.

I have these secrets set as the guide explains but it is trying to send still and i dont think the backup is finishing. Im close! But not there yet....

image

for anything I dont want to use or dont have, I have each secret set like this. I do have nextcloud set up, but I doubt it will ever work until I figure out how to add totp to this for login.

im not sure as I cant find this explained anywhere, but the organizationID i am using is from my self-hosted vaultwarden. not sure if that is what was meant for it but that is what I am trying to back up so I assumed that to be the case.

danktankk commented 4 weeks ago

bummer, blew a whole day on this. ah well.. win some lose some. ill check back in a few months. thanks for your hard work!

peterbata commented 3 weeks ago

Should we comment out the services that we are not subsribed to? Eg. Dropbox, PCloud, etc.

peterbata commented 3 weeks ago

hello danktankk, as far as I see this problem

ERROR - Error authenticating Bitwarden client: Access token is not in a valid format: Doesn't contain a decryption key

it is rather that these variables are not set correctly , every time you use a new machine you have to change a new ACCESS_TOKEN ORGANIZATION_ID= ACCESS_TOKEN=

I located both my ACCESS_TOKEN ORGANIZATION_ID & ACCESS_TOKEN and entered them into the appropriate fields. I have also created secrets as per your instructions / screenshot. However, I am still receiving the very same warning. This is about as far as I can get. Thank you for your time and assistance. Peter

lazywarden

Mosher23 commented 2 weeks ago

Reading this thread, I would suggest removing support of Vaultwarden from the Readme.md. Vaultwarden doesn't have Secrets Manager

danktankk commented 2 weeks ago

Reading this thread, I would suggest removing support of Vaultwarden from the Readme.md. Vaultwarden doesn't have Secrets Manager

It works, you just have to set up a free bitwarden account and proceed as the docs say. just getting anyone to answer back is the trick here.... I would LOVE it if we could just comment these unwanted accounts out. Thats the last thing I need to do for this to work.

danktankk commented 2 weeks ago

I was able to get it to work. thank you for your help! this project is great, but not explained well. your insight was perfect!

Yes I have to improve the README 🀣 there are some confusing things I agree, glad it worked for you.

@Querylab Haaaaaaallllp! lol Can you help me sort these errors I am getting from accounts I dont even have? Its the last thing left I need to get it running. I explained it above. The ability to just not use them would be ideal.

querylab commented 2 weeks ago

@danktankk I understand, I see the errors are related to services you don't have configured, such as pCloud, Dropbox, Nextcloud, Seafile, Filebase, Todoist and CalDAV. Currently, you need to configure all accounts in Bitwarden Secret Manager, even if you are not using them, and I know this can be inconvenient. I am working on improving error handling so that unnecessary accounts are not required. Unfortunately, I haven't had time to work on it, but it's on my list of improvements.

From what I've seen, the backup was successful, and the hash is correct. Where are you storing the backup locally?

danktankk commented 2 weeks ago

@danktankk I understand, I see the errors are related to services you don't have configured, such as pCloud, Dropbox, Nextcloud, Seafile, Filebase, Todoist and CalDAV. Currently, you need to configure all accounts in Bitwarden Secret Manager, even if you are not using them, and I know this can be inconvenient. I am working on improving error handling so that unnecessary accounts are not required. Unfortunately, I haven't had time to work on it, but it's on my list of improvements.

From what I've seen, the backup was successful, and the hash is correct. Where are you storing the backup locally?

@querylab Thank you for replying and I definitely understand being swamped and busy irl. I installed this on unraid last night and to prevent any confusion I will explain what happened on it since I have already shown logs from my ubuntu-server above.

I do have the services configured exactly as you have yours configured per the docs on the github page explaining how to do it. For each variable, I have a corresponding entry in Bitwarden Secrets Manager and I still get these errors as shown below in the logs. If the variables werent filled out and accessible, the process would stop and not allow me to continue. The things that troubles me is that the backup process never "completes" from what I am seeing even though it claims to be done.

Adding the new accounts that were recently added on your github also error in the same way (IE Storj, Vikunja, R2, etc).

The lazywarden backup file does save to '/mnt/user/backup/lazywarden'

Here is the output from the unraid install (the email is commented out because gmail thinks its phishing or something):

2024-10-25 00:48:45,074 - WARNING - Optional environment variable TELEGRAM_CHAT_ID is not set. Some functionality may be disabled.
2024-10-25 00:48:45,074 - WARNING - Optional environment variable SLACK_WEBHOOK_URL is not set. Some functionality may be disabled.
2024-10-25 00:48:45,074 - WARNING - Optional environment variable GOOGLE_SERVICE_ACCOUNT_FILE is not set. Some functionality may be disabled.
2024-10-25 00:48:45,074 - WARNING - Optional environment variable GOOGLE_FOLDER_ID is not set. Some functionality may be disabled.
2024-10-25 00:48:45,074 - WARNING - Optional environment variable SMTP_SERVER is not set. Some functionality may be disabled.
2024-10-25 00:48:45,074 - WARNING - Optional environment variable SMTP_PORT is not set. Some functionality may be disabled.
2024-10-25 00:48:45,074 - WARNING - Optional environment variable SMTP_USERNAME is not set. Some functionality may be disabled.
2024-10-25 00:48:45,075 - WARNING - Optional environment variable SMTP_PASSWORD is not set. Some functionality may be disabled.
2024-10-25 00:48:45,075 - WARNING - Optional environment variable EMAIL_RECIPIENT is not set. Some functionality may be disabled.
2024-10-25 00:48:45,075 - WARNING - Optional environment variable SENDER_EMAIL is not set. Some functionality may be disabled.
2024-10-25 00:48:45,075 - INFO - Environment variables loaded successfully
2024-10-25 00:48:45,075 - INFO - Logging configured successfully
2024-10-25 00:48:46,123 - INFO - Server configured successfully.
2024-10-25 00:48:46,123 - INFO - Server configured successfully
2024-10-25 00:48:46,130 - INFO - Bitwarden client setup successfully
2024-10-25 00:48:46,315 - INFO - Authenticated successfully using access token.
2024-10-25 00:48:46,316 - INFO - Bitwarden client authenticated successfully
2024-10-25 00:48:50,096 - INFO - Secrets retrieved successfully
2024-10-25 00:48:51,175 - INFO - Vault is locked.
2024-10-25 00:48:51,175 - INFO - Unlocking the Bitwarden vault.
2024-10-25 00:48:52,194 - ERROR - Failed to unlock the vault. Output: You are not logged in.
2024-10-25 00:48:52,194 - INFO - Bitwarden vault unlocked successfully
2024-10-25 00:48:52,194 - INFO - Environment variables and secrets loaded successfully.
2024-10-25 00:48:52,194 - WARNING - Google Drive is not configured. Uploads to Google Drive will be skipped.
2024-10-25 00:48:54,244 - INFO - User is not logged in or vault is locked.
 Saved setting `config`.You are logged in!

To unlock your vault, set your session key to the `BW_SESSION` environment variable. ex:
$ export BW_SESSION="redact"
> $env:BW_SESSION="redact"

You can also pass the session key to any command with the `--session` option. ex:
$ bw list items --session <redact> - INFO - Login process output: You are logged in!

To unlock your vault, set your session key to the `BW_SESSION` environment variable. ex:
$ export BW_SESSION="<redact>"
> $env:BW_SESSION="<redact>"

You can also pass the session key to any command with the `--session` option. ex:
$ bw list items --session <redact>

2024-10-25 00:48:58,376 - INFO - Bitwarden login successful, attempting to unlock the vault.
2024-10-25 00:48:58,376 - INFO - Unlocking the Bitwarden vault.
2024-10-25 00:49:00,840 - INFO - Vault unlocked successfully.
2024-10-25 00:49:03,172 - INFO - Logged in and synced
Bitwarden Backup:  15%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š                                     | 20/130 [00:09<00:52,  2.11it/s]2024-10-25 00:49:12,802 - INFO - ZIP file created at /root/lazywarden/backup-drive/bw-backup_<redact>.zip
2024-10-25 00:49:12,813 - INFO - Hash for ZIP file: baf26675c961abf5196910c7165d8b27e7408cdefb72746bac57ecfda97c9ecd
2024-10-25 00:49:12,813 - WARNING - Telegram is not configured. Notifications to Telegram will be skipped.
2024-10-25 00:49:13,098 - INFO - Notification sent to Discord
2024-10-25 00:49:13,098 - WARNING - Slack is not configured. Notifications to Slack will be skipped.
Bitwarden Backup:  38%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‰                           | 50/130 [00:09<00:11,  7.01it/s]2024-10-25 00:49:13,238 - ERROR - Error refreshing Dropbox token: 400 Client Error: Bad Request for url: https://api.dropboxapi.com/oauth2/token
2024-10-25 00:49:13,238 - ERROR - Error uploading to Dropbox: 400 Client Error: Bad Request for url: https://api.dropboxapi.com/oauth2/token
2024-10-25 00:49:13,238 - WARNING - Google Drive is not configured. Uploads to Google Drive will be skipped.
Bitwarden Backup:  46%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž                       | 60/130 [00:10<00:07,  9.10it/s]2024-10-25 00:49:13,239 - pcloud - INFO - Using pCloud API endpoint: https://api.pcloud.com/
2024-10-25 00:49:13,239 - INFO - Using pCloud API endpoint: https://api.pcloud.com/
2024-10-25 00:49:13,239 - pcloud - INFO - Using username/password authentication method.
2024-10-25 00:49:13,239 - INFO - Using username/password authentication method.
2024-10-25 00:49:13,439 - ERROR - Error uploading to pCloud: {'result': 2000, 'error': 'Log in failed.'}
2024-10-25 00:49:13,439 - ERROR - Error uploading to pCloud: {'result': 2000, 'error': 'Log in failed.'}
2024-10-25 00:49:14,078 - ERROR - Error uploading to Mega: 'int' object is not subscriptable
2024-10-25 00:49:14,078 - ERROR - Error uploading to Mega: 'int' object is not subscriptable
2024-10-25 00:49:14,362 - ERROR - Error checking the directory: HTTP error: 401
2024-10-25 00:49:14,362 - ERROR - HTTP error: 401
2024-10-25 00:49:14,362 - ERROR - Error uploading to Nextcloud: HTTP error: 401
2024-10-25 00:49:14,363 - ERROR - Error uploading the file to Seafile: Invalid URL 'I do not have an account/api2/auth-token/': No scheme supplied. Perhaps you meant https://I do not have an account/api2/auth-token/?
2024-10-25 00:49:14,363 - ERROR - Error uploading to Seafile: Invalid URL 'I do not have an account/api2/auth-token/': No scheme supplied. Perhaps you meant https://I do not have an account/api2/auth-token/?
2024-10-25 00:49:14,737 - ERROR - Error checking bucket: An error occurred (403) when calling the HeadBucket operation: Forbidden
2024-10-25 00:49:14,737 - ERROR - Error uploading file to Filebase: An error occurred (403) when calling the HeadBucket operation: Forbidden
2024-10-25 00:49:14,737 - ERROR - Error uploading to Filebase: An error occurred (403) when calling the HeadBucket operation: Forbidden
2024-10-25 00:49:14,947 - ERROR - Failed to fetch Todoist projects
2024-10-25 00:49:14,947 - ERROR - Error creating Todoist task: Failed to fetch Todoist projects
2024-10-25 00:49:14,950 - ERROR - Connection error: Invalid URL 'I do not have an account': No scheme supplied. Perhaps you meant https://I do not have an account?
2024-10-25 00:49:14,950 - ERROR - Error creating CalDAV event: Invalid URL 'I do not have an account': No scheme supplied. Perhaps you meant https://I do not have an account?
2024-10-25 00:49:14,950 - WARNING - SMTP is not configured. Sending emails will be skipped.
Bitwarden Backup:  54%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹                    | 70/130 [00:11<00:10,  5.95it/s]
2024-10-25 00:49:14,955 - ERROR - Error setting up the S3 client for Storj: Invalid endpoint: i dont use this
2024-10-25 00:49:14,955 - ERROR - Failed to upload the ZIP file to Storj. No notification will be sent.
2024-10-25 00:49:14,957 - ERROR - Error creating S3 client for Cloudflare R2: Invalid endpoint: i dont use this
2024-10-25 00:49:14,957 - ERROR - Failed to upload the ZIP file to Cloudflare R2. No notification will be sent.
2024-10-25 00:49:14,957 - INFO - Starting to create a task and add a label in Vikunja...
2024-10-25 00:49:14,958 - ERROR - Error during request to Vikunja: Invalid URL 'i dont use this/projects': No scheme supplied. Perhaps you meant https://i dont use this/projects?
2024-10-25 00:49:14,958 - ERROR - Failed to retrieve projects from Vikunja.
2024-10-25 00:49:14,958 - INFO - Project 'Bitwarden Drive Backup' does not exist. Creating it.
2024-10-25 00:49:14,958 - ERROR - Error during request to Vikunja: Invalid URL 'i dont use this/projects': No scheme supplied. Perhaps you meant https://i dont use this/projects?
2024-10-25 00:49:14,958 - ERROR - Could not connect to the API.
2024-10-25 00:49:14,958 - ERROR - Could not create or find the project.
2024-10-25 00:49:14,958 - ERROR - Failed to create task in Vikunja. No notification will be sent.
2024-10-25 00:49:14,958 - WARNING - Vikunja environment variables are not set. The backup process will not be logged in Vikunja.
2024-10-25 00:49:14,959 - INFO - Bitwarden backup completed successfully
2024-10-25 00:49:17,272 - INFO - Vault is unlocked.

As you can see, it stops progressing at about 54% or 70/130 and it gives the hash when it is at 15% complete. This happens for both installations I currently have for testing. Below is just a handful of these erroring entries from secrets manager:

Bitwarden Backup:  15%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š                                     | 20/130 [00:09<00:52,  2.11it/s]2024-10-25 00:49:12,802 - INFO - ZIP file created at /root/lazywarden/backup-drive/bw-backup_<redact>.zip
2024-10-25 00:49:12,813 - INFO - Hash for ZIP file: baf26675c961abf5196910c7165d8b27e7408cdefb72746bac57ecfda97c9ecd

image

All entries are like this and all entries are added to the .env file for unraid.

Please let me know if you require anything else and thank you for your time and efforts! Its important to have good backup practices imho.

querylab commented 2 weeks ago

@danktankk I see the problem, you are using Unraid πŸ™ƒ. I haven't worked with Unraid, but I notice that the storage directory is β€˜/mnt/user/backup/lazywarden’. For the program to work properly, you must run it as root user.

Have you tried backup up directly with the Python main.py file? I see you have 10 secrets configured, in addition to the 4 Bitwarden secrets required:

In addition, I assume you also have the 4 encryption secrets. In total, 36 secrets are required to be configured in Bitwarden Secret Manager for it to work properly. I know some secrets you won't use, like Seafile, but you need to assign a value to them, even if it's a placeholder. For example:

danktankk commented 2 weeks ago

@danktankk I see the problem, you are using Unraid πŸ™ƒ. I haven't worked with Unraid, but I notice that the storage directory is β€˜/mnt/user/backup/lazywarden’. For the program to work properly, you must run it as root user.

Have you tried backup up directly with the Python main.py file? I see you have 10 secrets configured, in addition to the 4 Bitwarden secrets required:

  • BW_URL: Vaultwarden URL
  • BW_USERNAME: Vaultwarden username
  • BW_PASSWORD: Vaultwarden Password
  • BW_TOTP_SECRET: TOTP code of Vaultwarden

In addition, I assume you also have the 4 encryption secrets. In total, 36 secrets are required to be configured in Bitwarden Secret Manager for it to work properly. I know some secrets you won't use, like Seafile, but you need to assign a value to them, even if it's a placeholder. For example:

  • SEAFILE_SERVER_URL=n/a
  • SEAFILE_USERNAME=n/a
  • SEAFILE_PASSWORD=n/a

@querylab It is being run as the root user.
image image

With regard to the secrets, I said "Below is just a handful of these erroring entries from secrets manager:" I have themall added in there, its just more than one page now because of all the unnecessary entries that are required. I can assure you they are all there.

image

If you look at the logs I sent not only for unraid, but from ubuntu-server over a week ago the errors look like:

2024-10-25 00:49:14,958 - ERROR - Error during request to Vikunja: Invalid URL 'i dont use this/projects': No scheme supplied. Perhaps you meant https://i dont use this/projects?
2024-10-25 00:49:14,958 - ERROR - Failed to retrieve projects from Vikunja.

2024-10-12 17:50:39,649 - ERROR - Error uploading to Seafile: Invalid URL 'I do not have an account/api2/auth-token/': No scheme supplied. Perhaps you meant https://I do not have an account/api2/auth-token/?

I have been putting in the actual UUID for ALL accounts whether i use them or not so I am not sure where the disconnect is here. but it is the same issue on both implementations. Both run as root, both have all variables added the secrets manager and all accounts UUID are present.

querylab commented 1 week ago

@danktankk Greetings! Sorry for the delay, it's been a busy few days. I've made an update where you now only need to use the Bitwarden Secret Manager variables corresponding to the services you will actually use. Please let me know if you are still experiencing errors.

danktankk commented 1 week ago

@querylab

Thank you for the effort and hard work! I gave it another shot and way less errors but the same things happens in the logs. It never reaches 100% It claims that it backed up succesfully, but i wonder why it never completes?

At any rate here are the logs from the test. I only did minimal settings as there would be less to break.

Please wait while we securely back up your vault data... πŸ”„

 2024-11-03 21:38:25,986 - WARNING - Optional environment variable TELEGRAM_TOKEN is not set. Some functionality may be disabled.
2024-11-03 21:38:25,986 - WARNING - Optional environment variable TELEGRAM_CHAT_ID is not set. Some functionality may be disabled.
2024-11-03 21:38:25,987 - WARNING - Optional environment variable SLACK_WEBHOOK_URL is not set. Some functionality may be disabled.
2024-11-03 21:38:25,987 - WARNING - Optional environment variable GOOGLE_SERVICE_ACCOUNT_FILE is not set. Some functionality may be disabled.
2024-11-03 21:38:25,987 - WARNING - Optional environment variable GOOGLE_FOLDER_ID is not set. Some functionality may be disabled.
2024-11-03 21:38:25,987 - WARNING - Optional environment variable SMTP_SERVER is not set. Some functionality may be disabled.
2024-11-03 21:38:25,987 - WARNING - Optional environment variable SMTP_PORT is not set. Some functionality may be disabled.
2024-11-03 21:38:25,987 - WARNING - Optional environment variable SMTP_USERNAME is not set. Some functionality may be disabled.
2024-11-03 21:38:25,987 - WARNING - Optional environment variable SMTP_PASSWORD is not set. Some functionality may be disabled.
2024-11-03 21:38:25,987 - WARNING - Optional environment variable EMAIL_RECIPIENT is not set. Some functionality may be disabled.
2024-11-03 21:38:25,988 - WARNING - Optional environment variable SENDER_EMAIL is not set. Some functionality may be disabled.
2024-11-03 21:38:25,988 - WARNING - Optional environment variable NTFY_URL is not set. Some functionality may be disabled.
2024-11-03 21:38:25,988 - INFO - Environment variables loaded successfully
2024-11-03 21:38:25,988 - INFO - Logging configured successfully
2024-11-03 21:38:29,731 - INFO - Server configured successfully.
2024-11-03 21:38:29,732 - INFO - Server configured successfully
2024-11-03 21:38:29,733 - INFO - Bitwarden client setup successfully
2024-11-03 21:38:30,008 - INFO - Authenticated successfully using access token.
2024-11-03 21:38:30,009 - INFO - Bitwarden client authenticated successfully
2024-11-03 21:38:30,531 - WARNING - Optional secret 'PCLOUD_USERNAME' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,532 - WARNING - Optional secret 'PCLOUD_PASSWORD' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,532 - WARNING - Optional secret 'MEGA_EMAIL' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,533 - WARNING - Optional secret 'MEGA_PASSWORD' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,533 - WARNING - Optional secret 'DROPBOX_ACCESS_TOKEN' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,534 - WARNING - Optional secret 'DROPBOX_REFRESH_TOKEN' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,534 - WARNING - Optional secret 'DROPBOX_APP_KEY' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,534 - WARNING - Optional secret 'DROPBOX_APP_SECRET' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,535 - WARNING - Optional secret 'TODOIST_TOKEN' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,535 - WARNING - Optional secret 'CALDAV_URL' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,536 - WARNING - Optional secret 'CALDAV_USERNAME' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,536 - WARNING - Optional secret 'CALDAV_PASSWORD' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,536 - WARNING - Optional secret 'NEXTCLOUD_URL' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,537 - WARNING - Optional secret 'NEXTCLOUD_USERNAME' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,537 - WARNING - Optional secret 'NEXTCLOUD_PASSWORD' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,537 - WARNING - Optional secret 'SEAFILE_SERVER_URL' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,538 - WARNING - Optional secret 'SEAFILE_USERNAME' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,538 - WARNING - Optional secret 'SEAFILE_PASSWORD' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,539 - WARNING - Optional secret 'FILEBASE_ACCESS_KEY' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,539 - WARNING - Optional secret 'FILEBASE_SECRET_KEY' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,539 - WARNING - Optional secret 'KEEPASS_PASSWORD' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,540 - WARNING - Optional secret 'STORJ_ACCESS_KEY' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,540 - WARNING - Optional secret 'STORJ_SECRET_KEY' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,540 - WARNING - Optional secret 'STORJ_ENDPOINT' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,541 - WARNING - Optional secret 'R2_ACCESS_KEY_ID' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,541 - WARNING - Optional secret 'R2_SECRET_ACCESS_KEY' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,541 - WARNING - Optional secret 'R2_ENDPOINT_URL' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,541 - WARNING - Optional secret 'VIKUNJA_API_TOKEN' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,541 - WARNING - Optional secret 'VIKUNJA_URL' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,542 - WARNING - Optional secret 'B2_APP_KEY_ID' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,542 - WARNING - Optional secret 'B2_APP_KEY' is not set. Some functionality may be disabled.
2024-11-03 21:38:30,627 - INFO - Secrets retrieved successfully
2024-11-03 21:38:34,341 - INFO - Vault is locked.
2024-11-03 21:38:34,341 - INFO - Unlocking the Bitwarden vault.
2024-11-03 21:38:38,038 - ERROR - Failed to unlock the vault. Output: You are not logged in.
2024-11-03 21:38:38,038 - INFO - Bitwarden vault unlocked successfully
2024-11-03 21:38:38,039 - INFO - Environment variables and secrets loaded successfully.
2024-11-03 21:38:38,039 - WARNING - Google Drive is not configured. Uploads to Google Drive will be skipped.
Saved setting `config`.
2024-11-03 21:38:45,457 - INFO - User is not logged in or vault is locked.
2024-11-03 21:38:45,458 - ERROR - Error generating TOTP code: Non-base32 digit found
2024-11-03 21:38:45,458 - ERROR - Error generating TOTP code: Non-base32 digit found
2024-11-03 21:38:45,458 - INFO - Execute login command: bw login --nointeraction user@gmail.com ********
2024-11-03 21:38:55,983 - INFO - Login process stdout:
You are logged in!

To unlock your vault, set your session key to the `BW_SESSION` environment variable. ex:
$ export BW_SESSION="123456789=="
> $env:BW_SESSION="123456789=="

Yo0u can also pass the session key to any command with the `--session` option. ex:
$ bw list items --session 123456789==
2024-11-03 21:38:55,984 - INFO - Login process stderr:

2024-11-03 21:38:55,984 - INFO - Bitwarden login successful, attempting to unlock the vault.
2024-11-03 21:38:55,984 - INFO - Unlocking the Bitwarden vault.
2024-11-03 21:39:01,850 - INFO - Vault unlocked successfully.
Syncing complete.
2024-11-03 21:39:07,297 - INFO - Logged in and synced
Bitwarden Backup:  15%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š                                     | 20/130 [00:22<02:01,  1.10s/it]2024-11-03 21:39:29,776 - INFO - ZIP file created at /root/lazywarden/backup-drive/bw-backup.zip
2024-11-03 21:39:29,783 - INFO - Hash for ZIP file: 123456789-hash 
Bitwarden Backup:  31%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ                              | 40/130 [00:22<00:37,  2.40it/s]2024-11-03 21:39:29,784 - WARNING - Telegram is not configured. Notifications to Telegram will be skipped.
2024-11-03 21:39:30,177 - INFO - Notification sent to Discord
2024-11-03 21:39:30,179 - WARNING - Slack is not configured. Notifications to Slack will be skipped.
2024-11-03 21:39:30,179 - WARNING - ntfy is not configured. Notifications to ntfy will be skipped.
2024-11-03 21:39:30,180 - WARNING - Dropbox is not configured. Uploads to Dropbox will be skipped.
Bitwarden Backup:  46%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž                       | 60/130 [00:22<00:16,  4.37it/s]2024-11-03 21:39:30,181 - WARNING - Google Drive is not configured. Uploads to Google Drive will be skipped.
2024-11-03 21:39:30,182 - WARNING - pCloud is not configured. Uploads to pCloud will be skipped.
2024-11-03 21:39:30,182 - WARNING - Mega is not configured. Uploads to Mega will be skipped.
2024-11-03 21:39:30,182 - WARNING - Nextcloud is not configured. Uploads to Nextcloud will be skipped.
2024-11-03 21:39:30,182 - WARNING - Seafile is not configured. Uploads to Seafile will be skipped.
2024-11-03 21:39:30,182 - WARNING - Filebase is not configured. Uploads to Filebase will be skipped.
2024-11-03 21:39:30,183 - WARNING - Todoist is not configured. Task creation in Todoist will be skipped.
2024-11-03 21:39:30,183 - WARNING - CalDAV is not configured. Event creation in CalDAV will be skipped.
2024-11-03 21:39:30,183 - WARNING - SMTP is not configured. Sending emails will be skipped.
Bitwarden Backup: |                                                 | 150/? [00:22<00:00,  6.57it/s]
2024-11-03 21:39:30,184 - WARNING - Backblaze B2 is not configured. Uploads to Backblaze B2 will be skipped.
2024-11-03 21:39:30,184 - WARNING - Storj is not configured. Uploads to Storj will be skipped.
2024-11-03 21:39:30,184 - WARNING - Cloudflare R2 is not configured. Uploads to Cloudflare R2 will be skipped.
2024-11-03 21:39:30,187 - INFO - Bitwarden backup completed successfully
2024-11-03 21:39:35,638 - INFO - Vault is unlocked.

I removed anything questionable, but you can see it finished and it didnt. lol you could call this shrodingers backup lolol

jk jk

any ideas for that odd incomplet completion?

thanks again for your effort!