search

bitwarden / clients

Bitwarden client apps (web, browser extension, desktop, and cli).
https://bitwarden.com
Other
9.32k stars 1.26k forks source link

Exporting encrypted vault with Personal API Key? #2739

Open berkant opened 3 years ago

berkant commented 3 years ago

Hello.

Is there a way to export an encrypted vault after logging in with --apikey flag.

The scenario is running a periodic task with GitHub Actions to export and re-upload my encrypted vault to another storage medium. I thought I could call export without a master password but the case seems not. I don't feel like providing GitHub with my master password.

I just noticed bw populates .config/Bitwarden\ CLI/data.json with encrypted stuff seemingly including my logins, but I am not sure if I can restore using that file sometime in the future. Is data exported by bw export any different than data.json?

Opinion: You should probably back up data.json instead of encrypted exports. Encrypted exports don't include your encrypted private key and other crucial data used to decrypt your vault. That means you will have to connect to Bitwarden's servers to retrieve them, which is obviously not ideal to many.

MGibson1 commented 3 years ago

Hi @0xbkt, thanks for the question and interest! The short answers right now are No and No, you cannot use data.json as a replacement to bw export.

The decision was made when implementing vault export across all of Bitwarden to require a master password to export. With encrypted export and the CLI being used for automation on partially-trusted machines I think there's a good argument for relaxing that requirement for encrypted exports only. That work hasn't been done, though. If you'd like to, It would be awesome if you could create a feature request over on https://community.bitwarden.com.

As for data.json that is the lowdb database file we use to store all the persistent state information for bw. It does contain your vault, but the format and information is not guaranteed to be 1:1 to an export so it can't be used to import so it would not be useful as the export you're looking for.

berkant commented 3 years ago

@MGibson1 I feel like I can implement this myself. Does it need to be approved first so we can make sure it gets merged in the end?

MGibson1 commented 3 years ago

@0xbkt, That's great! I'd be happy to help if you need it. I'm sure you've found the export.command.ts file. I believe this is where all the changes you'll be making will be.

You don't need pre-approval to submit a PR, but most people look for it before working because it stinks to work on something that is just patently not going to be approved.

This is a feature we've discussed internally and would absolutely love to approve the right PR ;)

anthonygedeon commented 3 years ago

Any updates on this issue? I would like to tackle this if nobody is currently working on this.

MGibson1 commented 3 years ago

@anthonygedeon, I haven't heard anything about it since February, I think it's fair game if you want to work on it

cjemorton commented 2 years ago

This here, would be very nice. I am looking to setup automated backups of my data stored in bitwarden on my personal machine. Looks like unattended unencrypted full backups are not possible yet - in the context of what I was wanting to do.

I wanted to setup a simple cron that ran a task to pull down the latest everything. I was then going to encrypt it with my personal gpg public key and write it directly to DVD-RAM

Every time a backup fired off - I wanted to call a hook to my purshover service letting me know on my phone that the backup has been done. How much space I have left on the DVD-RAM, etc.

Then I can physically remove the disk say monthly/weekly, etc and physically store a copy somewhere else. When a DVD-RAM is full, burn to DVD-R & Archive.

I am getting more and more information in bitwarden; As logins and as notes, etc. I am looking for ways to preserve and protect this information, from accidental deletion on my part. Also in the unlikely case the service disappears.

Having the ability to "Roll Back" and look at "The State of things" at any given date by browsing the backups on that date, is something I would very much like to have.

I was thinking of implementing git into the backup chain and process as well. This would allow me to push everything automatically to a remote server after every scheduled backup. Also having unencrypted .json and bit warden backups managed in a git archive would allow me to utilize deduplication in my backups at the file level. Using Git in this way would allow me to easily see exactly what changed and at what point in time.

An "API-KEY" that allows downloading of encrypted backups in an "unattended way" and then a way to decrypt that downloaded json locally with the master passwords. From there I could implement whatever backup scheme I deem appropriate.

Just some thoughts on how I imagine setting up my backups. Looks like the "Cli" isn't quite ready yet for this.

EDIT: Using passwords passed to bw unlock and export I may have found a way to do this.

Alexander-Shukaev commented 2 years ago

Hi, I have a few questions related to login --apikey and subsequent unlock:

MGibson1 commented 2 years ago

Hi @Alexander-Shukaev. Thanks for your questions.

When login --apikey is done, where/how does bw cache that state?

The CLI uses a local lowdb cache to store state information. The location varies depending on the system and evironment

That is issuing login --apikey again would indeed report that You are already logged in as ...@.... How does it know?

Login in via api key still pulls down some basic information about the user and stores it in data.json. This includes email and userId as well as some crytographic settings.

I also notice that this works system-wide, i.e. if root user logged in

The only way this happens is if you share a the data.json file for any of the reasons in the link above. Most of the determination for the data directory is based on user-specific environment settings, but if both environments set BITWARDENCLI_APPDATA_DIR or if you're running from a directory with a bw-data directory in it, you would expect this behavior simply because the two users are using the same file to store state. No communication to Bitwarden servers is used to determine that You are already logged in as ...@....

What's the value of API key, if a subsequent unlock with the corresponding master passphrase is still required?

Actions not requiring decryption are available to a CLI logged in with API key, but not unlocked. Anything in the public api, for example -- as long as your API key has the proper scope.

API key is also used to bypass both Two-step Login and any potential captcha request. In this way, it's more usable for scripting or automation purposes even if you do need cryptographic access. The program flow is more predictable. If you use a valid API key and you can communicate with Bitwarden, you will always be logged in without further intervention. We don't make the same kind of guarantees with our login command. From time to time we change that flow. For example, we recently activated a possible extra authentication step if it appears a request is coming from a bot, but this would never have impacted an API key login (#383).

Alexander-Shukaev commented 2 years ago

@MGibson1, thanks, appreciate your detailed input, the intention is more clear to me now. The only point that still bothers me is exposing the master password to automated system, though I guess this is unavoidable with this architecture. How does rotating the API key protect one's data? Let's say one realized that the automated system got compromised, i.e. even the master password was exposed. Then rotating the API key would prevent any further logins, but the local copy on the automated system is still accessible and can be decrypted by attacker with obtained master password, correct? Then only the 2nd authentication stage is preventing attacker to also access web vault (until one changes master password as well of course)? In this case not only master password but also encryption key has to be rotated. I also noticed the following statement in documentation:

Making changes in a session with a “stale” encryption key will cause data corruption that will make your data unrecoverable.

Why cannot all the sessions be terminated automatically upon rotation? I think making this manual action is error-prone, especially if this will lead to data corruption. Upon any kind of rotation, be it API key, master password, and/or encryption key, I would expect unconditional termination of all active sessions to prevent any further infiltration and ensure that everything resets properly and all devices can reconcile the new state afterwards.

Alexander-Shukaev commented 2 years ago

By the way, for everyone monitoring this thread, I managed to implement fully automated backups of the vault including all attachments a couple of days ago, so I can confirm that whatever functionality is currently offered by bw CLI tool is enough to achieve the goal with some additional scripting. There were a few issues and/or bw deficiencies along the way, which I currently worked around, but I already reported them. The major one though, I want to mention here because it's both security and interface consistency concern (#378), which I am looking forward to be addressed the sooner the better.

fuxoft commented 2 years ago

Hello, I am the author of bitwarden/cli#91 which was recently closed and redirected here. I've read this whole thread and I am not sure whether I understand the discussion and whether the discussion is even about the identical topic that my request was. There seems to be a lot of mentions of "entering master passphrase" in this thread which is something I wanted to avoid in my original request (indeed that was the main point of my request). Please allow me to rephrase my question:

Is there a way (however complex) to create a script that periodically downloads and backs up an encrypted latest version of my Bitwarden vault which can be later used to restore the whole vault, all of this without the script having to contain my master password and without any user interaction? Of course, during the potential restoration, I would be using my master password if necessary. The restoration process does not have to be user friendly or automated, I will only use it when something extremely bad happens.

For example, can I somehow automatically log in using --apikey option and then back up the .config/Bitwarden\ CLI/data.json file and later use it to recover my whole vault in case of extreme failure (e.g. Bitwarden servers disappearing overnight)? https://github.com/bitwarden/clients/issues/2739 seems to suggest that this is not possible beacuse "information is not guaranteed to be 1:1 to an export" which I don't quite understand. I don't need a file in official "export format", I need any sort file that can somehow be used to restore my vault.

I have created an API key and logged into bw CLI using it but asking for encrypted_json export still asks me for my master passphrase so forgive me for being confused.

Currently I am periodically backing up all data in the directory of my Bitwarden Chrome extension but I am not even sure these are of any use if something bad happens.

Thanks and sorry for my ignorance, I am not a TypeScript programmer.

Alexander-Shukaev commented 2 years ago

Yes, there is a way. I did it, including attachments. I have to admit that it's rather convoluted and has lots of quirks due to either missing security features and lack of CLI inconsistency.

fuxoft commented 2 years ago

Yes, there is a way. I did it, including attachments. I have to admit that it's rather convoluted and has lots of quirks due to either missing security features and lack of CLI inconsistency.

OK, can you give some pointers how you did it?

Alexander-Shukaev commented 2 years ago

It took a while to figure all that stuff out and to refine it. Note that I stripped some irrelevant stuff out. If this works for you, tips for the beer are welcome :sweat_smile:, enjoy...

bw-export

user="${1-}"
dir="${2-}"
dir="${dir%/}"
attachments_dir="${dir:+${dir}/}attachments"
vault_file="${dir:+${dir}/}vault.json"

clientsecret="$(bw-clientsecret "${user}")" || exit "${?}"
    password="$(bw-password     "${user}")" || exit "${?}"
#   ^^^^^^^^- This is the absolute worst part of it, which can only be changed
#             when Bitwarden team finally reworks CLI.

filter="$(printf -- '%s\\u0000'                                              \
                    '\(.id)'                                                 \
                    '\($parent.id)'                                          \
                    "${attachments_dir}"'/\($parent.id)/\(.fileName)')"
filter="$(printf -- '%s | '                                                  \
                    '.[]'                                                    \
                    'select(.attachments != null)'                           \
                    '. as $parent'                                           \
                    '.attachments[]')\"${filter}\""

bw() {
  command bw --nointeraction "${@}"
}

bw_export() {
  command bw export --format json --output "${@}"
}

bw_login() {
  BW_CLIENTID="${1-}" BW_CLIENTSECRET="${2-}" bw login --apikey --raw
}

bw_session() {
  BW_SESSION="$(P="${1-}" bw unlock ${1+--passwordenv P} --raw)" &&
    export BW_SESSION
}

bw_sync() {
  bw sync "${@}"
}

bw_login "user.${user}" "${clientsecret}" || :
bw_session "${password}"                  || exit "${?}"
#             ^^^^^^^^- OMG WTF o_0
bw_sync                                   || exit "${?}"
bw_export "${vault_file}" "${password}"   || exit "${?}"
#                            ^^^^^^^^- OMG WTF o_0
bw list items | jq -j -r "${filter}" | xargs -0 -n 3 -P 2 -- bw-get-attachment

bw-get-attachment

bw() {
  command bw --nointeraction "${@}"
}

bw_get_attachment() {
  [ "${#}" -le 3 ]                    || return "${?}"
  [ "${#}" -lt 3 ] || rm -f -- "${3}" || return "${?}"
  bw get ${2+--itemid   "${2}"}                                              \
         ${3+--output   "${3}"}                                              \
         ${1+attachment "${1}"} && {
    [ "${#}" -lt 3 ] || [ -f "${3}" ]
  }
}

[ "${#}" -le 3 ] && bw_get_attachment "${@}"

bw-clientsecret: Implement your secret storage, e.g. I suggest getting it from an encrypted keyring or password manager. bw-password: Implement your secret storage, e.g. I suggest getting it from an encrypted keyring or password manager.

fuxoft commented 2 years ago

It took a while to figure all that stuff out and to refine it. Note that I stripped some irrelevant stuff out. If this works for you, tips for the beer are welcome sweat_smile, enjoy...

Thank you for this but I still don't understand how can your script succesfully invoke bw unlock --passwordenv if your script has no access to your master password?

Is this some secret undocumented feature of bw CLI?

Alexander-Shukaev commented 2 years ago

Not sure what your level with shell scripting is, but there is P="${1-}" in front and then it's --passwordenv P. The invocation is

bw_session "${password}"

so P gets contents of password. Password itself comes from

password="$(bw-password "${user}")"

There are no hidden features, it's mostly self-made. You have to come up with a way to securely feed password to a script. I gave you my suggestions on how the implementation of bw-password could look like to be secure enough. For instance, it could be as simple as using secret-tool to extract your master password from a system keyring.

fuxoft commented 2 years ago

Sorry, I am not very proficient in BASH scripting, I do most of my scripting in Lua.

so P gets contents of password. Password itself comes from

password="$(bw-password "${user}")"

So your solution assumes that the script has access to master password through some non-interactive way (e.g. by invoking secret-tool). So whoever has full access to your machine, he/she can find out your master password by running the same bw-password command your script is using?

If this is true then it seems you are solving different problem than the problem I have. I need my backup script to work on a machine which has no access to my master password in any form, however obscured - only to my API key. E.g. I need a script that logs into Bitwarden server using API key, downloads the encrypted vault, never decrypts it and backs it up somewhere. If my machine is compromised, the attacker knows my API key but still cannot decrypt my vault.

Alexander-Shukaev commented 2 years ago

So your solution assumes that the script has access to master password through some non-interactive way (e.g. by invoking secret-tool). So whoever has full access to your machine, he/she can find out your master password by running the same bw-password command your script is using?

Only if that somebody has unlocked my keyring (just running the script is not enough). It means the attacker must do that when the machine is running and have privileges for the user to whom this keyring belongs, and if for example, this is root user, then you get the point. The possibility of such attack is close to impossible on a well-secured Unix system. The only realistic way would be to perform a cold boot attack on a running machine and then scan the memory for the password in the unlocked keyring. If the keyring only unlocks for a fraction of a second just to retrieve the password and then locks again, it's EXTREMELY unlikely to retrieve the password even with a cold boot attack. When machine is offline, the keyring is encrypted at rest.

If this is true then it seems you are solving different problem than the problem I have. I need my backup script to work on a machine which has no access to my master password in any form, however obscured - only to my API key. E.g. I need a script that logs into Bitwarden server using API key, downloads the encrypted vault, never decrypts it and backs it up somewhere. If my machine is compromised, the attacker knows my API key but still cannot decrypt my vault.

I'm fully with you, and that would be indeed a great feature to have. Unfortunately, not possible at the moment. The fun part is that this feature is simpler to implement than all the other already existing features related to decrypted export. Inability to also download a complete bundle including attachments is also a pain in the ass to deal with. I'm happy at least that I was able to cover the automation part with decent level of security.

fuxoft commented 2 years ago

Only if that somebody has unlocked my keyring (just running the script is not enough).

Unfortunately, I need my backup script to run on a Linux server that 1) randomly restarts, even several times daily (and I cannot always log into it and unlock the keyring manually after restart), 2) is physically accesible by other people (so my master password must not be retrievable on it in any non-interactive way).

Thanks for your time anyway.

berkant commented 2 years ago

@fuxoft Here's the Actions workflow I have been using since mid-2021 to exactly do what you describe + upload to a remote storage: You can later restore offline once you put the data.json in the right place with the right Bitwarden version installed. I would recommend keeping an offline copy of that Bitwarden version somewhere in case it is disappears from NPM, GitHub, or such.

name: Export Bitwarden Vault
on:
  workflow_dispatch:
  schedule:
    - cron: '0 21 * * *'
jobs:
  export:
    runs-on: ubuntu-20.04
    steps:
      - run: sudo npm install -g @bitwarden/cli@1.19.1
      - run: bw login --apikey
        env:
          BW_CLIENTID: ${{ secrets.BW_CLIENTID }}
          BW_CLIENTSECRET: ${{ secrets.BW_CLIENTSECRET }}
      - run: curl https://rclone.org/install.sh | sudo bash
      - run: rclone copy --log-level=ERROR -P "$HOME/.config/Bitwarden CLI/data.json" :drive:Bitwarden_CLI_Backup/
        env:
          RCLONE_DRIVE_TOKEN: ${{ secrets.RCLONE_DRIVE_TOKEN }}
Logs from last pipeline run: ``` 2022-02-04T21:09:17.8972315Z Requested labels: ubuntu-20.04 2022-02-04T21:09:17.8972425Z Job defined at: berkant/***/.github/workflows/cron.yml@refs/heads/main 2022-02-04T21:09:17.8972452Z Waiting for a runner to pick up this job... 2022-02-04T21:09:18.4404467Z Job is waiting for a hosted runner to come online. 2022-02-04T21:09:24.0737257Z Job is about to start running on the hosted runner: Hosted Agent (hosted) 2022-02-04T21:09:27.3401296Z Current runner version: '2.287.1' 2022-02-04T21:09:27.3427973Z ##[group]Operating System 2022-02-04T21:09:27.3428605Z Ubuntu 2022-02-04T21:09:27.3429080Z 20.04.3 2022-02-04T21:09:27.3429378Z LTS 2022-02-04T21:09:27.3429765Z ##[endgroup] 2022-02-04T21:09:27.3430177Z ##[group]Virtual Environment 2022-02-04T21:09:27.3430685Z Environment: ubuntu-20.04 2022-02-04T21:09:27.3431111Z Version: 20220131.1 2022-02-04T21:09:27.3431773Z Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20220131.1/images/linux/Ubuntu2004-Readme.md 2022-02-04T21:09:27.3432558Z Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20220131.1 2022-02-04T21:09:27.3433128Z ##[endgroup] 2022-02-04T21:09:27.3433614Z ##[group]Virtual Environment Provisioner 2022-02-04T21:09:27.3434085Z 1.0.0.0-main-20220201-1 2022-02-04T21:09:27.3434472Z ##[endgroup] 2022-02-04T21:09:27.3435613Z ##[group]GITHUB_TOKEN Permissions 2022-02-04T21:09:27.3436427Z Actions: write 2022-02-04T21:09:27.3436921Z Checks: write 2022-02-04T21:09:27.3437383Z Contents: write 2022-02-04T21:09:27.3437852Z Deployments: write 2022-02-04T21:09:27.3438365Z Discussions: write 2022-02-04T21:09:27.3438737Z Issues: write 2022-02-04T21:09:27.3439043Z Metadata: read 2022-02-04T21:09:27.3439441Z Packages: write 2022-02-04T21:09:27.3439818Z Pages: write 2022-02-04T21:09:27.3440188Z PullRequests: write 2022-02-04T21:09:27.3440548Z RepositoryProjects: write 2022-02-04T21:09:27.3440963Z SecurityEvents: write 2022-02-04T21:09:27.3441389Z Statuses: write 2022-02-04T21:09:27.3441756Z ##[endgroup] 2022-02-04T21:09:27.3445659Z Secret source: Actions 2022-02-04T21:09:27.3446152Z Prepare workflow directory 2022-02-04T21:09:27.4298673Z Prepare all required actions 2022-02-04T21:09:27.5590027Z ##[group]Run sudo npm install -g @bitwarden/cli@1.19.1 2022-02-04T21:09:27.5590594Z sudo npm install -g @bitwarden/cli@1.19.1 2022-02-04T21:09:27.6234596Z shell: /usr/bin/bash -e {0} 2022-02-04T21:09:27.6234987Z ##[endgroup] 2022-02-04T21:09:31.3477608Z npm WARN EBADENGINE Unsupported engine { 2022-02-04T21:09:31.3479837Z npm WARN EBADENGINE package: '@bitwarden/cli@1.19.1', 2022-02-04T21:09:31.3480505Z npm WARN EBADENGINE required: { node: '~14', npm: '~7' }, 2022-02-04T21:09:31.3481387Z npm WARN EBADENGINE current: { node: 'v16.13.2', npm: '8.1.2' } 2022-02-04T21:09:31.3482139Z npm WARN EBADENGINE } 2022-02-04T21:09:34.5301523Z 2022-02-04T21:09:34.5303021Z added 117 packages, and audited 118 packages in 7s 2022-02-04T21:09:34.5304190Z 2022-02-04T21:09:34.5305788Z 8 packages are looking for funding 2022-02-04T21:09:34.5306775Z run `npm fund` for details 2022-02-04T21:09:34.5362820Z 2022-02-04T21:09:34.5363212Z 1 low severity vulnerability 2022-02-04T21:09:34.5363808Z 2022-02-04T21:09:34.5364127Z To address all issues (including breaking changes), run: 2022-02-04T21:09:34.5364890Z npm audit fix --force 2022-02-04T21:09:34.5365202Z 2022-02-04T21:09:34.5365423Z Run `npm audit` for details. 2022-02-04T21:09:34.5647907Z ##[group]Run bw login --apikey 2022-02-04T21:09:34.5648319Z bw login --apikey 2022-02-04T21:09:34.5700440Z shell: /usr/bin/bash -e {0} 2022-02-04T21:09:34.5700908Z env: 2022-02-04T21:09:34.5701559Z BW_CLIENTID: *** 2022-02-04T21:09:34.5702010Z BW_CLIENTSECRET: *** 2022-02-04T21:09:34.5702349Z ##[endgroup] 2022-02-04T21:09:35.4278019Z Could not find dir, "/home/runner/.config/Bitwarden CLI"; creating it instead. 2022-02-04T21:09:35.4287513Z Could not find data file, "/home/runner/.config/Bitwarden CLI/data.json"; creating it instead. 2022-02-04T21:09:41.4827944Z You are logged in! 2022-02-04T21:09:41.4828596Z 2022-02-04T21:09:41.4830507Z To unlock your vault, use the `unlock` command. ex: 2022-02-04T21:09:41.5090295Z $ bw unlock 2022-02-04T21:09:41.5111985Z ##[group]Run curl https://rclone.org/install.sh | sudo bash 2022-02-04T21:09:41.5112442Z curl https://rclone.org/install.sh | sudo bash 2022-02-04T21:09:41.5166853Z shell: /usr/bin/bash -e {0} 2022-02-04T21:09:41.5167170Z ##[endgroup] 2022-02-04T21:09:41.5365354Z % Total % Received % Xferd Average Speed Time Time Time Current 2022-02-04T21:09:41.5367599Z Dload Upload Total Spent Left Speed 2022-02-04T21:09:41.5367961Z 2022-02-04T21:09:41.9690196Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2022-02-04T21:09:41.9699091Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2022-02-04T21:09:41.9706364Z 100 4497 100 4497 0 0 10361 0 --:--:-- --:--:-- --:--:-- 10337 2022-02-04T21:09:43.5053871Z Archive: rclone-current-linux-amd64.zip 2022-02-04T21:09:43.5055073Z creating: tmp_unzip_dir_for_rclone/rclone-v1.57.0-linux-amd64/ 2022-02-04T21:09:43.5174729Z inflating: tmp_unzip_dir_for_rclone/rclone-v1.57.0-linux-amd64/README.txt [text] 2022-02-04T21:09:43.5183549Z inflating: tmp_unzip_dir_for_rclone/rclone-v1.57.0-linux-amd64/git-log.txt [text] 2022-02-04T21:09:43.5320878Z inflating: tmp_unzip_dir_for_rclone/rclone-v1.57.0-linux-amd64/rclone.1 [text] 2022-02-04T21:09:43.5465628Z inflating: tmp_unzip_dir_for_rclone/rclone-v1.57.0-linux-amd64/README.html [text] 2022-02-04T21:09:43.9233189Z inflating: tmp_unzip_dir_for_rclone/rclone-v1.57.0-linux-amd64/rclone [binary] 2022-02-04T21:09:44.2618565Z Updating index cache for path `/usr/local/share/man/man1'. Wait... 2022-02-04T21:09:44.2619487Z Purging old database entries in /usr/share/man... 2022-02-04T21:09:44.2620087Z Processing manual pages under /usr/share/man... 2022-02-04T21:09:44.2675622Z Purging old database entries in /usr/share/man/nl... 2022-02-04T21:09:44.2676043Z Processing manual pages under /usr/share/man/nl... 2022-02-04T21:09:44.2676436Z Purging old database entries in /usr/share/man/de... 2022-02-04T21:09:44.2676919Z Processing manual pages under /usr/share/man/de... 2022-02-04T21:09:44.2677276Z Purging old database entries in /usr/share/man/tr... 2022-02-04T21:09:44.2677629Z Processing manual pages under /usr/share/man/tr... 2022-02-04T21:09:44.2677984Z Purging old database entries in /usr/share/man/da... 2022-02-04T21:09:44.2678327Z Processing manual pages under /usr/share/man/da... 2022-02-04T21:09:44.2678654Z Purging old database entries in /usr/share/man/pl... 2022-02-04T21:09:44.2679008Z Processing manual pages under /usr/share/man/pl... 2022-02-04T21:09:44.2679355Z Purging old database entries in /usr/share/man/id... 2022-02-04T21:09:44.2679696Z Processing manual pages under /usr/share/man/id... 2022-02-04T21:09:44.2680037Z Purging old database entries in /usr/share/man/fr... 2022-02-04T21:09:44.2680361Z Processing manual pages under /usr/share/man/fr... 2022-02-04T21:09:44.2680701Z Purging old database entries in /usr/share/man/hu... 2022-02-04T21:09:44.2681049Z Processing manual pages under /usr/share/man/hu... 2022-02-04T21:09:44.2681395Z Purging old database entries in /usr/share/man/es... 2022-02-04T21:09:44.2681738Z Processing manual pages under /usr/share/man/es... 2022-02-04T21:09:44.2682160Z Purging old database entries in /usr/share/man/pt... 2022-02-04T21:09:44.2682445Z Processing manual pages under /usr/share/man/pt... 2022-02-04T21:09:44.2682750Z Purging old database entries in /usr/share/man/sl... 2022-02-04T21:09:44.2683050Z Processing manual pages under /usr/share/man/sl... 2022-02-04T21:09:44.2683350Z Purging old database entries in /usr/share/man/it... 2022-02-04T21:09:44.2683828Z Processing manual pages under /usr/share/man/it... 2022-02-04T21:09:44.2684150Z Purging old database entries in /usr/share/man/ko... 2022-02-04T21:09:44.2684485Z Processing manual pages under /usr/share/man/ko... 2022-02-04T21:09:44.2684835Z Purging old database entries in /usr/share/man/zh_CN... 2022-02-04T21:09:44.2685424Z Processing manual pages under /usr/share/man/zh_CN... 2022-02-04T21:09:44.2686326Z Purging old database entries in /usr/share/man/fi... 2022-02-04T21:09:44.2686678Z Processing manual pages under /usr/share/man/fi... 2022-02-04T21:09:44.2687010Z Purging old database entries in /usr/share/man/sr... 2022-02-04T21:09:44.2687357Z Processing manual pages under /usr/share/man/sr... 2022-02-04T21:09:44.2687837Z Purging old database entries in /usr/share/man/cs... 2022-02-04T21:09:44.2688178Z Processing manual pages under /usr/share/man/cs... 2022-02-04T21:09:44.2688575Z Purging old database entries in /usr/share/man/sv... 2022-02-04T21:09:44.2688899Z Processing manual pages under /usr/share/man/sv... 2022-02-04T21:09:44.2689251Z Purging old database entries in /usr/share/man/zh_TW... 2022-02-04T21:09:44.2689614Z Processing manual pages under /usr/share/man/zh_TW... 2022-02-04T21:09:44.2689967Z Purging old database entries in /usr/share/man/ru... 2022-02-04T21:09:44.2690313Z Processing manual pages under /usr/share/man/ru... 2022-02-04T21:09:44.2690659Z Purging old database entries in /usr/share/man/ja... 2022-02-04T21:09:44.2690983Z Processing manual pages under /usr/share/man/ja... 2022-02-04T21:09:44.2691333Z Purging old database entries in /usr/share/man/pt_BR... 2022-02-04T21:09:44.2691688Z Processing manual pages under /usr/share/man/pt_BR... 2022-02-04T21:09:44.2692038Z Purging old database entries in /usr/local/man... 2022-02-04T21:09:44.2692550Z Processing manual pages under /usr/local/man... 2022-02-04T21:09:44.2692909Z Purging old database entries in /usr/local/share/man... 2022-02-04T21:09:44.2693287Z Processing manual pages under /usr/local/share/man... 2022-02-04T21:09:44.2997288Z done. 2022-02-04T21:09:44.3034352Z Checking for stray cats under /usr/local/share/man... 2022-02-04T21:09:44.3034797Z Checking for stray cats under /var/cache/man/local... 2022-02-04T21:09:44.3035197Z 1 man subdirectory contained newer manual pages. 2022-02-04T21:09:44.3035541Z 1 manual page was added. 2022-02-04T21:09:44.3035855Z 0 stray cats were added. 2022-02-04T21:09:44.3036181Z 0 old database entries were purged. 2022-02-04T21:09:44.3551126Z 2022-02-04T21:09:44.3557504Z rclone v1.57.0 has successfully installed. 2022-02-04T21:09:44.3562907Z Now run "rclone config" for setup. Check https://rclone.org/docs/ for more details. 2022-02-04T21:09:44.3563457Z 2022-02-04T21:09:44.3585904Z ##[group]Run rclone copy --log-level=ERROR -P "$HOME/.config/Bitwarden CLI/data.json" :drive:Bitwarden_CLI_Backup/ 2022-02-04T21:09:44.3586519Z rclone copy --log-level=ERROR -P "$HOME/.config/Bitwarden CLI/data.json" :drive:Bitwarden_CLI_Backup/ 2022-02-04T21:09:44.3637790Z shell: /usr/bin/bash -e {0} 2022-02-04T21:09:44.3638048Z env: 2022-02-04T21:09:44.3639430Z RCLONE_DRIVE_TOKEN: *** 2022-02-04T21:09:44.3639702Z ##[endgroup] 2022-02-04T21:09:45.3487218Z Transferred: 185.202 KiB / 185.202 KiB, 100%, 0 B/s, ETA - 2022-02-04T21:09:45.3488448Z Transferred: 0 / 1, 0% 2022-02-04T21:09:45.3489663Z Elapsed time: 0.9s 2022-02-04T21:09:45.3490200Z Transferring: 2022-02-04T21:09:45.8489359Z * data.json:100% /185.202Ki, 0/s, -Transferred: 185.202 KiB / 185.202 KiB, 100%, 0 B/s, ETA - 2022-02-04T21:09:45.8490087Z Transferred: 0 / 1, 0% 2022-02-04T21:09:45.8490593Z Elapsed time: 1.4s 2022-02-04T21:09:45.8491174Z Transferring: 2022-02-04T21:09:46.1027929Z * data.json:100% /185.202Ki, 0/s, -Transferred: 185.202 KiB / 185.202 KiB, 100%, 185.015 KiB/s, ETA 0s 2022-02-04T21:09:46.1029395Z Transferred: 1 / 1, 100% 2022-02-04T21:09:46.1033527Z Elapsed time: 1.7s 2022-02-04T21:09:46.1222277Z Cleaning up orphan processes ```
Alexander-Shukaev commented 2 years ago

@berkant, the implementation details of this file might change in the future as far as I understood, so it's not guaranteed to work forever. Additionally, I guess this method does not take into account attachments. Thus, still looking forward to proper out-of-the-box solution.

MGibson1 commented 2 years ago

My mistake, I misread this when attaching bitwarden/cli#446. This is not yet implemented

TomiBelan commented 2 years ago

I looked into this issue and I think I understand it pretty well now. But I'm just a user so I welcome corrections from the Bitwarden team.

Q: Why does bw export --format=encrypted_json even need the master password?

A: In old CLI versions, bw export didn't truly need it. It asked for the password but didn't actually use it for anything. I have an old patched build of bw 1.15.1 which just deletes the password prompt code from export.command.ts and vault.program.ts, and it used to work just fine.

Nowadays it's not so simple. If I try to delete the relevant exitIfLocked() call, bw export crashes with Cannot read properties of null (reading 'encKey'). This is because of bitwarden/jslib#369, which added a new field named encKeyValidation_DO_NOT_EDIT to the encrypted_json format. Everything else in the encrypted_json output could be produced without having the master password, only this single field is an obstacle.

The field contains a random GUID encrypted with the user's account encryption key. It is used as an integrity check during import, to verify that the account encryption key still matches. Without this check, importing an encrypted_json file to a different account (or same account after rotating the account encryption key) could lead to silent corruption. See also Encrypted Exports docs.

Q: How to change bw export to allow using it without the master password?

A: One possible idea could be to add a new option --without-integrity-check which omits the encKeyValidation_DO_NOT_EDIT field from the output. Such json files are valid because the import command simply skips the key validation if the field is missing. But it increases the risk of issues from account encryption key rotation, so this mode shouldn't be the default for normal users when they make encrypted_json backups manually.

Then the code could be modified so that if --without-integrity-check is present, and format equals encrypted_json, and the --password option was not given, bw export won't ask for the master password and won't unlock the vault.

I could write a PR if there is interest, but I don't like this idea very much.

Q: Is it even desirable to generate encrypted_json exports automatically and use them as backups?

A: I'm not so sure. Even if you could run bw export without a master password prompt, the output is still tied to your current account encryption key. Per Encrypted Exports docs, the file can't be imported into a different account, and it'll stop working if you ever rotate the account encryption key.

By contrast, the data.json file is self-contained. Even if all your devices and all Bitwarden servers explode in the same instant, as long as you know your master password and you have a backup of data.json, you'll be able to get your passwords back. And although technically data.json is "not guaranteed to be 1:1 to an export" (e.g. you can't bw import it directly into another Bitwarden account), you can run BITWARDENCLI_APPDATA_DIR=/path/to/backup bw export --type=json manually during restore.

Of course, your choice may depend on your threat model and whether this is something you want.

(Explanation: The account encryption key is a random number initially generated when you register. Bitwarden client encrypts it with your master password and saves the encrypted form in data.json and on Bitwarden servers. It is not saved in encrypted_json. When you rotate it, Bitwarden generates a new random number, re-encrypts all saved passwords with it, and saves the new number encrypted with your new master password in data.json and on Bitwarden servers.)

Q: What about attachments?

A: Sorry, I don't know anything about them.

Alexander-Shukaev commented 2 years ago

@TomiBelan, I guess in this case, in order for encrypted export feature to remain reasonably portable, it's unavoidable to also export the encryption key (in the encrypted from of course) as a separate file or maybe as part of encrypted JSON itself. So the only protection left would be the master password, which should be strong anyway regardless of this feature. Additionally, it probably makes sense to have a separate CLI sub-command that instead of directly importing encrypted JSON, merely decrypts it and writes decrypted JSON with all the vault data to standard output. This sounds portable enough to me, reducing vendor lock on Bitwarden infrastructure, except for encryption/decryption algorithm.

TomiBelan commented 2 years ago

Right, that matches my understanding. Personally, my conclusion is the same as berkant's. I just make backups of the data.json file. I want a format that can be decrypted completely offline using only my master password, even if Bitwarden servers are down or if I lose access to my account, and data.json is exactly that.

In my case I'm using bw sync instead of bw login --apikey, but that's up to personal preference. Either way it works without the master password or system keyring. bw sync doesn't need the password since bw 2022.9.0 as long as you're logged in.

My backup script Keep in mind that this script does not handle Bitwarden attachments! ```bash #!/bin/bash set -e IFS=$'\n' here=$(dirname "$0") export BITWARDENCLI_APPDATA_DIR="$here/appdata" today=$(date +%Y-%m-%d) # Keep last 50 backups rm -rf $(printf "%s\n" "$here"/backups/* | sort | head -n-50) if ! [[ -e "$here/backups/$today" ]]; then # This is just a hack to get rid of the "Syncing complete." success message, but preserve any errors. if out=$(bw sync 2>&1); then true else res=$? echo "bw sync: failed with $res" >&2 echo "$out" >&2 exit $res fi mkdir "$here/backups/$today" cp -a "$here/appdata/"* "$here/backups/$today/" fi ```

For data.json-based backups, you could say "bw export" already fulfills the role of "separate CLI sub-command that instead of directly importing encrypted JSON, merely decrypts it". That's another advantage over format=encrypted_json.