Vulnerability: Sensitive information is not purged from process memory on app lock or logout

[bwbug]

Steps To Reproduce

1. Ensure that the vault contains secret items (e.g., logins, secure notes, etc.), and that "Close to Tray icon" is enabled; to minimize known risks, also ensure that vault timeout is not set to "Never", and that PIN locking is disabled.
2. Log out from vault on Bitwarden client app, then fully exit the app using File > Quit Bitwarden.
3. Launch a memory inspector/editor utility. I used HxD.
4. Launch Bitwarden client app. My results shown below are for the Desktop app, but I have confirmed that the same vulnerability exists in the Chrome browser extension, as well as in the web vault.
5. In the memory inspector utility, open the process memory for the Bitwarden client app. (In HxD, go to Tools > Open main memory). The Bitwarden desktop app has four processes, but only one of those will contain the decrypted vault and master password; you may need to repeat the memory searches described below in all four memory regions to find the correct process, but in my experience it is typically the process at the top of the list that contains the information we are looking for.
6. In the Bitwarden client app, perform the various actions described in the results section below (e.g., type master password on login screen, log in, lock, log out); after each action, click the "X" in the top right corner of the Bitwarden app to "close" the app to the systray, then bring another app into focus (e.g., the memory inspector app), and wait 10 seconds or longer (up to 4 hours in my tests) before proceeding with the memory search described in Steps 7-8 below.
7. In the memory inspector utility, refresh the view of the process memory. (In HxD, go to View > Refresh).
8. In the memory inspector utility, search the process memory for strings of interest, such as the master password, or contents of known vault items. (In HxD, go to Search > Find > Text-string, enter the search string, set the search direction to "All", and click Search All). Inspect the search results.
9. Repeat Steps 6-8 for various states of the vault.

Expected Result

Based on Bitwarden documentation cited below, I expect the master password to either never be stored in process memory ("We do not keep the Master Password stored locally or in memory on the Bitwarden Client" [2][3]), or at worst, be stored only temporarily until it is no longer needed ("The Master Password is cleared from memory after usage" [1])-- i.e., after calculation of the Master Key and Master Password Hash.

Furthermore, I expect all vault data (i.e., decrypted secrets) to be to be cleared from process memory either immediately upon locking the vault ("memory is cleaned up whenever the application is locked" [2],[3]), or at worst, within 10 seconds after locking the vault ("We also reload the application's renderer process after 10 seconds of inactivity on the lock screen to make sure any managed memory addresses which have not yet been garbage collected are purged" [3]).

Obviously, the same expectations would also hold when one has logged out of an account, which is supposed to be even more safe than locking the vault ("Logging out of your vault completely removes all vault data from your device" [4]).

Sources:

[1] From the Bitwarden Security Whitepaper:

The Master Password is cleared from memory after usage

[2] From the Bitwarden Security Whitepaper:

We do not keep the Master Password stored locally or in memory on the Bitwarden Client. Your encryption key (Symmetric Key) is kept in memory while the app is unlocked. This is needed to decrypt data in your Vault. When the Vault is locked, this data is purged from memory. After a certain time frame of inactivity on lock screen, we reload the application processes to make sure that any leftover managed memory addresses are also purged. We do our best to ensure that any data that may be in memory for the application to function is only held in memory for as long as you need it and that memory is cleaned up whenever the application is locked. We consider the application to be completely safe while in a locked state.

[3] From Bitwarden's Security FAQ:

Q: Is my Bitwarden master password stored locally? A: No. We do not keep the master password stored locally or in memory. Your encryption key (derived from the master password) is kept in memory only while the app is unlocked, which is required to decrypt data in your vault. When the vault is locked, this data is purged from memory.

We also reload the application's renderer process after 10 seconds of inactivity on the lock screen to make sure any managed memory addresses which have not yet been garbage collected are purged. We do our best to ensure that any data that may be in memory for the application to function is only held in memory for as long as you need it and that memory is cleaned up whenever the application is locked. We consider the application's encrypted data to be completely safe while the application is in a locked state.

[4] From Bitwarden Online Help:

Log Out Logging out of your vault completely removes all vault data from your device.

Actual Result

• Multiple instances of the clear-text master password are preserved in the process memory after locking the vault and even after logging out. In the two accounts that I have tested, one can find the master password even if it is not known in advance, by searching for all instances of the string {"nbf":, and examing the 120 bytes (give or take) than precede each occurrence of the {"nbf": string.

• Decrypted vault secrets are preserved in clear-text in process memory after locking the vault and even after logging out.

• The only way for the user to ensure that sensitive information is cleared from local memory is to manually terminate the Bitwarden process. Arguably, quitting the app while the vault is unlocked is safer than leaving the app running in the background while the vault is logged out.

Screenshots or Videos

The vault that was used in the tests below had a master password CorrectHorseBatteryStaple!@#; for privacy reasons, screenshots that reveal the account email have been retouched to obfuscate the email address (so that it is displayed here as dummy@dummy.com). The vault stored a login item named My Bank that contained the username mybankaccount and password Password82Brut3F0rc3, as well as a secure note titled Important Info with the text Secret PIN is 42.

The screenshot in Fig. 1 below shows 4 instances of the master password kept in memory after the password has been entered on the app login screen, but before completing the login process (by hitting the "Enter" key or clicking the "Log In" button). It seems that it would be unavoidable to keep one copy of the text entered into the keyboard buffer, at least until the login process has been completed, so this observation is mentioned primarily for completeness.

Fig. 1. Stored master password prior to login.

Fig. 2 shows 11 instances of the full master password found after the login process had completed, including the original 4 instances (from Fig. 1), augmented by 7 additional copies of the master password. Thus, not only does the master password persist in memory past the point when its plain-text value is needed (because the Master Key and Master Password Hash have been already computed during login), but the number of instances of the plain-text master password has almost tripled. At least one of the copies of the master password is stored in close vicinity to a memory location holding the account email address.

Fig. 2. Stored master password (and account email) after login.

The more concerning findings begin after locking the vault. As shown in Fig. 3, as many as 5 copies of the plain-text master password are still discoverable in process memory after the vault has been locked (even if inactive for more than 10 seconds). Again, the account email is found juxtaposed to at least one instance of the stored master password. Furthermore, the decrypted vault secrets (including item names, login usernames, login passwords, and secure notes) are also still present in memory in clear-text, even though the vault has been locked (see Figs. 4-6).

Fig. 3. Stored master password (and account email), visible in memory after the vault has been locked and inactive for 10 seconds or more.

Fig. 4. Decrypted login username, visible in memory after the vault has been locked and inactive for 10 seconds or more.

Fig. 5. Decrypted login password, visible in memory after the vault has been locked and inactive for 10 seconds or more.

Fig. 6. Decrypted secure note, visible in memory after the vault has been locked and inactive for 10 seconds or more.

After logging out of the locked dummy@dummy.com account (and allowing for a further 10 seconds or more of inactivity), it appears that the Bitwarden client app does not make any further attempts to purge sensitive information from memory. As shown in Figs. 7-10, the memory locations that had contained sensitive information in the locked state of the vault (Figs. 3-6) still contain the same information after logging out of the vault.

Fig. 7. Stored master password (and account email), visible in memory after the vault has been logged out and inactive for 10 seconds or more.

Fig. 8. Decrypted login username, visible in memory after the vault has been logged out and inactive for 10 seconds or more.

Fig. 9. Decrypted login password, visible in memory after the vault has been logged out and inactive for 10 seconds or more.

Fig. 10. Decrypted secure note, visible in memory after the vault has been logged out and inactive for 10 seconds or more.

In some tests, the Bitwarden client app has been left inactive for up to 4 hours after logging out, and the memory has still not been purged. It appears that the only method to ensure the plain-text master password and decrypted vault are cleared from memory is to terminate the app's process.

Additional Context

This is basically a demonstration that the vulnerabilities described in the 2019 ISE report Password Managers: Under the Hood of Secrets Management still exist in Bitwarden in 2022. Interestingly, a 2019 Reddit post claims that the memory is in fact cleared by the Bitwarden desktop app, but this claim is contradicted by the findings I have reported above. Perhaps the code base has changed since 2019.

The issues I have reported for the desktop client app also exist for the Chrome browser extension and for the web vault.

If the ability to scrub memory is limited due to programming language or framework, then at least the official Bitwarden documentation should not be making misleading statements about the memory management capabilities.

I would suggest an interim work-around of providing an option to have the vault timeout action include automatic termination of the running Bitwarden process. In addition, I recommend that the option "Close to Tray icon" be accompanied by a security warning.

Operating System

Windows

Operating System Version

Windows 10 (20H2 Build 19042.1826)

Installation method

Direct Download (from bitwarden.com)

Build Version

2022.6.2

[programme-zero]

After performing the same steps on Windows 10 with Bitwarden version 2022.6.2 Desktop I was able to access sensitive data in memory more than 1 minute after putting the vault in its locked state.

[bwbug]

@programme-zero Are you also able to retrieve a previously unknown master password by searching for instances of {"nbf": and examining the memory regions just before the occurrences of that string (i.e., can you find your master password by examining those regions when the vault is locked)? I wasn't sure if that was a coincidence in my case, or a general finding.

[programme-zero]

@bwbug I just tested it with the string {"nbf": and it leads to a data structure containing various informations including the account email and the master password. Of course all of this after locking the vault.

[bwbug]

@programme-zero Thank you for checking. In my case, I have to look in the 100 or so bytes that precede each instance of {"nbf":, and I will always find the master password in one of those regions (there is typically only a handful of instances to examine, so this would be a very efficient way for somebody to steal the master password from an unattended computer, or using exploits that can scan the computer memory).

[kspearrin]

Thanks for the report @bwbug . This seems like a regression around our process reload functionality on logout/lock. The team is looking into it and will follow back up.

[bwbug]

Thank you, @kspearrin -- I had suspected a regression given the language in the documentation about process reloading (which doesn't seem to work in the current releases).

Not sure if this is possible in Bitwarden, but in response to the 2019 ISE report, KeePassXC touted their use of "modern Windows memory security techniques" that prevent their memory contents from being read by non-elevated processes (and also prevent memory dumps):

KeePassXC uses modern Windows memory security techniques available to all processes. None of the other password managers featured in the ISE report have implemented this security. If they had, the ISE attacks would have failed outright! We specifically disable reading the memory of KeePassXC. (Note: it is not possible to prevent an administrator from accessing memory) We also disable "core dumps" which can expose secrets if the application crashes.

A related discussion on SourceForge reveals that these "memory security techniques" involve assignment of a DACL to the KeePassXC process, but that this carries a risk of incompatibility issues.

If anything further can be done to make the memory used by Bitwarden processes more secure from unauthorized access (without compromising stability), that would be welcome. Restoring the lost process reload functionality should be a great start!

[dbosompem]

Hi @bwbug, an update: the team has reproduced this internally and we are working towards a resolution with the highest priority. @djsmith85 is leading this effort and we will update once we have confirmed a fix date.

[djsmith85]

@bwbug Thank you for bringing this to our attention and the detailed report.

The PR linked here, solves the issue with the process reload not executing. Which addresses the points 3-10 in your report.

In case of points 1 and 2 I am still investigating if we can free up more instances.

Point 1: As you mentioned, this is expected as it needs to be in memory after entering it.

Point 2: This has partially to do how forms with submit work and the browser/runtime tends to hang onto things for longer until the GC clears them up. A process reload to wipe memory, would not work in this case, as this would lock/log you out immediately. I've created a ticket on our internal ticket tracking, to investigate if we can improve the handling/clearing of memory after unlocking. I'll link a PR to this issue, if we find a way to improve this.

Kind regards,

Daniel

[bwbug]

@djsmith85 Thank you for the update, and for your work on this.

[dbosompem]

@bwbug , an update on this PR. An initial work had been done by @djsmith85 as we sought to solve the issue of process reload clearing memory. However as it passes through QA with a number of test cases, looks like the master password occasionally persists in memory in some clients. To make sure we have a consistent behaviour across all clients, the team is taking a further look into this, so for transparency sake I am reopening this issue till we fully resolve the problem. Rest assured, the team would continue to investigate till we arrive at a resolution.

[bwbug]

Thank you for the update.

[dbosompem]

Hi y'all, please find below some detailed updates: To decrypt the vault, the user needs to enter the master password which then is temporarily resident in memory. Bitwarden cannot clear the master password from memory directly as the application cannot perform any direct memory management. The mechanism that is used to clear the master password from memory is to reload application processes that will then purge any leftover managed memory addresses. The linked PR by @djsmith85 addresses several instances where the reload was not functioning properly or at all. The frequency of this occurrence has decreased, yet the team is still able to identify uncleared memory in certain clients. We are continuing to investigate why this is happening with guidance from our founder and CTO, @kspearrin. However, as memory management and process reload is complex, this may take additional time. To be clear, this could only affect someone with an already compromised system. We will continue to publish updates as we learn more.

[bwbug]

@dbosompem Thank you for the update.

The linked PR by @djsmith85 addresses several instances where the reload was not functioning properly or at all.

I'm not fully familiar with the PR processes -- have these (partial) fixes found their way into any of the publicly released client binaries?

[dbosompem]

@bwbug the partial fixes would be included in the upcoming release.

[TheNightRider12]

Just commenting to see where the progress is on this bug. As I see that it is still opened. Has this been fully resolved? Or still In-Progress?

[dbosompem]

Hi @TheNightRider12 , this is still in progress, although the partial fixes have been released. The team is currently looking into ways to immediately clear master password after the process reload, and handle the inconsistent results in terms of making sure everything is wiped. This is still on our radar, and we will be sure to update as soon as we discover more.

[TheNightRider12]

Awesome. Thank you @dbosompem for the update to the issue.

[dragani]

I tested this with the latest Bitwarden Chrome extension version (2022.12.1) and I was able to reproduce the point 3 (didn't have time to test them all). The master password was still present in memory after locking the vault and waiting for at least 10 seconds ('Lock with master password on restart' was enabled).

Ironically, contrary to what the *help page hints, sensitive data (PIN and vault entries) was purged from memory successfully when I had 'Lock with master password on restart' disabled. I could actually see the extension icon flash as the process was purged from memory a few seconds afted I had clicked 'Lock Now'.

*https://bitwarden.com/help/unlock-with-pin/ If you turn off the Lock with master password on restart option, the Bitwarden application may not fully purge sensitive data from application memory when entering a locked state. If you are concerned about your device's local memory being compromised, you should keep the Lock with master password on restart option turned on.

[JamesBelchamber]

Someone seems to have built a tool for exploiting this now:

• https://redmaple.tech/blogs/2023/extract-bitwarden-vault-passwords/
• https://github.com/markuta/bw-dump

[dbosompem]

@JamesBelchamber thanks. As an update, the team is continuously looking into this issue as it has been on our radar. We are exploring ways to tackle this and the community would be updated in due time. Thank you for your patience!

[Hinton]

Hey,

Quick update on this, we've identified a solution that got merged in #5813 which involves destroying the renderer process in electron using forcefullyCrashRenderer. During our internal testing we can no longer reproduce this issue and we're looking into getting it released as a fix in the near future.