New user (recovered user) can not create and share password

jiansong commented 4 years ago

(This issue has been edited, to provide more details, also to correct its scope, in the beginning we thought it mostly happens on older/slower laptop, with more extended tests, turns out it's actually reproducible on every laptop)

On 2019-10-23 (the next day of the Firefox 70 release), we had the first case of creating new password via the web UI stopped working.

Since then we had several more cases, and noticed it only affects newly created users and users did account recovery on another computer or browser.

For existing users, as long as they stay on original computer / browser, everything is working fine.

The symptom

When creating new password via the web UI, it hangs with the pop-up message:

Encrypting ... Take a deep breath and enjoy being in the present moment ...

When sharing a password with other user, it shows an error message:

Error Misformed armored text

plus error message in the Console (of browser dev tool):

error {name: "Error", message: "Misformed armored text"} (anonymous) @ share.js:55 Promise.then (async) submitChanges @ share.js:54 (anonymous) @ share.js:90 dispatch @ jquery.js:5237 elemData.handle @ jquery.js:5044

API server version

We were running the API server 2.8.3 when the first case appears, then we upgraded the server version twice, from 2.8.3 to 2.11.0, then from 2.11.0 to 2.12.0, and the problem remained.

Now we're running

Passbolt Version: 2.12.0/2.12.0
Platform and Target: -- Browser: Firefox 72.0.1 (64-bit) -- Browser: Chrome 79.0.3945.117 (Official Build)(64-bit) -- Client OS: Windows 10 v1903 (OS Build 18362.535) -- Client OS: macOS 10.15.2

What you did

Create a new password in the web UI, by clicking the blue "create" button.

What happened

The interface hangs, with a pop-up showing:

Encrypting ... Take a deep breath and enjoy being in the present moment ...

Not responding to any mouse action, click the "x" on right-upper corner can not close it, can only refresh the window with F5 key.

It works when using the browser extension pop-up (by clicking the blue "create new" button).

What you expected to happen

The new password entry should be created in a moment (at most few seconds).

TB-effective commented 4 years ago

Can someone please look at this? This is a 100% reproduceable problem for us. Every user that we migrate to a new laptop and recover their passbolt login can subsequently no longer create new credentials. Somehow I cannot imagine that we're the only ones running into this, but if we are, maybe someone has an idea how we can proceed to debug it further to find any potential local setup issue that may be causing it.

jiansong commented 4 years ago

This issue has been changed a bit since the release of 2.13, now users can create new password entry, but sharing and editing remain broken.

Updated info:

What's working:

Create new password.

What's not working:

Share password
Edit password

Browser versions:

Chrome 83.0.4103.97 (Official Build) (64-bit) / passbolt extension 2.13.2
Firefox 77.0 / passbolt addon 2.13.2

stripthis commented 4 years ago

@jiansong which API version are you using? We are not able to reproduce this issue here.

jiansong commented 4 years ago

At the moment we're on 2.12.0 for production, 2.12.1 on testing (the testing is a copy of production, then updated to 2.12.1), both are having this problem.

Vanilla and fresh installation does not have this issue.

stripthis commented 4 years ago

It seems to me this is an issue specific to your instance. What is the output of:

./bin/cake passbolt cleanup --dry-run

stripthis commented 4 years ago

Another issue could be with the user key, like maybe the user cannot decrypt. Do they have an issue copying passwords to clipboard?

jiansong commented 4 years ago

What is the output of:
./bin/cake passbolt cleanup --dry-run

     ____                  __          ____
    / __ \____  _____ ____/ /_  ____  / / /_
   / /_/ / __ `/ ___/ ___/ __ \/ __ \/ / __/
  / ____/ /_/ (__  |__  ) /_/ / /_/ / / /
 /_/    \__,_/____/____/_.___/\____/_/\__/

 Open source password manager for teams
---------------------------------------------------------------
 Cleanup shell (dry-run)
---------------------------------------------------------------
35 orphan records found in table GroupsUsers (hard deleted groups)
35 issues detected, please run the same command without --dry-run to fix them.

jiansong commented 4 years ago

Another issue could be with the user key, like maybe the user cannot decrypt. Do they have an issue copying passwords to clipboard?

No problem on copying password to clipboard.

stripthis commented 4 years ago

Yeah so it seems you database integrity is corrupted, most likely because some data was deleted manually in the DB. You can try, running the command again without --dry-run, and see if it fixes it.

jiansong commented 4 years ago

Yeah so it seems you database integrity is corrupted, most likely because some data was deleted manually in the DB.

We didn't manually manipulate the DB. Is there a script can verify (and fix) the DB intergrity?

You can try, running the command again without --dry-run, and see if it fixes it.

I tried it on the test installation, it didn't fix it.

TB-effective commented 4 years ago

[Speaking about the same install as Jiansong.]

It seems to me this is an issue specific to your instance.

Hi, yes, it's not happening on a fresh install we did for comparison. The problem is there is no obvious way to migrate our data to the new instance (without also migrating the problem)...

Yeah so it seems you database integrity is corrupted, most likely because some data was deleted manually in the DB. You can try, running the command again without --dry-run, and see if it fixes it.

We'll give the DB cleanup a shot and see what happens. I don't think we ever made any manual changes in the DB – could it be an artefact of our upgrade history? We did go through quite a few versions on that installation, maybe something happened along the way?

(BTW, could such low level diagnostics/maintenance commands perhaps be documented somewhere? Or is it already and we missed it? If we'd known about it, we certainly would have tried it before.)

jiansong commented 4 years ago

When editing a password, after click Save, it shows following message and hanging.

Updating password

Take a deep breath and enjoy being in the present moment...
Retrieving users                              0%

jiansong commented 4 years ago

BTW, ./bin/cake passbolt healthcheck returns [PASS] on all items.

TB-effective commented 4 years ago

Another issue could be with the user key, like maybe the user cannot decrypt. Do they have an issue copying passwords to clipboard?

No, it seems "read" access is fine, only sharing and new item creation have been broken for these users. And apparently creation is now working again, but not editing or sharing.

Although I remember there were also some differences in behavior (for some of them) between the extension UI and the main browser window UI.

Are there any further diagnostics we could do? Any debug logging we could enable, or similar?

jiansong commented 4 years ago

Although I remember there were also some differences in behavior (for some of them) between the extension UI and the main browser window UI.

Yes, that's for creating password on pre-2.13.x version:

Creating password in the main web UI does not work.
Creating password via the extension/addon 's mini-popup window works (by clicking the extension icon, then click "create new" button).

TB-effective commented 4 years ago

Also maybe it's worth restating that the single most predictive factor for this breakage was account recovery/creation: On this installation, every user we migrated to a new laptop using the recovery mechanism after a certain time experienced this issue on the new machine. Also any new user added has not been able to create or share credentials. (And I assume can now create again, but we don't have confirmation from all affected users yet.)

stripthis commented 4 years ago

@TB-effective @jiansong

Are you working in the same organization? Or is it the same problem for both of you on two different instances?
Can you check if there are any error the browser extension console (it’s different from the regular console)? -- Firefox: Shift + Ctrl + J (or Shift + Command + J on mac) -- Chrome: go to chrome://extensions/, click on index.html for passbolt extension

TB-effective commented 4 years ago

same org, same instance.

TB-effective commented 4 years ago

One thing just occurred to me regarding the orphaned records. In the course of cleanup on #84, we did remove a large number of records, but only from action_logs, because it had filled up to gigabytes of session checks (AuthCheckSession.checkSessionGet), which we didn't want to retain.

We felt safe removing these as they're just logs, but is there any possibility that may be related in some unexpected way?

cedricalfonsi commented 4 years ago

One thing just occurred to me regarding the orphaned records. In the course of cleanup on #84, we did remove a large number of records, but only from action_logs, because it had filled up to gigabytes of session checks (AuthCheckSession.checkSessionGet), which we didn't want to retain.

This should not be a problem.

As @stripthis mentioned in his post, the extension console log would help to understand the problem.

jiansong commented 4 years ago

Are you working in the same organization? Or is it the same problem for both of you on two different instances?

Yes, we're from same org, and working on the same instance, sorry if this confuses you in first place.

Can you check if there are any error the browser extension console (it’s different from the regular console)? -- Firefox: Shift + Ctrl + J (or Shift + Command + J on mac) -- Chrome: go to chrome://extensions/, click on index.html for passbolt extension

(error messages from Firefox) When refreshing the main window (Cmd-R), there's two error messages in Browser Console:

This one appears twice (with different numeric ending)

sendRemoveListener on closed conduit passbolt@passbolt.com.824633721984 (2) ConduitsChild.jsm:108
    _send resource://gre/modules/ConduitsChild.jsm:108
    _send self-hosted:948
    removeListener resource://gre/modules/ExtensionChild.jsm:762
    removeListener resource://gre/modules/ExtensionChild.jsm:985
    onChanged chrome://extensions/content/child/ext-storage.js:337
    removeListener resource://gre/modules/ExtensionCommon.jsm:2518
    revoke resource://gre/modules/ExtensionCommon.jsm:2540
    close resource://gre/modules/ExtensionCommon.jsm:2545
    unload resource://gre/modules/ExtensionCommon.jsm:910
    close resource://gre/modules/ExtensionContent.jsm:926
    destroyed resource://gre/modules/ExtensionContent.jsm:1015
    observe resource://gre/modules/ExtensionContent.jsm:1033

This one only appears once

Content Security Policy: The page’s settings blocked the loading of a resource at inline (“default-src”). panel.js:78:24

When editing a password (and hang after click "Save"), there's no output in Browser Console, but there's error message in Web Console (Cmd-Opt-K):

{…}
react-app.js:103:2728
message: "Misformed armored text"
name: "Error"
<prototype>: {…}
__defineGetter__: function __defineGetter__()
__defineSetter__: function __defineSetter__()
__lookupGetter__: function __lookupGetter__()
__lookupSetter__: function __lookupSetter__()
__proto__: 
constructor: function Object()
hasOwnProperty: function hasOwnProperty()
isPrototypeOf: function isPrototypeOf()
propertyIsEnumerable: function propertyIsEnumerable()
toLocaleString: function toLocaleString()
toString: function toString()
valueOf: function valueOf()
<get __proto__()>: function __proto__()
<set __proto__()>: function __proto__()
    save moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/react-app.js:103
    handleFormSubmit moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/react-app.js:103
    l moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    p moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    v moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    v moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    at moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    it moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    lt moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    pt moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    I moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    U moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    Zt moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    Gt moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    unstable_runWithPriority moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:108
    Vi moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    z moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    Kt moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    Yt moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    Qt moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    Xa moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    cs moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    us moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    ls moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    Jl moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    qi moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    unstable_runWithPriority moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:108
    Vi moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    qi moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    Hi moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    Yl moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    enqueueSetState moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:88
    setState moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/vendors/vendors~react-app.js:74
    handleResourceEditDialogOpenEvent moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/react-app.js:403
    _onMessage moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/react-app.js:6
    t moz-extension://3451f964-ce5b-5745-b067-008646104ae7/data/js/dist/react-app.js:6

each line of the <prototype>: {…} is expandable (some in many levels), I didn't expand them all.

cedricalfonsi commented 4 years ago

Hello @jiansong @TB-effective,

Can you send us a message to support@passbolt.com, I'd like to organize a call to help you with your issue.

Best regards

TB-effective commented 4 years ago

Hi @cedricalfonsi – thanks a lot for the offer, yes we'll write to you to set something up.

TB-effective commented 4 years ago

Quick update here: With @cedricalfonsi's help, we have been able to trace the source of the issue to one GPG key that for some reason has an empty armored_key -- this should obviously never happen, as according to Cédric the field is controlled by the API. We still haven't cleaned this up (looking into whether we can simply remove the user or need to reinstate the proper key), so can't give final confirmation that this was it yet, but it's highly likely. I think it's safe to close this issue.

(Whether the devs would like to open a follow-up API issue because this state should never have been reached is up to them. The root cause must have been some bug in the API, but who knows if it was only present in whatever version we were running back in September when this started. It seems we were the only ones ever hit by this.)

stripthis commented 4 years ago

Thanks for the follow up.

Would be nice to have the original public key to see if this is indeed the API. Normally this cannot happen, as all the data is validated prior to insertion and this is not a valid string. I suspect direct DB entry edit/insert (for example when reloading data from backup maybe?) considering this is the only report ever of such issue, but hard to know.

We'll be adding additional tasks to re-validate the content of the entire DB in the next release. We'll also change the keyring synchronization in the extension to skip invalid keys.

TB-effective commented 4 years ago

Thanks for the follow up.

Would be nice to have the original public key to see if this is indeed the API. Normally this cannot happen, as all the data is validated prior to insertion and this is not a valid string. I suspect direct DB entry edit/insert (for example when reloading data from backup maybe?) considering this is the only report ever of such issue, but hard to know.

I'm pretty sure there was no manual DB manipulation involved.

I'll see if we have backups from back then. Which table/column is this?

We'll be adding additional tasks to re-validate the content of the entire DB in the next release. We'll also change the keyring synchronization in the extension to skip invalid keys.

++

stripthis commented 4 years ago

Which table/column is this?

gpgkeys table and armored_key column.

passbolt / passbolt_browser_extension