Infinite 'joplin.data.get' loop when configuring the Email plugin

adeverteuil commented 1 year ago

Hi,

I'm trying to configure the Email Plugin, but I think there is an infinite loop bug.

Email provider

I host my personal mailcow: dockerized mail server on DigitalOcean.

Steps to reproduce

Open the Email plugin.
Click "Manually connect to IMAP".
Fill in the information.
1. Email address.
2. Password.
3. Domain.
4. Port 993.
5. TLS enabled.
Click LOGIN.

Problem conditions

The problem does NOT happen on an empty Joplin profile.

If I change to a new profile and configure the plugin, it successfully logs in and imports notes from the mailbox.

Then, if I import my notes from the .jex backup into the new profile, it continues working and importing notes from the mailbox.

But after importing my notes from the .jex backup, and try to logout and log back in with the Email plugin, the problem happens again.

So I don't know if the bug is in Joplin or in the Email plugin. It seems to only happen when there are notes and notebooks present at the time of logging in. But I'm not sure how to isolate the exact minimal conditions to reproduce the issue.

Negative testing

I tried with the wrong password, and I got a popup message saying "Error: Authentication failed."

The infinite loop doesn't happen in that case.

Logs

I enabled debug logs by following these instructions. Here is an extract of the output, starting at the first instance of the string "Email".

2023-06-06 13:54:24: PluginRunner: "Got message (3): joplin.views.panels.postMessage", "["plugin-view-Bishoy.EmailPlugin-panel","enableLoginScreen"]"
2023-06-06 13:54:24: UserWebview: "Got message", "postMessageService.plugin_message", "{"message":"enableLoginScreen"}"
2023-06-06 13:54:24: UserWebview: "Got message", "postMessageService.response", "{"message":{"responseId":"userWebview_16860812622680.20734508102333438","error":null}}"
2023-06-06 13:54:24: "ResourceService::indexNoteResources: Start"
2023-06-06 13:54:24: "ResourceService::indexNoteResources: Completed"
2023-06-06 13:54:24: "ResourceService::deleteOrphanResources:", "[]"
2023-06-06 13:54:25: PluginRunner: "Got message (3): joplin.views.panels.setHtml", "<hidden>"
2023-06-06 13:54:25: UserWebview: "Got message", "postMessageService.response", "{"message":{"responseId":"userWebview_16860812652820.8807690608651975","error":null}}"
2023-06-06 13:54:25: UserWebview: "Got message", "setHtml", "{"hash":"f99d236664e4e7f4a8ab425f23498b34","html":"<div class=\"container\">\n\n<div class=\"row\" style=\"font-size: large;\">\n\n  <div class=\"col-md-9 col-lg-7 col-xl-5 mx-auto\">\n\n  <div class=\"card border-0 shadow rounded-3 my-3 align-middle d-flex justify-content-center\" style=\"opacity: 0.97; top: 100px;\" >\n\n      <div class=\"card-body p-4 p-sm-5\">\n\n        <h1 class=\"card-title text-center mb-3 fw-light fs-1\">Login</h1>\n\n        <form action=\"\" onsubmit=\"loginManually(); return false\">\n          \n          <div class=\"form-floating mb-3\">\n            <input type=\"email\" class=\"form-control\" id=\"email\" placeholder=\"name@example.com\" required>\n            <label for=\"floatingInput\">Email address</label>\n          </div>\n          \n          <div class=\"form-floating mb-3\">\n            <input type=\"password\" class=\"form-control\" id=\"password\" placeholder=\"Password\" onclick = \"addBlockquote()\" required>\n            <label for=\"floatingPassword\">Password</label>\n          </div>\n\n          <div class=\"input-group\">\n            <span class=\"input-group-text\">Server</span>\n            <input type=\"text\" aria-label=\"First name\" class=\"form-control\" placeholder=\"imap.example.com\"\n              id=\"server\" required>\n          </div>\n\n        <br>\n\n        <div class=\"input-group\">\n          <span class=\"input-group-text\">PORT</span>\n          <input type=\"number\" aria-label=\"First name\" class=\"form-control\" placeholder=\"993\" min=\"1\" id=\"port\" required>\n        </div>\n\n        <br>\n\n        <div class=\"form-check\">\n          <input class=\"form-check-input\" type=\"checkbox\" value=\"\" id=\"ssl_tls\" checked>\n          <label class=\"form-check-label\" for=\"ssl_tls\">\n            SSL/TLS\n          </label>\n        </div>\n\n        <blockquote class=\"blockquote\">\n          <cite style = \"font-size: 15px;\" id = \"quote\"></cite>\n        </blockquote>\n\n          <div class=\"d-grid\">\n            <button id=\"login_btn\" class=\"btn btn-outline-primary btn-login text-uppercase fw-bold\" type=\"submit\" style=\"font-size:large;\">Login</button>\n          </div>\n\n        </form>\n\n        <br>\n\n        <div class=\"container\" style=\"text-align: center;\">\n          <button type=\"button\" class=\"btn btn-outline-info\" style=\"width: 75%; font-size: large;\"\n            onclick=\"loginScreen()\">Login Screen</button>\n        </div>\n\n        <hr class=\"my-4\">\n\n        <div class=\"container\" style=\"text-align: center;\">\n          <button type=\"button\" class=\"btn btn-outline-danger\" onclick=\"hide()\">Close</button>\n        </div>\n\n      </div>\n    </div>\n  </div>\n</div>\n</div>\n"}"
2023-06-06 13:54:40: UserWebview: "Got message", "postMessageService.response", "{"message":{"responseId":"userWebview_16860812806990.713801072367042","error":null}}"
2023-06-06 13:54:41: PluginRunner: "Got message (3): joplin.data.get", "[["folders"]]"
2023-06-06 13:54:41: PluginRunner: "Got message (3): joplin.data.get", "[["folders","52beed25bcb54227972c77e7494fea90"]]"
2023-06-06 13:54:41: PluginRunner: "Got message (3): joplin.data.get", "[["folders","93fc6e32a77b40cea499eacd8d1583e3"]]"
2023-06-06 13:54:41: PluginRunner: "Got message (3): joplin.data.get", "[["folders","93fc6e32a77b40cea499eacd8d1583e3"]]"
2023-06-06 13:54:41: PluginRunner: "Got message (3): joplin.data.get", "[["folders","52beed25bcb54227972c77e7494fea90"]]"
2023-06-06 13:54:41: PluginRunner: "Got message (3): joplin.data.get", "[["folders","93fc6e32a77b40cea499eacd8d1583e3"]]"
2023-06-06 13:54:41: PluginRunner: "Got message (3): joplin.data.get", "[["folders","52beed25bcb54227972c77e7494fea90"]]"
2023-06-06 13:54:41: PluginRunner: "Got message (3): joplin.data.get", "[["folders","52beed25bcb54227972c77e7494fea90"]]"
2023-06-06 13:54:41: PluginRunner: "Got message (3): joplin.data.get", "[["folders","52beed25bcb54227972c77e7494fea90"]]"

This loop of joplin.data.get messages continues seemingly forever, at a rate of about 500 per second, and with repeating UIDs, until I stop the Joplin application.

In Dovecot logs, I see a successful login, but then nothing else until I close the Joplin application.

imap-login:  Login: user=<[REDACTED]>, method=PLAIN,  rip=[REDACTED], lip=[REDACTED], mpid=342267, TLS, TLSv1.3 with  cipher TLS_AES_256_GCM_SHA384 (256/256 bits)

imap([REDACTED])<343090><Zc37KXz9VKtiPIes>: Disconnected: Connection closed (IDLE running for 0.001 + waiting input for 1.925 secs, 2 B in + 10 B out, state=wait-input) in=90 out=1176 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0

adeverteuil commented 1 year ago

I think I found the bug. In the source code, the plugin attempts to get all the folders like this:

    let joplinFolders: JopinFolders;
    const folders = [];

    do {
        joplinFolders = await joplin.data.get(['folders']);

        [build internal index of folders...]
        }
    } while (joplinFolders.has_more);

However, this is not the correct way to make paginated request; the page query parameter is missing. When there are more than 100 items to return and joplinFolders.has_more is true, only page 1 is requested and the plugin enters an infinite loop.

I think this would work better:

    let joplinFolders: JopinFolders;
    const folders = [];
    let pageNum = 1;

    do {
        joplinFolders = await joplin.data.get(['folders'], {"page": pageNum++});

        [build internal index of folders...]
        }
    } while (joplinFolders.has_more);

Please note that I have not tested this and I'm not a Typescript or a Javascript programmer.

I also found two places in src/ui/screens.ts where this infinite loop may happen: https://github.com/joplin/plugin-email/blob/34fe1a48573316b9e055f0d04e625a0fc8dcc294/src/ui/screens.ts#L207 https://github.com/joplin/plugin-email/blob/34fe1a48573316b9e055f0d04e625a0fc8dcc294/src/ui/screens.ts#L331

References:

bishoy-magdy commented 1 year ago

Hi @adeverteuil, Thanks very much for this great description of the issue.

However, this is not the correct way to make paginated request; the page query parameter is missing. When there are more than 100 items to return and joplinFolders.has_more is true, only page 1 is requested and the plugin enters an infinite loop.

This is correct; the bug occurs when the number of folders is greater than 100. I will fix it, and thank you for mentioning it.

To be sure, this is the issue you encounter. Is the login button going to always spin infinity, and you unable to login?

adeverteuil commented 1 year ago

Hi @bishoy-magdy, thank you for your response!

Is the login button going to always spin infinity, and you unable to login?

Yes, when I click on the Login button, then it changes to a Loading spinner that spins infinitely, and I am unable to login.

Here is a screenshot: Screenshot 2023-06-19 12-50-07 loading

chopstik commented 8 months ago

@bishoy-magdy are you able to produce a new release from the latest code? The current 1.0.0 release is out of date and leading to loading/loop issues.

founfabug commented 5 months ago

@bishoy-magdy Thank you for this plugin, it is the reason I moved from tidliwiki to Joplin. Could you please make a new release of the plugin from the latest code?

joplin / plugin-email