[bug]: oldState doesn’t reflect sinceState in */changes

iNPUTmice commented 10 months ago

What happened?

Hi,

Ltt.rs developer here. I finally found the time to do some interop testing with my jmap library and I fairly quickly ran into the issue of not being able to retrieve changes properly.

In the following example I've started with an empty Inbox and sent one email to it.

Sorry the examples are a bit verbose but they are directly from my library. They are a snapshot from Stalwart 0.3.1 but I briefly checked and the problem persists in 0.4.0

{
  "methodCalls": [
    [
      "Mailbox/changes",
      {
        "accountId": "e",
        "sinceState": "sfi"
      },
      "0"
    ],
    [
      "Mailbox/get",
      {
        "accountId": "e",
        "#ids": {
          "resultOf": "0",
          "name": "Mailbox/changes",
          "path": "/created"
        }
      },
      "1"
    ],
    [
      "Mailbox/get",
      {
        "#properties": {
          "resultOf": "0",
          "name": "Mailbox/changes",
          "path": "/updatedProperties"
        },
        "accountId": "e",
        "#ids": {
          "resultOf": "0",
          "name": "Mailbox/changes",
          "path": "/updated"
        }
      },
      "2"
    ],
    [
      "Email/changes",
      {
        "accountId": "e",
        "sinceState": "sfi"
      },
      "3"
    ],
    [
      "Email/get",
      {
        "fetchTextBodyValues": true,
        "accountId": "e",
        "#ids": {
          "resultOf": "3",
          "name": "Email/changes",
          "path": "/created"
        },
        "properties": [
          "id",
          "id",
          "threadId",
          "mailboxIds",
          "keywords",
          "size",
          "receivedAt",
          "messageId",
          "inReplyTo",
          "references",
          "sender",
          "from",
          "to",
          "cc",
          "bcc",
          "replyTo",
          "subject",
          "bodyValues",
          "textBody",
          "htmlBody",
          "attachments",
          "sentAt",
          "bodyStructure",
          "header:User-Agent:asText",
          "header:Autocrypt:asText:all",
          "header:Autocrypt-Draft-State:asText",
          "header:Autocrypt-Setup-Message:asText"
        ]
      },
      "4"
    ],
    [
      "Email/get",
      {
        "accountId": "e",
        "#ids": {
          "resultOf": "3",
          "name": "Email/changes",
          "path": "/updated"
        },
        "properties": [
          "keywords",
          "mailboxIds"
        ]
      },
      "5"
    ],
    [
      "Thread/changes",
      {
        "accountId": "e",
        "sinceState": "sfi"
      },
      "6"
    ],
    [
      "Thread/get",
      {
        "accountId": "e",
        "#ids": {
          "resultOf": "6",
          "name": "Thread/changes",
          "path": "/created"
        }
      },
      "7"
    ],
    [
      "Thread/get",
      {
        "accountId": "e",
        "#ids": {
          "resultOf": "6",
          "name": "Thread/changes",
          "path": "/updated"
        }
      },
      "8"
    ],
    [
      "Email/query",
      {
        "collapseThreads": true,
        "accountId": "e",
        "filter": {
          "inMailbox": "a"
        },
        "limit": 20,
        "sort": [
          {
            "isAscending": false,
            "property": "receivedAt"
          }
        ]
      },
      "9"
    ],
    [
      "Email/get",
      {
        "accountId": "e",
        "#ids": {
          "resultOf": "9",
          "name": "Email/query",
          "path": "/ids"
        },
        "properties": [
          "threadId"
        ]
      },
      "10"
    ]
  ],
  "using": [
    "urn:ietf:params:jmap:core",
    "urn:ietf:params:jmap:mail"
  ]
}

And I got the following response:

{
  "methodResponses": [
    [
      "Mailbox/changes",
      {
        "accountId": "e",
        "oldState": "n",
        "newState": "sfm",
        "hasMoreChanges": false,
        "created": [],
        "updated": [
          "a"
        ],
        "destroyed": [],
        "updatedProperties": [
          "totalEmails",
          "unreadEmails",
          "totalThreads",
          "unreadThreads"
        ]
      },
      "0"
    ],
    [
      "Mailbox/get",
      {
        "accountId": "e",
        "state": "sfm",
        "list": [],
        "notFound": []
      },
      "1"
    ],
    [
      "Mailbox/get",
      {
        "accountId": "e",
        "state": "sfm",
        "list": [
          {
            "totalEmails": 1,
            "unreadEmails": 1,
            "totalThreads": 1,
            "unreadThreads": 1,
            "id": "a"
          }
        ],
        "notFound": []
      },
      "2"
    ],
    [
      "Email/changes",
      {
        "accountId": "e",
        "oldState": "n",
        "newState": "sfm",
        "hasMoreChanges": false,
        "created": [
          "a"
        ],
        "updated": [],
        "destroyed": []
      },
      "3"
    ],
    [
      "Email/get",
      {
        "accountId": "e",
        "state": "sfm",
        "list": [
          {
            "id": "a",
            "threadId": "a",
            "mailboxIds": {
              "a": true
            },
            "keywords": {},
            "size": 4913,
            "receivedAt": "2023-10-23T15:56:08Z",
            "messageId": [
              "CAN-aAr8e=n3Z6o72zAjEFq3bbZ2KkzDsgtLNy1NvOyB8XeLrmA@mail.gmail.com"
            ],
            "inReplyTo": null,
            "references": null,
            "sender": null,
            "from": [
              {
                "name": "Daniel Gultsch",
                "email": "daniel@gultsch.de"
              }
            ],
            "to": [
              {
                "name": null,
                "email": "lttrs@mateus-mello-labs.xyz"
              }
            ],
            "cc": null,
            "bcc": null,
            "replyTo": null,
            "subject": "This is a test",
            "bodyValues": {
              "0": {
                "isEncodingProblem": false,
                "isTruncated": false,
                "value": "Hi,\n\nallow me to send a quick test email.\n\ncheers\nDaniel\n"
              }
            },
            "textBody": [
              {
                "partId": "0",
                "blobId": "eecab2rfh2",
                "size": 63,
                "name": null,
                "type": "text/plain",
                "charset": "UTF-8",
                "disposition": null,
                "cid": null,
                "language": null,
                "location": null
              }
            ],
            "htmlBody": [
              {
                "partId": "0",
                "blobId": "eecab2rfh2",
                "size": 63,
                "name": null,
                "type": "text/plain",
                "charset": "UTF-8",
                "disposition": null,
                "cid": null,
                "language": null,
                "location": null
              }
            ],
            "attachments": [],
            "sentAt": "2023-10-23T15:55:55Z",
            "bodyStructure": {
              "partId": "0",
              "blobId": "eecab2rfh2",
              "size": 63,
              "name": null,
              "type": "text/plain",
              "charset": "UTF-8",
              "disposition": null,
              "cid": null,
              "language": null,
              "location": null
            },
            "header:User-Agent:asText": null,
            "header:Autocrypt:asText:all": [],
            "header:Autocrypt-Draft-State:asText": null,
            "header:Autocrypt-Setup-Message:asText": null
          }
        ],
        "notFound": []
      },
      "4"
    ],
    [
      "Email/get",
      {
        "accountId": "e",
        "state": "sfm",
        "list": [],
        "notFound": []
      },
      "5"
    ],
    [
      "Thread/changes",
      {
        "accountId": "e",
        "oldState": "n",
        "newState": "sfm",
        "hasMoreChanges": false,
        "created": [
          "a"
        ],
        "updated": [],
        "destroyed": []
      },
      "6"
    ],
    [
      "Thread/get",
      {
        "accountId": "e",
        "state": "sfm",
        "list": [
          {
            "id": "a",
            "emailIds": [
              "a"
            ]
          }
        ],
        "notFound": []
      },
      "7"
    ],
    [
      "Thread/get",
      {
        "accountId": "e",
        "state": "sfm",
        "list": [],
        "notFound": []
      },
      "8"
    ],
    [
      "Email/query",
      {
        "accountId": "e",
        "queryState": "sfm",
        "canCalculateChanges": true,
        "position": 0,
        "ids": [
          "a"
        ]
      },
      "9"
    ],
    [
      "Email/get",
      {
        "accountId": "e",
        "state": "sfm",
        "list": [
          {
            "threadId": "a",
            "id": "a"
          }
        ],
        "notFound": []
      },
      "10"
    ]
  ],
  "sessionState": "3e25b2a0"
}

The problem here is that oldState in the response doesn’t match sinceState in the request.

How can we reproduce the problem?

I can reproduce the problem by doing the following steps:

Start with empty mailbox
Send email to the account
Send mailbox/changes query

Version

v0.4.x

What database are you using?

None

What blob storage are you using?

None

Where is your directory located?

None

What operating system are you using?

None

Relevant log output

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

mdecimus commented 10 months ago

Hi,

Nice to meet you! I'm glad to hear that you are doing interop testing with Stalwart. Ltt.rs is one of the few JMAP clients out there that implements the full specification.

Thanks for the report, this issue has now been fixed and a patch will be included in the next release. The sinceState passed by the client was not being echoed back in oldState. However, the changes returned by the server should be correct.

mdecimus commented 10 months ago

Version 0.4.2 has just been released including this fix as well as JMAP for Quotas and JMAP Blob Management support.

stalwartlabs / jmap-server