Surprising "UUID does not match " error just started yesterday

[rdhyee]

Yesterday, I started getting the error message:

UUID does not match. This could indicate tampering or that something is wrong with the server. Exiting

error message after I enter username/password/2fa code (correctly). Up to that point, I had been successfully using standardnotes-fs without problems for a couple of months. Things look fine on my official SN clients, however. I don't know how to get rid of the problem. I downloaded and restored my data but still have the same problem.

Could this issue be related to API v20190520 migration · Issue #19 · tannercollin/standardnotes-fs? Maybe the API changes used by the SN clients have changed the UUIDs of the notes in such a way that standardnotes-fs no longer recognizes as valid?

[rdhyee]

To clarify, I run into the problem when I run snfs --foreground --ext .md /Volumes/ryvault1/standardnotes -- which prompts me for my username/password/2fa token. I get the error message after typing in my credentials and a 2fa token correctly. Also, I'm running on macOS 10.14.5 and Python 3.7.1

[rdhyee]

As I've posted in Slack in conversation with @mobitar:

I think I have found the item that is causing snfs headaches. Item with uuid d86a57fd-6d82-45a8-90a1-c232f2d22651 --> when I have the debugger at https://github.com/tannercollin/standardnotes-fs/blob/98974de48ed513c42339c91f762632aa111c9ad6/standardnotes_fs/api.py#L88 and analyzing the response object (which has the keys: ['retrieved_items', 'saved_items', 'unsaved', 'sync_token', 'cursor_token']), I see that the only item about the retrieved_itemswith a content key whose 3rd subfield does not match the uuid is the problem item. To express it in Python:

[item for item in response['retrieved_items'] if len(item['content'].split(':')) > 2 and not item['content'].split(':')[2]] turns up (I put .... in :disappointed:

[{'uuid': 'd86a57fd-6d82-45a8-90a1-c232f2d22651', 'content': '003:a9f9......d7af7919e2777a5175c41::b23cea8b....'}] Note the empty string between :: -- this is where the UUID should be.

Is this a situation that comes up in the new API?

(I'm not surprised that the note involved is one that might have had conflicts -- I was working on it yesterday and have several versions of it in the end....)

[tannercollin]

Hmm the UUID is required, it's part of the spec: https://standardfile.org/#item-encryption (see item 3 in "Decrypting a string").

Perhaps one of the clients accidentally dropped it?

I haven't touched snfs in a while, and am completely unfamiliar with v20190520 API unfortunately.

@mobitar is it safe to skip that UUID check if the UUID is empty? I imagine not, as a malicious actor could remove it.

[rdhyee]

From what I can see, it's the server sending an empty UUID -- so I don't know what that means. I haven't packet-sniffed the SN clients to see a UUID is missing with the v20190520 API.

[moughxyz]

Yeah, strange, the UUID is missing from the final string. Haven't seen this before. Will need to find a way to get it replicated. But in any case, I think snfs should just skip that item, and not exit completely, if it has an issue decrypting.

[rdhyee]

@mobitar Can you see the specific note in question? (Item with uuid d86a57fd-6d82-45a8-90a1-c232f2d22651)? I figured that you would be able to debug this issue with this item (without being able to decrypt the content itself) -- is that right? That's why I haven't tried to work around the problem (by, say, deleting the note?)

[moughxyz]

Well in the database I do see that it's missing that UUID. But there's nothing I can do to debug it, as I'd need to be able to make these corrupted items at will.

[rdhyee]

May it's some interaction between your clients (which use the latest version of the API) and snfs (which uses an older version of the API) that caused the missing UUID. I don't know how to reproducibly create a corrupted item either.

So I'm going to delete the corrupted item in hopes that I can get snfs working for me again. I will look at the snfs code, possibly submit a PR changing its behavior to not stopping on this one error -- and see how hard it would be to implement the new API.

[rdhyee]

I nuked the problem note in question, and snfs works for me again.

[tannercollin]

Were the official clients able to display that note? How do they handle the note if they don't know the UUID?

[rdhyee]

@tannercollin Yes, the official clients were able to display the note -- that's how I was able to delete the note without the UUID and then get snfs working again. Don't know why the official clients accepted the note without a UUID though.

[moughxyz]

I think it would display the note in the list, but it wouldn't display its contents. It would say "Error Decrypting" or "Missing Keys". This way you can still delete it.

[rdhyee]

@mobitar It was only snfs that complained about the lack of the UUID -- the official clients were totally fine with the note with the missing UUID. Were the official clients ok because they were using the latest version of the API and see something different about the note that made the clients happy with the note?

[moughxyz]

Ah, you know what, it looks like our code is checking for the existence of a uuid before comparing it. If no UUID, it doesn't do the check. Will need to get this updated.

[tannercollin]

I manually nuked the uuid of note 4f0c1de0-4ab2-11e9-8651-d43d7edad4b3 ("james") on my test account with credentials:

standardnotes-fs@domain.com testaccount

If you'd like to use it for debugging. It now looks like this:

{'uuid': '', 'content': '003:e2e127ddbe5025d565ac1781c389e4346c69728bbe3d28cc2442cc12c888ac99:4f0c1de0-4ab2-11e9-8651-d43d7edad4b3:8e3e0f187035d2dfd5da325275082396:w0zCnxQJqWiyGECouxDIRfNh6zR9k0udIhFKSxM8tXs1R5MuYxWng/PAMIrLsx+EzxNyqQciGgOgxucfMgK2xt2J8Z1210v1Tfgh+pPp/3fC86dxCRtJug18o1gjyOADMX29hyG5wzvBBpEuo25yb5d83aASXXa6Tpj6NgaqfKoZqw2CJINnI6pDHql+fIEi2jOITR08l2L14qCSo3PSfw==', 'content_type': 'Note', 'enc_item_key': '003:fe501a6889f2fc38d56760c986eb15891979fa9d90f2d66d2771d192764dce2f:4f0c1de0-4ab2-11e9-8651-d43d7edad4b3:b2473b10bc8d126dfdacfbc127a8a0dd:K00tj0uuYaPh0dirhHMlCcAONUE9fKBzpcfE0rTw+uAepcpnVrUK+5vZtHW6MPjpT30T+WkU4UVD7Db6hX6kYmXlz5LJdvKhsxhCj5OExzo/sGSFyNo0wwt8tJSsJ/XGFDnO0R+32nhNxptEKj6r4OjDryXwdmCT9Kz6OahkMsEJFQnJgFJfWVfJuDeFnuUQ', 'auth_hash': None, 'created_at': '2019-03-20T01:48:50.233Z', 'deleted': False}

It looks like it took on a new UUID of 3de335fc-745d-4854-9db6-badd435826a5.

This is such an edge case that I'd rather snfs continue to exit instead of dealing with the logic of having a skipped item. I do think I should add more logging to the most verbose setting though, to help in the future.