search

tannercollin / standardnotes-fs

Mount your Standard Notes as a filesystem.
GNU General Public License v3.0
148 stars 12 forks source link

Unknown Traceback on Mount #31

Closed nickAtWybar closed 4 years ago

nickAtWybar commented 4 years ago

Hio, long term snfs user here. Has been quite well for me for a long time. Ran into a new error today though.

In the middle of the day, not doing anything particular out of the ordinary (or so I remember), the auto-syncing crashed out. This happens every now and then for me, thought no big deal. Just remount the fs.

(base) nick@apollo:~/Programming/coding/linux_scripts$ ./mount_snfs.sh INFO : Using config directory "/home/nick/.config/standardnotes-fs". INFO : Loaded config file "/home/nick/.config/standardnotes-fs/standardnotes-fs.conf". INFO : Using creds directory "/home/nick/.cache/standardnotes-fs". INFO : Loaded creds file "/home/nick/.cache/standardnotes-fs/standardnotes-fs.conf". INFO : Using sync URL "https://sync.standardnotes.org". INFO : Successfully logged into account "[nick's email]@[email].com". INFO : Config written to file "/home/nick/.config/standardnotes-fs/standardnotes-fs.conf". INFO : Creds written to file "/home/nick/.cache/standardnotes-fs/standardnotes-fs.conf". INFO : Starting FUSE filesystem. Traceback (most recent call last): File "/usr/local/bin/snfs", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/dist-packages/standardnotes_fs/standardnotes_fs.py", line 260, in main fuse = FUSE(StandardNotesFUSE(sn_api, sync_sec, args.ext), File "/usr/local/lib/python3.8/dist-packages/standardnotes_fs/sn_fuse.py", line 26, in init self.item_manager = ItemManager(sn_api, ext) File "/usr/local/lib/python3.8/dist-packages/standardnotes_fs/itemmanager.py", line 253, in init self.sync_items() File "/usr/local/lib/python3.8/dist-packages/standardnotes_fs/itemmanager.py", line 84, in sync_items response = self.sn_api.sync(dirty_items) File "/usr/local/lib/python3.8/dist-packages/standardnotes_fs/api.py", line 110, in sync response = self.api.post('/items/sync', dict(sync_token=self.sync_token, File "/usr/local/lib/python3.8/dist-packages/standardnotes_fs/api.py", line 27, in post return res.json() File "/usr/lib/python3/dist-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib/python3/dist-packages/simplejson/init.py", line 518, in loads return _default_decoder.decode(s) File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode obj, end = self.raw_decode(s) File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I've looked through standard notes I've modified in the last day, and nothing weird seems to be contained within the entries. I've also emptied the trash in Standard Notes (sometimes causes issues).

I tried this to sort of unsinstall reinstall, but same error after (I did not blow away the config) - sudo pip3 install git+https://github.com/tannercollin/standardnotes-fs --upgrade

Something I notice in the output is that it is calling python 3.8; that's strange to me, since I regularly use Python 3.69. python --version in the terminal reveals Python 3.6.9 :: Anaconda, Inc.

Looking for some help - how can I troubleshoot this a little further? Thanks!

tannercollin commented 4 years ago

@mobitar did the API change? I'm getting a 500 error from /items/sync.

moughxyz commented 4 years ago

We did have a large version release today, but it should be backward compatible with any existing clients. Do you have the full 500 response?

tannercollin commented 4 years ago

It's just a generic 500 page:

DEBUG   : https://sync.standardnotes.org:443 "POST /items/sync HTTP/1.1" 500 None
<Response [500]>
<!DOCTYPE html>
<html>
<head>
  <title>We're sorry, but something went wrong (500)</title>
  <meta name="viewport" content="width=device-width,initial-scale=1">
  <style>
  body {
    background-color: #EFEFEF;
    color: #2E2F30;
    text-align: center;
    font-family: arial, sans-serif;
    margin: 0;
  }

  div.dialog {
    width: 95%;
    max-width: 33em;
    margin: 4em auto 0;
  }

  div.dialog > div {
    border: 1px solid #CCC;
    border-right-color: #999;
    border-left-color: #999;
    border-bottom-color: #BBB;
    border-top: #B00100 solid 4px;
    border-top-left-radius: 9px;
    border-top-right-radius: 9px;
    background-color: white;
    padding: 7px 12% 0;
    box-shadow: 0 3px 8px rgba(50, 50, 50, 0.17);
  }

  h1 {
    font-size: 100%;
    color: #730E15;
    line-height: 1.5em;
  }

  div.dialog > p {
    margin: 0 0 1em;
    padding: 1em;
    background-color: #F7F7F7;
    border: 1px solid #CCC;
    border-right-color: #999;
    border-left-color: #999;
    border-bottom-color: #999;
    border-bottom-left-radius: 4px;
    border-bottom-right-radius: 4px;
    border-top-color: #DADADA;
    color: #666;
    box-shadow: 0 3px 8px rgba(50, 50, 50, 0.17);
  }
  </style>
</head>

<body>
  <!-- This file lives in public/500.html -->
  <div class="dialog">
    <div>
      <h1>We're sorry, but something went wrong.</h1>
    </div>
    <p>If you are the application owner check the logs for more information.</p>
  </div>
</body>
</html>
tannercollin commented 4 years ago

I think I might be using an even older version. My sync request data only looks like this:

data {'sync_token': None, 'items': []}
headers {'Authorization': 'Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX3V1aWQiOiJkZDkzMzgxYy0zMGIxLTQ4NjItYWQ4MS02YmFmZDRiZjIzNWEiLCJwd19oYXNoIjoiNjYzNGZlYTFkMDg1ZDY1OWMxNDgxYzJmYTcyMjA1OTA1NDFmY2MwNDRlNzI4MTNhYjU4YWFlMmFjZjNjNDJjOSJ9.gjdfVc_9UybMJmRCEvXPvyZwlAO0Z2rGgW1ZVaWlVoQ'}

(test account)

I notice your web client is sending "api", "compute_integrity", "limit", and "item" keys. Are any of those required?

moughxyz commented 4 years ago

It will default to 20161215 if you don't supply an API version. But that is indeed an old version, you should be passing 20190520. I do see in the logs the cause of the 500 error from your client, but still investigating it, because it shouldn't be happening.

tannercollin commented 4 years ago

Yes, if I pass api: "20190520" it works, but I get the "Note UUID does not match." error that we were talking about in #20.

moughxyz commented 4 years ago

What's the UUID of the item?

tannercollin commented 4 years ago

UUID: 3de335fc-745d-4854-9db6-badd435826a5, Local UUID: 4f0c1de0-4ab2-11e9-8651-d43d7edad4b3

I think this was from debugging #20 before, and unrelated to this issue though. Deleting that note fixed everything.

moughxyz commented 4 years ago

Ah ok cool. Btw, protocol version 004 will be released soon, which features new encryption algorithms. You'll need to use Libsodium most likely. https://github.com/standardnotes/snjs/blob/004/specification.md

Is there anyway to use the SNJS JavaScript library in your Python environment? I feel like you're gonna be in for a hell of a time implementing all this stuff from scratch in Python..

tannercollin commented 4 years ago

That looks intense. I'll DM you on Slack about it.

@nickAtWybar issue should be fixed in f3e544dee21563eceabe43f7318f7dca33678b9c. Can you backup, update, and test?

nickAtWybar commented 4 years ago

I updated through: sudo pip3 install --upgrade git+https://github.com/tannercollin/standardnotes-fs

Mounting issue appears to be gone now - working for now!

This bug thread is a thing of beauty to me. Was floored that I posted, had dinner, then basically saw it was resolved. Having active devs who care about their projects (aka the community) is intensely satisfying as someone who relies pretty heavily on both SN and SNFS as a swiss army tool for my life. Thanks for contributing guys - lots of appreciation! @mobitar @tannercollin

nickAtWybar commented 4 years ago

@tannercollin I've noticed something strange now that I've had time to investigate this a little more. With the fix above, it seems like now I can:

Thoughts?

tannercollin commented 4 years ago

@mobitar I'm getting a sync conflict when I try to send a change to the server:

Contents:

{
    "title": "test after new api",
    "text": "hello world works\n\n\nsup\n\nx\n",
    "references": [],
    "appData": {
        "org.standardnotes.sn": {
            "client_updated_at": "2020-10-08T19:54:00.602323Z"
        }
    },
    "preview_plain": "hello world works\n\n\nsup",
    "preview_html": null
}

Request data:

{
    "sync_token": "MjoxNjAyMTg3NTM5Ljk3OTA4NzY=\n",
    "items": [
        {
            "uuid": "3413f65a-08fe-11eb-b705-d43d7edad4b3",
            "duplicate_of": null,
            "items_key_id": null,
            "content": "003:3aa3253d1da06fde2b9a5f6aa0d0081320043004df2d0ef97649d36a638e27f4:3413f65a-08fe-11eb-b705-d43d7edad4b3:ae501f612b482a595dc786c8cd38b3f5:VFmicUhRx70qmdz1AdzBFqTSwKF0SD1qUzASyNFnOPiO5UeYOP59qQAoqNJYVnqFXgExtD22Sf2n1kl8SXcsoeP+c7QFIjN7/ttM/7DCz/cOVFSohTYKbMO2v4BYK/SihuXyJXOq+QELGCBkpPfXuawKxYivbiT2z3p2TIEJyw6rh484wzwCfQMJZjmEHjjkrJ5sr2hw+8Pk0AT6O8NOwt/7eta5YqmUeUlLUhm+ZRe9C7l6QW+aeYjgszfN8Fcc/DcFdU+k0/1niGWBrDs0563IXCeiJGcO3pC7QEXRUQAWH9s5vSCxSwATMQb6zg/nVCeCcu8H95AkRQwLokvrkw==",
            "content_type": "Note",
            "enc_item_key": "003:ccbced572bd3d64e73d8072f0370d8f807db8e73b1598e13c492ad0b95db4107:3413f65a-08fe-11eb-b705-d43d7edad4b3:56230e8e8de679539df879d7962da93b:Kzj5bZmCgV3Z6l5slukp+OpsCw59ZVooZSTRaRs10EG/N+5wtD1fOORXUHAvwViTTi7kV/qcGfYJJSGPn2tLes7jasCvNov5srZUs0A/vErFa8RFAILtq9nHXC1F0p75Iyx4urTVQXLwUAohO3V2LMh8WDt13oKOzYg/LtKKMU9e/7+0r3nXJ2WK94My1D8A",
            "auth_hash": null,
            "created_at": "2020-10-08T00:35:45.337Z",
            "deleted": false
        }
    ],
    "api": "20190520"
}

Response 200:

{
    "retrieved_items": [],
    "saved_items": [],
    "conflicts": [
        {
            "server_item": {
                "uuid": "3413f65a-08fe-11eb-b705-d43d7edad4b3",
                "duplicate_of": null,
                "items_key_id": null,
                "content": "003:0f8b21a51c41388a70d18177ff01343fcc8c5c8222192757f4791088ac9cf77a:3413f65a-08fe-11eb-b705-d43d7edad4b3:88d3deafcd250211b1118fbee6288efc:BihFH+EDzQPO1qBRJWMxNsSYxxHA/dpjLdtoy43rUY8DI3y8gBdjQ5TEtf4icYzlXPq/7wGgzmLNX8tvcpXwuH+Lk34DBUxIB9zUWuXwqp2Svg7lDDSVKlAETDNy/ZRKtFbE85p/KdoXipmHrK8ylgBc3IBQX3OXJHH/hQ/9y+qGHcRBkqdx7gdkhV6eFgec3VMx910whnbZU2WMTX22gHHxGW+AcRr4Gzkal0lhgGqlJ+OragDKQG6hfabwKxnWTWtPzB5QKe25i9GLlR8K200Ewh4KS/MehADEs9wtyhw4tUfKG1JjTyJF9IBT7lvI:eyJwd19ub25jZSI6IjNlZGI1YzY3Y2VjODRlOTI2ZjU3MWUzZjNiYmQ2MzkzZWZmMGY5MDczM2E5YTFjYjg3OThhNjBhOTQ3NjNiMDUiLCJwd19jb3N0IjoxMTAwMDAsImlkZW50aWZpZXIiOiJzdGFuZGFyZG5vdGVzLWZzQGRvbWFpbi5jb20iLCJ2ZXJzaW9uIjoiMDAzIn0=",
                "content_type": "Note",
                "enc_item_key": "003:2b77fe1cf2fe3c4783baaf7271eb4e3f10f0ad871954d0bab2f89bc60d448c67:3413f65a-08fe-11eb-b705-d43d7edad4b3:ddb81911c0a168aa90a4aa81c29e846d:vzJHkhUEVyOKMJ4lSidNujEHBSaz74uNSK/EboBEQ/1bBferwTbrG555v5FVStS2TW58eQ558MTtIG8pM7ODZMFZ1//InRwVF4RXsC7pB0fUDT+SOx/iZOeQWP92ZGH1vbhrd5MgHSLwKfYggk93xPDYCLPG6UH0a/p9FPWR7xvP5viNSPUhVIsRTCtp3Wip:eyJwd19ub25jZSI6IjNlZGI1YzY3Y2VjODRlOTI2ZjU3MWUzZjNiYmQ2MzkzZWZmMGY5MDczM2E5YTFjYjg3OThhNjBhOTQ3NjNiMDUiLCJwd19jb3N0IjoxMTAwMDAsImlkZW50aWZpZXIiOiJzdGFuZGFyZG5vdGVzLWZzQGRvbWFpbi5jb20iLCJ2ZXJzaW9uIjoiMDAzIn0=",
                "auth_hash": null,
                "created_at": "2020-10-08T00:35:45.337Z",
                "updated_at": "2020-10-08T00:35:54.817Z",
                "deleted": false
            },
            "type": "sync_conflict"
        }
    ],
    "sync_token": "MjoxNjAyMTg3NTcwLjQxNTgyNzM=\n",
    "cursor_token": null
}

Any idea why? Where can I read about the 20190520 api spec? I can't find it in the docs.

moughxyz commented 4 years ago

You have to make sure that for any item you get back, you update your client's item's updated_at field with the value reported from the server.

Seems like your request payload is missing updated_at altogether.

tannercollin commented 4 years ago

Okay I'm setting updated_at to when it changes, but I'm still getting a conflict.

Request:

{
    "sync_token": "MjoxNjAyMTg4OTcyLjE2ODY0OTQ=\n",
    "items": [
        {
            "uuid": "3413f65a-08fe-11eb-b705-d43d7edad4b3",
            "duplicate_of": null,
            "items_key_id": null,
            "content": "003:500608059accc5c356ca2396e130ef7e52778cb117d7d9e87f85a0036e074921:3413f65a-08fe-11eb-b705-d43d7edad4b3:d76f01f47e38f5f88881b83568854dcb:Jue7AbqYLUpQS3xZTGtigrhGPq0KZ7MgAHlI3Zw4MIdOXIH+nVyIZstoeVHMN+GcgBQaW/3N4J/zP4H9tdIDt3omqtQWQdAiVQcS/q8mLwQwnUCFYPZaA9I6A1cKIFrEVT9yFiqmL3y2TucEAjj06VWFAa090EzS17wYkc6wSxSRfsGHTRHYW4AxRQ2dECRC1v04ERpDCjW5RL23C6n55vG3lSzwqIVA7EOWAc50/UHJ16+nBIS6koWhCfIuwIjL4E8lzfNq6+nX1AxfFw1s84hEqwSC1q9lN9rNpmjWj4771SpJ5QhP5ZWbVNOzPFifJHfb2MGkixBp0JINRivc5A==",
            "content_type": "Note",
            "enc_item_key": "003:505fe376e6ab0eb77a11a2f8cefc9dfe7d3078a585f77ef76f6490d29c2cb328:3413f65a-08fe-11eb-b705-d43d7edad4b3:9309769e2e360e0ce3a55f727bcad817:m1e8r3eB6BH21Cwblhla4GjPsOYM1ge8pyYSeOeMQNdy50zi3lGTmGS7w9FtOk3SmMY3m4RLyOVFvHdyljdeU5+YVbigA4YbiqiEr7dQy0M2uan3hHg303PcF2sZbhI0NWrZJf68ITpbu10dQA7McNkwQCI39vwPxFGM8nyRAyBktxrKY7VQQBTpNiGpkTzz",
            "auth_hash": null,
            "created_at": "2020-10-08T00:35:45.337Z",
            "updated_at": "2020-10-08T20:29:31.804Z",
            "deleted": false
        }
    ],
    "api": "20190520"
}

Response:

{
    "retrieved_items": [],
    "saved_items": [],
    "conflicts": [
        {
            "server_item": {
                "uuid": "3413f65a-08fe-11eb-b705-d43d7edad4b3",
                "duplicate_of": null,
                "items_key_id": null,
                "content": "003:c1bac22d0e296f166884c8df5c04aebad787410db6427cf7692640e8f546ced8:3413f65a-08fe-11eb-b705-d43d7edad4b3:b8043006ad842e634b6dfd127f6cfdb1:Cv8tIBm8UNLdBh5/4us0tKOAIWyxlUrJ/wI3dxqqeETIQRbnybwNPrBXsqGiN7AoDsNMPNnMOS3DGvB5PKBkReWjaP1tPsfsYGQFWvE0o/dscty3RY+2DmjQdYyBKXQznzyhaiVXKLVNtCiOVs3LQ4kPk5thtzM5wVGqM1xBYgfRAj9UTLPIBUmlJ/ZSwuTzXtuWPtD9ulCEE7WhzBKggtxS3OBKeYItSmQkVdqLB94P9Fj3X2EIYJcMaPph2zEHFRzAlhNx81qh7fhzdrstsR3c+RpCJNCgpOP45SlAwmDXcDUWkIm8DtKnBKEYRwlg:eyJ2ZXJzaW9uIjoiMDAzIiwiaWRlbnRpZmllciI6InN0YW5kYXJkbm90ZXMtZnNAZG9tYWluLmNvbSIsInB3X25vbmNlIjoiM2VkYjVjNjdjZWM4NGU5MjZmNTcxZTNmM2JiZDYzOTNlZmYwZjkwNzMzYTlhMWNiODc5OGE2MGE5NDc2M2IwNSIsInB3X2Nvc3QiOjExMDAwMH0=",
                "content_type": "Note",
                "enc_item_key": "003:548c850a81deef754fac9d4044ed578f7a4020fe3d9f1a0ef0d5f46603be3a5c:3413f65a-08fe-11eb-b705-d43d7edad4b3:5624a4025d05b2c39cf95e5251256306:0hfptk9Flp/nvcRSA2TYNNW+aTaNUcSn9tIFt6z3XXouZ5qjAyVgKHD2gqmdKGY3iZObBdCbJDSmQTBAbu2y8uFPZd+r5Ygm0lxAnvKIYA53udjiBHIYT2gsXBxnhB0tbp2fmzRpK0ur2Ezixq2aOl0pUZWNi6Gu9CHQE9Q/5ZmN9Lth3kK2/cRSZcUPnnky:eyJ2ZXJzaW9uIjoiMDAzIiwiaWRlbnRpZmllciI6InN0YW5kYXJkbm90ZXMtZnNAZG9tYWluLmNvbSIsInB3X25vbmNlIjoiM2VkYjVjNjdjZWM4NGU5MjZmNTcxZTNmM2JiZDYzOTNlZmYwZjkwNzMzYTlhMWNiODc5OGE2MGE5NDc2M2IwNSIsInB3X2Nvc3QiOjExMDAwMH0=",
                "auth_hash": null,
                "created_at": "2020-10-08T00:35:45.337Z",
                "updated_at": "2020-10-08T20:12:16.899Z",
                "deleted": false
            },
            "type": "sync_conflict"
        }
    ],
    "sync_token": "MjoxNjAyMTg5MDAyLjcwMjIxNDU=\n",
    "cursor_token": null
}
tannercollin commented 4 years ago

Oh nevermind, I misread. I have to leave the updated_at value alone so it's the same as whatever the server sends.

Works now!

tannercollin commented 4 years ago

@nickAtWybar should be fixed in d306e149d1c7cbe763b3f4865d6b68c91e3a1573. Can you backup, update, and test?

nickAtWybar commented 4 years ago

seems to be working! Thanks gents once again for the quick response.

I did run into this once (in between 'Auto-syncing items...'), but didn't seem to have too much of an effect (it resumed):

ERROR : Uncaught exception from FUSE operation getattr, returning errno.EINVAL. Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/fuse.py", line 734, in _wrapper return func(*args, *kwargs) or 0 File "/usr/local/lib/python3.8/dist-packages/fuse.py", line 774, in getattr return self.fgetattr(path, buf, None) File "/usr/local/lib/python3.8/dist-packages/fuse.py", line 1027, in fgetattr attrs = self.operations('getattr', self._decode_optional_path(path), fh) File "/usr/local/lib/python3.8/dist-packages/fuse.py", line 1251, in call ret = getattr(self, op)(path, args) File "/usr/local/lib/python3.8/dist-packages/standardnotes_fs/sn_fuse.py", line 136, in getattr notes = self.item_manager.get_notes() File "/usr/local/lib/python3.8/dist-packages/standardnotes_fs/itemmanager.py", line 124, in get_notes notes = [(k, self.items[v]) for k, v in self.note_uuids.items()] File "/usr/local/lib/python3.8/dist-packages/standardnotes_fs/itemmanager.py", line 124, in notes = [(k, self.items[v]) for k, v in self.note_uuids.items()] RuntimeError: dictionary keys changed during iteration