port to simperium-python backend

samuelallan72 commented 6 years ago

what works:

auth
pulling/pushing notes (sort of)

what doesn't work:

reliable syncing

notes:

more information is returned from the index method, and less in the get method - v (the version) is only part of the index call.
no more syncnum and friends for syncing help. I'm guessing we'll need to use the since param (not a datetime) to index to help with partial syncing.
while the simperium api docs are comprehensive, there are no docs for simplenote specific things. For example, the delete endpoint returns a 400 client error on attempting to delete a note that doesn't have the deleted key set to true.

todo:

work out a syncing algorithm that fits with sncli's note model and simperium.
handle http errors (no connection, bad requests, not found, etc.)
test everything!

Thoughts, comments, code review, suggestions, etc. all appreciated! I'll need some help if I'm to complete this by October!

ref tracking issue #64

samuelallan72 commented 6 years ago

Everything seems to be mostly working now. I have hit a blocker though: creating new notes does not work. I have contacted @roundhill to see if he can help.

Basically the problem is that running something like:

api.note.new({'content': 'test'})

# or even
api.note.set('24a8924196ab4f6793d67819d46d06f5', {'content': 'test', 'tags': ['wat']}, include_response=True)

Does not work to create a new note. It simply returns a 400 client error status. I have also tried supplying all the fields following the example in the official electron client: https://github.com/Automattic/simplenote-electron/blob/99bca3f046d2fd6e3147f697728725140a5e4aac/lib/flux/app-state.js#L234-L273

Interested to find out why this is happening.

roundhill commented 6 years ago

Does not work to create a new note. It simply returns a 400 client error status.

Hey 👋 I think the schema requires creationDate, modificationDate, and content. Does it work if you set those three?

And you're right about deletion, it must have deleted set to true before Simperium will allow it to be deleted. The deleted property signifies that the Note is in the trash.

samuelallan72 commented 6 years ago

I think the schema requires creationDate, modificationDate, and content. Does it work if you set those three?

I had tried that before to no avail. After some more experimenting I worked out that it seems to require all fields to be set (including fields like publishURL). I was sure I already tested that, but it works this time. ¯\(ツ)\/¯

This latest commit seems to keep simperium happy. :) Thanks for your support @roundhill :+1:

N0ury commented 6 years ago

I have the following error:

Traceback (most recent call last):
  File "./sncli", line 32, in <module>
    sncli.main()
  File "/Users/nb/fouretout/sncli/simplenote_cli/sncli.py", line 1353, in main
    sncli(sync, verbose, config).gui(key)
  File "/Users/nb/fouretout/sncli/simplenote_cli/sncli.py", line 960, in gui
    'log'           : self.log
  File "/Users/nb/fouretout/sncli/simplenote_cli/view_titles.py", line 18, in __init__
    urwid.SimpleFocusListWalker(self.get_note_titles()))
  File "/Users/nb/fouretout/sncli/simplenote_cli/view_titles.py", line 143, in get_note_titles
    lines.append(self.get_note_title(n.note))
  File "/Users/nb/fouretout/sncli/simplenote_cli/view_titles.py", line 128, in get_note_title
    return urwid.AttrMap(self.format_title(note),
  File "/Users/nb/fouretout/sncli/simplenote_cli/view_titles.py", line 49, in format_title
    t = time.localtime(float(note['modificationDate']))
KeyError: 'modificationDate'

samuelallan72 commented 6 years ago

@nbenm aah oh no... this is probably because some keys have changed (modificationDate used to be modifyDate). Does it fix it if you delete your local database of notes and sync from scratch?

N0ury commented 6 years ago

Yes, it’s better.

Le 13 sept. 2018 à 07:47:52, Samuel Walladge notifications@github.com a écrit :

@nbenm aah oh no... this is probably because some keys have changed (modificationDate used to be modifyDate). Does it fix it if you delete your local database of notes and sync from scratch?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

N0ury commented 6 years ago

Why does sncli.log rotate so often? When I do a tail -f to see what’s happening, it hangs because of that. It was the same in previous version.

samuelallan72 commented 6 years ago

Logging is configured in simplenote_cli/sncli.py:

        self.loghandler = RotatingFileHandler(self.logfile, maxBytes=100000, backupCount=1)

It ends up being around 1000 lines for me before it rotates.

N0ury commented 6 years ago

If I move a note to trash on the web, and then empty trash, the json file remain locally.
I have imported all files in my personal account to the test one. All have been correctly imported

samuelallan72 commented 6 years ago

If I move a note to trash on the web, and then empty trash, the json file remain locally.

Does it get deleted locally after restarting sncli?

All have been correctly imported

:tada: :smile:

samuelallan72 commented 6 years ago

Fixed a couple of bugs I discovered during test. Hopefully should be fairly stable now. More testing and feedback appreciated! :)

N0ury commented 6 years ago

If I move a note to trash on the web, and then empty trash, the json file remain locally.

Does it get deleted locally after restarting sncli?

All have been correctly imported

🎉 😄

Yes it is deleted when I restart sncli. But it is not deleted otherwise...

samuelallan72 commented 6 years ago

@nbenm I'm not sure how to fix that... looking at the code, this appears to be the original behaviour too. Deleted notes from the server are only ever removed locally on a full server sync, and that only happens when sncli first starts. Maybe open a new issue and we can tackle that one later, since it doesn't appear to be an issue with the new api code?

N0ury commented 6 years ago

Ok, you are right. We are only testing the move to the new Simperium api. I’ll open an issue.

samuelallan72 commented 6 years ago

I'm removing the 'work in progress' tag because I feel it is complete now. I won't merge to master until some more testing has been done and everyone is happy with how it is working. We have until october anyway. :smile:

To anyone new reading this: testers wanted! Please checkout the next branch and let me know of any issues you discover. More testing now equals less grief later. Also if you're keen, the simperium-python3 library could do with some looking over to catch anything I missed while porting from the official version.

Finally something to note is that sncli now depends on the afore mentioned library. This means more difficulty for anyone installing through a linux system package manager (eg. through the AUR package), since simperium-python3 is currently only available through pypi or the git repo. Perhaps it could be useful to include directly with the source as a git submodule or similar? Interested in your thoughts.

N0ury commented 6 years ago

All tests I have done are ok for me. I haven't tested everything. Good job!

N0ury commented 6 years ago

When looking for previous version of a note (with '<'), I have:

Traceback (most recent call last):
  File "/usr/local/bin/sncli3", line 32, in <module>
    sncli.main()
  File "/usr/local/bin/simplenote_cli3/sncli.py", line 1353, in main
    sncli(sync, verbose, config).gui(key)
  File "/usr/local/bin/simplenote_cli3/sncli.py", line 1056, in gui
    self.sncli_loop.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urwid-2.0.1-py3.7-macosx-10.13-x86_64.egg/urwid/main_loop.py", line 286, in run
    self._run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urwid-2.0.1-py3.7-macosx-10.13-x86_64.egg/urwid/main_loop.py", line 384, in _run
    self.event_loop.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urwid-2.0.1-py3.7-macosx-10.13-x86_64.egg/urwid/main_loop.py", line 788, in run
    self._loop()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urwid-2.0.1-py3.7-macosx-10.13-x86_64.egg/urwid/main_loop.py", line 825, in _loop
    self._watch_files[fd]()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urwid-2.0.1-py3.7-macosx-10.13-x86_64.egg/urwid/raw_display.py", line 404, in <lambda>
    event_loop, callback, self.get_available_raw_input())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urwid-2.0.1-py3.7-macosx-10.13-x86_64.egg/urwid/raw_display.py", line 502, in parse_input
    callback(processed, processed_codes)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urwid-2.0.1-py3.7-macosx-10.13-x86_64.egg/urwid/main_loop.py", line 411, in _update
    self.process_input(keys)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urwid-2.0.1-py3.7-macosx-10.13-x86_64.egg/urwid/main_loop.py", line 511, in process_input
    k = self._topmost_widget.keypress(self.screen_size, k)
  File "/usr/local/bin/simplenote_cli3/sncli.py", line 907, in gui_frame_keypress
    self.gui_update_status_bar()
  File "/usr/local/bin/simplenote_cli3/sncli.py", line 308, in gui_update_status_bar
    self.gui_header_set(self.gui_body_get().get_status_bar())
  File "/usr/local/bin/simplenote_cli3/view_note.py", line 135, in get_status_bar
    t = time.localtime(float(self.old_note['versiondate']))
KeyError: 'versiondate'

N0ury commented 6 years ago

I have now, an auth issue. Time is GMT+2.

2018-09-14 23:14:32,583 [DEBUG] Starting new HTTPS connection (1): auth.simperium.com:443
2018-09-14 23:14:44,888 [DEBUG] https://auth.simperium.com:443 "POST /1/chalk-bump-f49/authorize/ HTTP/1.1" 500 None
2018-09-14 23:14:44,891 [DEBUG] 500 Server Error: INTERNAL SERVER ERROR for url: https://auth.simperium.com/1/chalk-bump-f49/authorize/
2018-09-14 23:14:46,005 [DEBUG] Sync worker: started
2018-09-14 23:14:46,006 [DEBUG] Starting full sync
2018-09-14 23:14:46,009 [DEBUG] Starting new HTTPS connection (1): auth.simperium.com:443
2018-09-14 23:14:46,730 [DEBUG] https://auth.simperium.com:443 "POST /1/chalk-bump-f49/authorize/ HTTP/1.1" 500 None
2018-09-14 23:14:46,734 [DEBUG] ERROR: Failed to get note list from server
2018-09-14 23:14:46,734 [DEBUG] Full sync completed
2018-09-14 23:15:01,740 [DEBUG] Starting full sync
2018-09-14 23:15:01,771 [DEBUG] Starting new HTTPS connection (1): auth.simperium.com:443
2018-09-14 23:15:02,707 [DEBUG] https://auth.simperium.com:443 "POST /1/chalk-bump-f49/authorize/ HTTP/1.1" 500 None
2018-09-14 23:15:02,712 [DEBUG] ERROR: Failed to get note list from server
2018-09-14 23:15:02,713 [DEBUG] Full sync completed

It seems to be a Simperium issue. I've tried manually. I can do "token = auth.create(....)" but not "token = auth.authorize(....)" with an existing user. Error message is:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simperium/core.py", line 60, in authorize
    r.raise_for_status()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py", line 939, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://auth.simperium.com/1/zone-stowage-192/authorize/

samuelallan72 commented 6 years ago

@nbenm thanks; fixed the viewing previous note issue. :+1:

Yes, the auth issue appears to be on Simperium's end, and is back online for me now.

N0ury commented 6 years ago

If in sncli I add a new tag (ie doesn't exist), on the web I can see it on the note. But it doesn't appear in the list of tags, when I click on "Tags".

[Edit]: it was the same on old version...

samuelallan72 commented 5 years ago

:tada: merged to master and made a new release. Hopefully it will be a smooth transition for users. Release notes and instructions added to https://github.com/insanum/sncli/releases/tag/0.3.0

Please open new issues for any discovered problems. :)

insanum / sncli

port to simperium-python backend #67