Problem with <, > character

[vuquangtam]

When my note contains those character (<, >), the note updated will transform them into >, < (looks like html encode). I use simplenote for my org agenda so it wont work when i sync from server because the schedule for example: <01-01-2018 Sun> will be transform to <01-01-2018 Sun>.

[schmudde]

I can confirm this is a problem as well. Characters such as &, <, and > are now getting HTML encodes on every sync. Here is the problem, broken down:

• Pull a well-formed article with markdown, such as > for quotes. They are correct in webview.
• simplenote-browse → [Sync with server]
• I get the unusual sync error reported in Issue #31.
• I open the file via Simplenote2 in Emacs. The >s are replaces by >s
• The original document is still well-formatted.

Issue #31 has been happening since early January. This is new since early February.

Any ideas? It's happening during the sync process, it's a new bug, and I had not updated any packages. I updated all my packages today and am seeing the same issue.

[vuquangtam]

I found that this issue maybe cause by simplenote api server (maybe v2 api). I try using sncli (simplenote commandline, it uses v2 api) and this issue also occurs, but when try using package simplenote.el (it uses v1 api), it doesnt cause this issue. So i try making some requests to simplenote api url to get note data and in the note's content field of server response has HTML encodes.

[schmudde]

I'm having trouble finding the Simplenote API documentation. Could you point me in the right direction, @vuquangtam?

I wrote Simplenote Support last week and this is what they wrote me. It wasn't very helpful:

Simplenote uses Simperium to store and sync notes across iOS, Android, Mac, and the web. Unfortunately, the current version of the Simperium API does not allow for third party development of Simplenote apps.

We still have plans to implement a token-based system for developers to access our API but I don't have a time frame for this and we have limited resources right now.

[vuquangtam]

@schmudde The one that i found http://kendersec.github.io/SimpleNote/SimpleNote-API-v2.1.3.pdf This document is old but i've test some apis and they works. I also look at code of simplenote2.el and sncli.py for more information about this Simplenote API. Note: the get index api is limited to 100 so you need to request multiple time to get all note indexes.

[kngtani]

I'm having the same problem. Here are my observations:

• It seems updates made on web, iOS, Mac app takes about 10 min or so until it syncs to Simplenote2 in Emacs
• Some times the updates made on web, iOS, Mac are either truncated or sync'ed with an old version of the note
• Updates made on Simplenote2 shows up immediately once pushed from Simplenote2
• As long as the note is not updated outside of Simplenote2 there is no issue

[schmudde]

I can duplicate @kngtani's experience.

I followed up with Automattic to see if they'll acknowledge that the API exists and ask some specific questions. Specifically, I don't know why they would change anything on the API seeing that it is so old. I would understand killing it off, but I hope they will replace it first.

After they denied the existence of a public API that offers token-based syncing, I sent them the document @vuquangtam referenced. This is their response:

This seems to do all the token-based syncing. Is it too out of date to use?

That seems to be fairly outdated (it's from 2011), but I'm double-checking with one of our developers to give you a definitive answer. We'll reach out when we have news.

I'll post here when I hear something.

[schmudde]

After some back-and-forth with Automattic, I finally got a clear and honest response:

We did previously allow access to the Simplenote API but this access was removed for new developers a couple of years ago for security reasons. There are some existing third-party apps which were made at the time and which are still available like nvALT. It's on the road map to introduce a token-based system again but it's not a top priority right now and I don't have an ETA I can give you, sorry.

I confirmed that both regular simplenote2-notes-mode and markdown formatted simplenote2-markdown-notes-mode notes have the same issue.

I confirmed that Download .zip pulls the notes correctly, with <, >, and & intact.

Finally, I took a look at the simplenote2-sync-notes function, which seems to do most of the work. My <, >, and & characters get obliterated on the pull, so I focused on this part of the function:

;; Step2-3: Update notes on local which are older than that on server.
                (lexical-let (keys-to-update)
                  (if (not arg)
                      (dolist (elem index)
                        (let* ((key (car elem))
                               (note-info (gethash key simplenote2-notes-info)))
                          ;; Compare syncnum on server and local data.
                          ;; If the note information isn't found, the note would be a
                          ;; newly created note on server.
                          (when (< (if note-info (nth 0 note-info) 0) (cdr elem))
                            (message "Updated on server: %s" key)
                            (push key keys-to-update))))
                    (setq keys-to-update (mapcar (lambda (e) (car e)) index)))
                  (deferred:$
                    (deferred:next
                      (deferred:lambda ()
                        (when keys-to-update
                          (let (keys)
                            (dotimes (i 20) (and keys-to-update
                                                 (add-to-list 'keys (pop keys-to-update))))
                            (deferred:nextc
                              (deferred:parallel
                                (mapcar (lambda (key) (simplenote2--get-note-deferred key))
                                        keys))
self)))))

The same function also generates the error we're seeing:

(deferred:error it
        (lambda (err)
          (message "Sync notes error: %s" err)
(setq simplenote2--sync-process-running nil)))

I don't see anything suspicious in there. This hasn't been updated for some time, so it leads me to believe that it is a breaking change on their side, so it means we're SOL. I could not get Automattic to confirm or deny they changed anything on their old API. Other ideas?

[schmudde]

Two other notes -

1. It's worth noting that something is broken with how simplenote2.el indexes changes. When I sync, it always writes about a dozen changes locally, even if I've changed only one file on the web.
2. I tried reverting back to the previous change, the URL in #30, and recompiled with no luck.

Unfortunately, I think I'm going to have to leave Simplenote rather than guess at what changes may or may not have happened with their API.

[alpha22jp]

I think this issue has been fixed with the latest version since it uses new Simperium API.