Open rdhyee opened 3 years ago
Unfortunately version 004 isn't supported by snfs :(
This goes for all the alternative SN clients. I don't think it's likely any of them will be updated to support 004 because of the amount of work involved.
@tannercollin Thanks for letting me know that it would a lot of work to get support in place for 004. I'm going to roll back to 003 because snfs is such an important part of my workflow!
@tannercollin Thank you so much for snfs. I'm going to close this issue because for the time being it sounds like the best thing I can do right now is roll back to 003 and live with that.
You're welcome!
I just fixed a bug and also added an error message if 004 is detected. Please upgrade snfs.
@mobitar is there any way to downgrade an account to 003 in case a person doesn't have backups?
You can (at your own risk) perform the following steps:
Your new account will use 003.
@tannercollin Thanks again for snfs
and for continuing to fix bugs. I've updated to the new code but I still can't get snfs
to work with my recovered setup. I'm going to try again with the advice from @mobitar -- I need to try a pre-3.5 desktop release.
Partial success. I logged off of SN everywhere, deleted my account, set up a new account using SN desktop on my mac (version 3.4.10). snfs works again for me. (big yay). I restored my backup that uses 003. app-legacy.standardnotes.org also works.
What doesn’t work fully:
Now, I'm getting closed to getting things working like they were. More success: I updated my mac SN client to 3.5.14 and that enabled me to get my Android SN client to be happy again. Also app.standardnotes.org started working for me. Presumably the mac app will start working for me soon….
The newest mobile app versions are fine working with 003? They don't try to upgrade the encryption?
@tannercollin The newest apps work fine for me after I used the 3.4.10 macos desktop app to load up my 003 backups and then upgrade my desktop app to 3.5.15 to get some magic cleaned up version to the server, which the latest Android and iPadOS apps could then be happy with. Does that make sense? Getting the sequencing of restoring my set up was the painful part here.
@mobitar @tannercollin Well, that was an adventure. I had grown so dependent on running Standard Notes on my two laptops, my Android phone, my iPad + snfs on my laptops that it was disruptive to have had part of that workflow fall apart when I thoughtlessly upgrade from 003 to 004. (I'm generally conservative about upgrading precisely because I want to avoid untimely disruptions like this.)
@mobitar It seems that in the longer time frame, if people want to keep running snfs, we'll have to implement 004. What pointers can you supply me so that I can see what I can to do that implementation (with guidance from @tannercollin if he has time.) I'm motivated to do an implementation if it's not way too hard and if there's enough time to do it before the plug on 003 is pulled.
I'd love it if you implemented 004 and I can provide guidance. There used to be a spec at this link, but it must have been moved:
https://github.com/standardnotes/snjs/blob/004/specification.md
Edit, found it here: https://github.com/standardnotes/snjs/blob/master/specification.md
Also note that we have the luxury of only implementing the bare minimum to get it to work. For example, we don't have to implement 003 -> 004 migration because we can tell the user to do it in an official client, etc.
My number one advice is to find a way to get JavaScript running in your env ;) This way you can take advantage of using SNJS. But probably not realistic in this case. Otherwise, there's really no way around it: you have to read the specification, and build the equivalent in Python (while also using our implementation as a guide).
Wow, Standard Notes really screwed us here. I hope this new encryption is absolutely amazing and fixes all kinds of security issues, because it just totally hosed the workflow of lots of people. I'll switch to a new system before I backup/delete account/restore new account/whatever.
Does anyone have any ideas for a way to share secrets between a local filesystem/web/mobile that won't be arbitrarily hosed at random? Might have to switch to something really heavy and obnoxious like One Password. Arg.
@pkulak As I wrote on the SN tools channel (https://standardnotes.slack.com/archives/CP3U7DY2D/p1611253197002500):
Anyone else out there enjoying the "Note to Self" feature in Signal (https://support.signal.org/hc/en-us/articles/360043272451-Note-to-Self) -- especially combined with the ability to delete messages (https://support.signal.org/hc/en-us/articles/360007320491-Delete-messages-and-alerts)? I used this a lot to send a private key around when configuring Standard Notes on my various devices.
This use of Signal is to share the occasional secret between devices for me. I personally also use 1password.
Does Signal note-to-self support local filesystem?
You could also self-host Bitwarden. It has a command line client that you could write a script around.
@rdhyee are you still interested in helping me implement 004? Do you have Telegram Messenger?
@tannercollin I'm still interested in doing the implementation but haven't had time to dig in yet. I do have Telegram.
I just started to wade into this issue a bit last night. I figure that I should first teach myself the ins and outs of how Encryption Protocol Specification v003 | Standard Notes Documentation works before studying Encryption Protocol Specification v004 | Standard Notes Documentation. And I can convince myself how to use the pieces in the current snfs to decrypt a static download of my notes before then setting up a new test Standard Notes account (using the 004
encryption version)) and figuring out how to decrypt and then encrypt notes using Python.
In other words, I'm just warming up to tackle this implementationl
Awesome, thanks @rdhyee !
As I reported on slack (https://standardnotes.slack.com/archives/C3KQ7V2H2/p1612370087018400?thread_ts=1612288313.016100&cid=C3KQ7V2H2):
I installed snjs yesterday but couldn’t figure out how to use it — my JS skills are primitive, and I figure that I need to learn how to use JS modules and try again.
I also haven’t been able to write the Python version of decrypt.html.
What would be really helpful is worked-out, simplest examples of encryption and decryption for 003 and 004 using specific values of uip, pw_cost, pw_nonce so that I can verify that I get the easiest case working.
I plan to write out more fully what I’ve attempted so far and where I’ve gotten stuck.
I've implemented a protocol v004 decryption demo in Python 3 here:
https://gist.github.com/tannercollin/d7474c6afba0dc2026dd996b9dedf197
Save it as a .py file and install the requirements listed. You can go through it step by step to see how the crypto works.
I've not forgotten about this issue. I'm getting some experience working with the sync API on 003 and then will work on integrating 004 into the codebase -- unless someone beats me to it.
Hello @rdhyee @tannercollin 😀
Any news about this issue ?
I can't help technically but I can help with a bounty if needed.
Thanks !
@johackim @tannercollin I've not forgotten about this issue but have been struggling to find time to work on it. I hope to get some time in the next week or two.
@tannercollin OK, a first very rough cut at incorporating your demo code for v004 into snfs: https://github.com/tannercollin/standardnotes-fs/compare/master...rdhyee:v004
I was able to run
import logging
import standardnotes_fs
from standardnotes_fs import api
from standardnotes_fs.api import SNAPIException, StandardNotesAPI
from standardnotes_fs.itemmanager import ItemManager
from standardnotes_fs.crypt import EncryptionHelper
username = "[USERNAME]"
password = "[PASSWORD]"
sync_url = "https://sync.standardnotes.org"
keys = None
try:
sn_api = StandardNotesAPI(sync_url, username)
if not keys:
keys = sn_api.gen_keys(password)
del password
keys = sn_api.sign_in(keys)
log_msg = 'Successfully logged into account "%s".'
logging.info(log_msg % username)
login_success = True
except SNAPIException as e:
print(e)
except ConnectionError:
log_msg = 'Unable to connect to the sync server at "%s".'
print(log_msg % sync_url)
sys.exit(1)
except MissingSchema:
log_msg = 'Invalid sync server url "%s".'
print(log_msg % sync_url)
sys.exit(1)
ext = ".md"
item_manager = ItemManager(sn_api, ext)
item_manager.sync_items()
print(item_manager.get_notes())
@tannercollin OK, a first very rough cut at incorporating your demo code for v004 into snfs: master...rdhyee:v004
I was able to run
import logging import standardnotes_fs from standardnotes_fs import api from standardnotes_fs.api import SNAPIException, StandardNotesAPI from standardnotes_fs.itemmanager import ItemManager from standardnotes_fs.crypt import EncryptionHelper username = "[USERNAME]" password = "[PASSWORD]" sync_url = "https://sync.standardnotes.org" keys = None try: sn_api = StandardNotesAPI(sync_url, username) if not keys: keys = sn_api.gen_keys(password) del password keys = sn_api.sign_in(keys) log_msg = 'Successfully logged into account "%s".' logging.info(log_msg % username) login_success = True except SNAPIException as e: print(e) except ConnectionError: log_msg = 'Unable to connect to the sync server at "%s".' print(log_msg % sync_url) sys.exit(1) except MissingSchema: log_msg = 'Invalid sync server url "%s".' print(log_msg % sync_url) sys.exit(1) ext = ".md" item_manager = ItemManager(sn_api, ext) item_manager.sync_items() print(item_manager.get_notes())
Nice work, keep going !
Looks great so far, I've pulled it into a v004 branch in the main repo.
Hello @tannercollin and @rdhyee ;)
I tested the v004 branch and I have an error :
Processing item xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
content_type Note
Decrypting enc_item_key
Traceback (most recent call last):
File "/home/johackim/Downloads/standardnotes-fs/env/bin/snfs", line 8, in <module>
sys.exit(main())
File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/standardnotes_fs.py", line 260, in main
fuse = FUSE(StandardNotesFUSE(sn_api, sync_sec, args.ext),
File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/sn_fuse.py", line 26, in __init__
self.item_manager = ItemManager(sn_api, ext)
File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/itemmanager.py", line 272, in __init__
self.sync_items()
File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/itemmanager.py", line 96, in sync_items
response = self.sn_api.sync(dirty_items)
File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/api.py", line 159, in sync
return self.handle_response_items_004(response)
File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/api.py", line 215, in handle_response_items_004
version, nonce, ciphertext, encoded_authenticated_data = enc_item_key.split(":")
ValueError: too many values to unpack (expected 4)
Thanks for your time !
Any update on this ? This is a great tool and too unfortunate to not be able to use in SN..
Thank you !
@krillin666 thanks for your patience -- I've not had time to work on this issue. But fixing snfs for 003 (https://github.com/tannercollin/standardnotes-fs/pull/35) has gotten me back into this code base -- so I hope to keep going and finish up this work. No promises though....
Any news ? @rdhyee
@johackim I've not made any progress on this issue because snfs
is broken right now for 003
(because of API changes) and I've not yet been able to fix the problem for 003
. I've not given up yet but haven't found time to fix the problem given how busy I've been.
Hello all, any news about this ? 😀
@johackim Sorry, no progress to report. I've been so busy with my other responsibilities that I've not found time to work on this issue. Between not being able to run snfs
and not being able to run something else of importance to me (https://github.com/bordaigorl/sublime-evernote/issues/219#issuecomment-894776245), I'm in the middle of migrating my note-taking from Evernote and Standard Notes to Obsidian.md -- where having a collection of interlinked markdown files in a file system is the centerpiece (and not tacked on).
It's exactly why I need this fix, I try to migrate my notes from Standard Notes to Obsidian too and I don't know how to do it quickly. Any idea ?
@johackim The approach I'm exploring right now -- use Standard Notes to download a decrypted backup of all your notes -- see https://standardnotes.com/help/14/how-do-i-create-and-import-backups-of-my-standard-notes-data. The menu flow I used was
Account -> Data Backups -> Download Backup (Decrypted)
You get a zip file that you can then unzip.
I'm going then look into writing Python scripts to ingest the unzipped files into a format to import into Obsidian.
Hope that helps.
After I updated the encryption version from
004
from003
(https://standardnotes.org/help/security), I started to encounter the following exception when runningstandardnotes-fs
:Has anyone else run into this problem? I've been running
standardnotes-fs
with few problems for over a year before today. I'll try to debug and fix this issue.