Notes being deleted from Dropbox while using NVAlt

[usethedata]

I've corresponded with Brett about this earlier, and the problem has occurred now with 2.2b (101), so I'm posting this here as a more formal issue report.

Synopsis: Notes are being deleted from the Dropbox folder where I store text notes. I have only seen this happen when using nvALT and it's the active note that I'm editing that seems to be the issue. I am using Dropbox for sync, using a folder "~/Dropbox/Text_Notes". SimpleNote synchronization is off. I do (often) have nvALT open on multiple computers. Today was the first time this has happened since I moved to the 2.2 beta stream about 3 weeks ago. I am running Lion on 2 of the 3 computers (latest release) and Mountain Lion on the third. I do have a couple of iOS apps that use that folder, but I've not seen this happen any time I'm actually using one of those apps.

Symptoms: When editing a note, a few times, I've seen the note just disappear from nvALT. The note is in the trash, and Dropbox does show the file as having been deleted, if I go into the web interface to recover the file. A few times, I've made changes to the note, moved focus away from nvALT, and I get a Growl notification that a file was deleted. In today's case, the file disappeared while I was editing it in nvALT on my MacBook (Lion). I know one other computer has nvALT open (an iMac on Lion). The third computer is shut down at the moment, so only the two are in play.

In that dropbox folder, I do see a file called Interim Note-Changes (Bruce Wilson's conflicted copy 2013-01-20), which is perhaps an issue.

Any particular suggestions on particular things to look for or logs to check to shed more light on this to help sort this out? This does seem to be a different issue from #110, as that seemed to be more about SimpleNote sync, which I'm not using.

[usethedata]

Happened again today. Crawling through Console, I find these lines that seem relevant from the incident three days ago and today:

Jan 20 11:34:46 lap0089132bew nvALT[9412]: FILE WAS MODIFIED: Shopping list.txt Jan 20 11:39:08 lap0089132bew nvALT[9412]: not writing note Shopping list because it is not controlled by NoteController Jan 20 11:39:11 lap0089132bew nvALT[9412]: Note was not visible--showing all notes and trying again Jan 20 12:31:48 lap0089132bew nvALT[9412]: not writing note Shopping list because it is not controlled by NoteController Jan 22 15:47:06 lap0089132bew nvALT[9412]: Could not get an fsref for file with name Genesis Notes.txt: -43 Jan 22 15:47:06 lap0089132bew nvALT[9412]: Couldn't move file to trash: -43 Jan 22 15:47:06 lap0089132bew nvALT[9412]: not writing note Genesis Notes because it is not controlled by NoteController

And there's another copy of Interim Notes-Changes, with a conflicted copy from the 20th

[usethedata]

Dug into this a bit further. I was editing the file on my laptop (lap0089132bew) and Dropbox indicates that the file was deleted by my iMac. So it looks like the system that's not editing the file is the one that's causing the deletion on Dropbox.

[rickarddahlstrand]

I had this issue as well, the last beta seems to be OK. But I can't be sure.

[ttscoff]

I'm having trouble with this myself. I haven't figured out why it happens, but notes are disappearing as I'm writing them. Will keep you posted.

On Mon, Feb 4, 2013 at 4:45 AM, Rickard Dahlstrand <notifications@github.com

wrote:

I had this issue as well, the last beta seems to be OK. But I can't be sure.

— Reply to this email directly or view it on GitHubhttps://github.com/ttscoff/nv/issues/129#issuecomment-13070524.

[arice]

I'm in the same boat. If I can help debug, let me know.

[acsexton]

Same -- happens to me frequently while using another note editor (resophnotes) on my laptop. Dropbox always reports that my mac running nvALT is the one doing the deletions.

[elasticthreads]

How many of you guys store your notes as a single database and how many as individual text files?

[arice]

I use individual text files.

On Mar 13, 2013, at 10:21 PM, David notifications@github.com wrote:

How many of you guys store your notes as a single database and how many as individual text files?

— Reply to this email directly or view it on GitHub.

[ryanfitzer]

Same issue. Mac at home deletes the newly created file that I'm editing on my Mac at work. Using text files and DropBox syncing. Also using Notesy on my iPhone 5, though I haven't seen any issues with it.

I've also noticed that I often get conflicts with the "Interim Note-Changes" file. My sense is that this is part of the issue. Are these generated by nvAlt? I figure Dropbox isn't able to diff these due to them being binary files.

Hope that helps the debugging.

[juranta]

Yes, nvALT creates "Interim Note-changes" and "Notes and Settings". The problems isn't just that it's binary, Dropbox handles the text files the same way: If there is a conflict, then it keeps the other version and creates a copy of the other version. It doesn't even try to merge conflicting text files which is a good thing.

One problem with this kind of 'block' files is that if two macs have a common base version and then one Mac changes one part of the binary and the other changes other part of the binary, and then Dropbox does the sync, it's always of conflict for the file. With separate text files, it's far less likely that the Macs modify the same files concurrently, and even if there is a conflict, it's fairly easy to handle.

Even without conflicts, it doesn't seem like a good idea to sync files such as Interim between two nvALTs. I understand that it holds some temporary note data which gets flushed later to that other file. Why would one nvALT want that Dropbox pushes to it other nvALT's Interim changes? Eek.

nvALT (Notational Velocity) is very quick and good in importing changes made to the text files in other programs or pushed by Dropbox, at least as long you stick to plain text/Markdown and OpenMeta tags. I think it would be a good idea to keep the text notes in Dropbox but somehow move the Interim file and the other Notes database file totally out of Dropbox, maybe in ~/Library. But this would change nv's normal functioning quite a lot. There isn't even a way that I know of to prevent these files syncing through Dropbox.

[elasticthreads]

I'm working on a fix to nvALT to keep this from happening and stabilize dropbox syncing. For now, that revolves around storing "Interim Note-changes" in ~/Library/Caches/ instead of the user's notes folder.

If you've experienced any lost notes, or are interested in testing this, please email me elasticthreads att gmail with the subject line "Test Dropbox sync" and I'll send out a beta in the near future. This is mainly for Dropbox sync so please only respond if you use that. Please include in this email the following information:

• what version of OS X you're running
• what hardware you use (i.e. "2012 Macbook Pro", or even better "Macbook Pro 5,1")
• what format notes database you use (i.e. single notes database or "individual text files with .txt extension")
• if you're using Dropbox + nvALT across multiple macs (and include the above 3 pieces of info for each mac you use)

thanks! David Halter aka elasticthreads

[anatomatic]

For those forgetful notetakers who leave nvALT running on multiple Macs from time-to-time:

I adapted an applescript to quit nvALT when computer is idle for 20 minutes (you can change your idle time). See it here: https://gist.github.com/anatomatic/5736900

[TALlama]

Any update on this? Is master still having this issue?

[adrianduke]

+1 on this issue. I use single database storage. Started to notice parts of some of my notes (recently edited ones) disappearing.

[elasticthreads]

@adrianduke I highly recommend that you do not mix single database storage and dropbox sync. There's a lot of potential to lose your notes when they're all contained in a single file that was never designed for diff-changes cloud sync. nvALT uses Notational Velocity's database model which goes back to 10.4, way before Dropbox. If you want to use the single database format, you should use SimpleNote sync.

[errriclee]

I just had it happen again to me.

Both using nvALT 2.2b(111), OSX 10.8.4. One is MacBookPro7,1 (editing) and the other is MacBookPro9,1. I am using individual plain text files for notes. Dropbox versions are 2.0.22 (editing) and 2.0.26.

I had created a new file Moving.txt on the editing computer and was making changes to it. The other computer had screensaver on the whole time and was not touched. nvALT was in the background.

Saw this log on computer editing file: 2:35:08 AM nvALT: Could not get an fsref for file with name Moving.txt: -43 2:35:08 AM nvALT: Couldn't move file to trash: -43

And on computer not editing file (nvAlt just running in background): 9/4/13 2:30:48.788 AM nvALT[29393]: Could not get an fsref for file with name Moving.txt: -43 9/4/13 2:30:48.788 AM nvALT[29393]: Couldn't move file to trash: -43 9/4/13 2:31:02.376 AM nvALT[29393]: FILE WAS MODIFIED: Moving.txt 9/4/13 2:31:05.887 AM nvALT[29393]: FILE WAS MODIFIED: Moving.txt 9/4/13 2:31:12.777 AM nvALT[29393]: FILE WAS MODIFIED: Moving.txt 9/4/13 2:32:10.901 AM nvALT[29393]: Could not get an fsref for file with name Moving.txt: -43 9/4/13 2:32:10.901 AM nvALT[29393]: Couldn't move file to trash: -43

(at this point I saw the file got deleted via Growl and I had to go to dropbox.com to restore it)

(and then some more editing...) 9/4/13 2:54:13.026 AM nvALT[29393]: FILE WAS MODIFIED: Moving.txt 9/4/13 2:54:17.019 AM nvALT[29393]: FILE WAS MODIFIED: Moving.txt 9/4/13 2:54:30.036 AM nvALT[29393]: FILE WAS MODIFIED: Moving.txt 9/4/13 4:17:09.848 AM nvALT[29393]: FILE WAS MODIFIED: Moving.txt 9/4/13 4:18:22.933 AM nvALT[29393]: Could not get an fsref for file with name Moving.txt: -43 9/4/13 4:18:22.933 AM nvALT[29393]: Couldn't move file to trash: -43

[beaugunderson]

@elasticthreads Do we still need to ask for a version with the potential fix? In the release notes for Version 2.2b (111) it looks like the ~/Library/Caches change is there.

[juranta]

I have iMac, Macbook, iPhone and iPad. I dare not to sync two macs with any version of nvALT. However, I just need nvALT on both of my Macs, so I decided to go back to Simplenote sync after their new update on iOS and also after reading Brett's update that nvALT is now using a newer Simplenote api. So, I currently use nvALT on two Macs with Simplenote sync. I have also set my recent iMac to use its database as blown text format in my Dropbox folder, it seems that somehow iMac with nvALT keeps working on the notes even when iMac is in sleep. Strange but useful. In any case, in the current situation, I'd say if you want to use nvALT in two or more Macs, perhaps it's better to use current Simplenote sync instead of Dropbox. ?

[jeff-h]

Just thought I'd add my 2c; I (re)discovered NVAlt about a month ago and gleefully unaware of this issue I started sharing notes between my iMac and my Air, where NVAlt is always running on both machines. I first tried Bittorrent Sync but that started losing my tag metadata (worked perfectly otherwise), so I switched to Copy (http://www.copy.com) as it was the latest of a long line of similar services I play with periodically.

It's worked flawlessly for me with Copy between my two Macs each permanently running NVAlt.

Haven't tried it in Dropbox.

[unclespeedo]

has anyone come up with a workaround for this? (I saw the simplenote suggestion, hoping to avoid an additional dependency)

[juranta]

No. I just read what I had written above, I want to kind of "retract" what I've seemingly said earlier in this thread concerning my own situation. The Simplenote sync didn't really work for me personally after all, I don't really remember all of the issues right now. My "workaround" is currently that I use nvALT only on my iMac to save the notes in a Dropbox folder. I don't use nvALT at all with my MacBook. It kind of sucks. :(

nvALT is totally great in importing changes in its notes folder in the "open text format", it seems to happen almost immediately. If there was just a way to prevent Dropbox from syncing that "Notes & Settings" file, it'd work much more smoothly as I can see, in most of the cases. It'd work just the same way as, for instance, Notesy with nvALT. Unfortunately, I haven't figured a way to do that.

[jeff-h]

As a followup to my comment above, I can confirm that after 6 months solid use, Copy (http://copy.com) is still working very well with nvALT between two Macs both permanently running nvALT.

[lastobelus]

@jeff-h is there any way to make a Copy-syncing setup work with an iOS note app that you know of? I was using Copy + simplenote sync, but simplenote sync kills my taskpapers (it converts tabs to spaces). Now stuck between a rock and a hard place, trying to achieve the goal of being able to edit notes AND taskpaper docs from 2 macs & 2 ios devices. (there is absolutely zero chance of my remembering to quit nvalt from a mac I'm not using, lol)

[acsexton]

I wish there was more I could advise, but I just got fed up with it, switched to evernote and it solved all my problems.

On Mon, May 12, 2014 at 3:37 PM, Michael Johnston notifications@github.comwrote:

@jeff-h https://github.com/jeff-h is there any way to make a Copy-syncing setup work with an iOS note app that you know of? I was using Copy + simplenote sync, but simplenote sync kills my taskpapers (it converts tabs to spaces). Now stuck between a rock and a hard place, trying to achieve the goal of being able to edit notes AND taskpaper docs from 2 macs & 2 ios devices. (there is absolutely zero chance of my remembering to quit nvalt from a mac I'm not using, lol)

— Reply to this email directly or view it on GitHubhttps://github.com/ttscoff/nv/issues/129#issuecomment-42877802 .

[ttscoff]

Dropbox issues have resolved for me, haven't seen a note disappear for a long time. Using nvALT on two machines and syncing to Notesy on iOS.

-Brett

On 12 May 2014, at 14:37, Michael Johnston wrote:

@jeff-h is there any way to make a Copy-syncing setup work with an iOS note app that you know of? I was using Copy + simplenote sync, but simplenote sync kills my taskpapers (it converts tabs to spaces). Now stuck between a rock and a hard place, trying to achieve the goal of being able to edit notes AND taskpaper docs from 2 macs & 2 ios devices. (there is absolutely zero chance of my remembering to quit nvalt from a mac I'm not using, lol)

---

Reply to this email directly or view it on GitHub: https://github.com/ttscoff/nv/issues/129#issuecomment-42877802

[lastobelus]

@ttscoff thanks, that's good to know! I'll give it a shot.

[jeff-h]

@lastobelus — the Copy app does have an iOS client, but the obvious question then is what to do with the files on the iOS device. Is there a chance an iOS text editing app could simply modify the .rtf files synced to the device by Copy (or Dropbox, or whatever)?

I don't use my iOS device for this so I can't really add much value to the question, sorry.

[jeff-h]

Incidentally, the most recent version of Bittorrent Sync now supports OS X extended attributes. See http://forum.bittorrent.com/topic/29236-extended-attributes-now-supported-details/ for a discussion.

I haven't tried it, but I expect this should mean that BTSync would now be able to sync NVAlt docs including the tag metadata.

[ttscoff]

I've tested and it doesn't seem to sync tags. I'd be pretty excited if I'm wrong.

[acsexton]

I've been missing plaintext, I'll have to give it another try. Thanks for the heads up!

On Monday, May 12, 2014, Brett Terpstra notifications@github.com wrote:

Dropbox issues have resolved for me, haven't seen a note disappear for a long time. Using nvALT on two machines and syncing to Notesy on iOS.

-Brett

On 12 May 2014, at 14:37, Michael Johnston wrote:

@jeff-h is there any way to make a Copy-syncing setup work with an iOS note app that you know of? I was using Copy + simplenote sync, but simplenote sync kills my taskpapers (it converts tabs to spaces). Now stuck between a rock and a hard place, trying to achieve the goal of being able to edit notes AND taskpaper docs from 2 macs & 2 ios devices. (there is absolutely zero chance of my remembering to quit nvalt from a mac I'm not using, lol)

---

Reply to this email directly or view it on GitHub: https://github.com/ttscoff/nv/issues/129#issuecomment-42877802

— Reply to this email directly or view it on GitHubhttps://github.com/ttscoff/nv/issues/129#issuecomment-42886811 .

[jeff-h]

@ttscoff — are you saying you've tried the most recent version of Bittorrent Sync and tags didn't appear to sync?

[ttscoff]

Yeah, I set up fswatch and started tagging things in a sync folder and watching them move between two macs. All changes showed up, but tags didn't transfer. They weren't lost, though, they just didn't make the jump.

On 15 May 2014, at 16:40, jeff-h wrote:

@ttscoff — are you saying you've tried the most recent version of Bittorrent Sync and tags didn't appear to sync?

---

Reply to this email directly or view it on GitHub: https://github.com/ttscoff/nv/issues/129#issuecomment-43268710

[jeff-h]

Ah — bit disappointing. Stuck with the commercial sync services for now I guess.

Somewhat off-topic, I'm intrigued by the idea of using fswatch (and node.js probably) to copy notes into different folders based on their tag. These separate folders could then be synced separately to different people, thereby allowing one NVAlt to sync notes in a fine-grained manner amongst different people. Any thoughts?

[ttscoff]

Have you seen this? Could fairly easily be subbed for an openmeta command line query vs title prefixes:

http://brettterpstra.com/2014/05/04/folderize-sync-nvalt-notes-to-nested-folders/

On 15 May 2014, at 17:30, jeff-h wrote:

Ah — bit disappointing. Stuck with the commercial sync services for now I guess.

Somewhat off-topic, I'm intrigued by the idea of using fswatch (and node.js probably) to copy notes into different folders based on their tag. These separate folders could then be synced separately to different people, thereby allowing one NVAlt to sync notes in a fine-grained manner amongst different people. Any thoughts?

---

Reply to this email directly or view it on GitHub: https://github.com/ttscoff/nv/issues/129#issuecomment-43272990

[jeff-h]

Woah, awesome!

That has now made me wonder whether it would be possible to do some really awesome stuff without any external sync solution (Dropbox etc) and without any filesystem watching IF the following were implemented...

Could NVAlt be extended to run a user-specified shell script with useful args any time a note is added, updated, or deleted? Using NSTask, I guess?

If so, we could use a shell script to:

• rsync the notes anywhere
• examine its metadata or contents and copy it anywhere
• implement all kinds of custom behaviour such as FTPing notes if they are tagged a certain way, or whatever

Bit of a geeky idea... but NVAlt is the geek's note solution :D

[ttscoff]

Without waiting for anyone to update nvALT, just use Hazel.

On 15 May 2014, at 17:57, jeff-h wrote:

Woah, awesome!

That has now made me wonder whether it would be possible to do some really awesome stuff without any external sync solution (Dropbox etc) and without any filesystem watching IF the following were implemented...

Could NVAlt be extended to run a user-specified shell script with useful args any time a note is added, updated, or deleted? Using NSTask, I guess?

If so, we could use a shell script to:

• rsync the notes anywhere
• examine its metadata or contents and copy it anywhere
• implement all kinds of custom behaviour such as FTPing notes if they are tagged a certain way, or whatever

---

Reply to this email directly or view it on GitHub: https://github.com/ttscoff/nv/issues/129#issuecomment-43275035

[vcavallo]

Dropbox issues have resolved for me, haven't seen a note disappear for a long time. Using nvALT on two machines and syncing to Notesy on iOS.

I've had notes disappear recently syncing between two macs and one Notesy. I remember reading about this a while back but didn't have to deal with it because I was only on one mac - but now that I'm using two regularly I've begun to notice the issue. Was this actually resolved, or did it seem to sort of ...go away for people?

[TALlama]

I still see it semi regularly.

[vcavallo]

I need to do some better testing, but my suspicion right now is that it's related more to Notesy and less to multiple nvALT instances.

[juranta]

A while ago this summer I installed one iOS app and pointed it to use my Dropbox notes folder, which is the same folder that nvALT uses. I also happened to have my MacBook with nvALT turned on next to me at the same time. After connecting the folder to the iOS app, MacBook started sending notifications about changed files. When I looked at notes on MacBook, they were all zero bytes.

Apparently when my new iOS Dropbox app was somehow just scanning/reading the notes folder through some API, it also changed some property of the notes in a way that caused nvALT to write zero bytes on them.( I got the notes back without any problems from backup). That is, I think the iOS software unknowingly caused Dropbox to sync some changed or maybe removed property back to MacBook's note files, which caused nvALT to destroy the notes.

I was also using Notesy at the time I had issues, by the way. This mass destruction was with another app. Could it be that sometimes these apps do something to those Mac specific resource fork files (of which I know nothing), or something similar to that? What about timestamps?

[pamphleteer-design]

Started getting this issue semi-regularly again. Although now Dropbox seems to prompt me when a note is being deleted so it can be caught in time.