nvALT lost all my notes

[dfreire]

I usually keep nvALT running for days in a row without closing it.

Today, I quit the app so I could do a computer restart. After the restart, I launched nvALT to find that all my notes had disappeared!

I keep the nvALT folder in dropbox, so I searched through the files history and tried to restore several versions, but no version seems to work.

This was no awkward restart, I quit the application normally without any problem.

[ttscoff]

There's something going on with Dropbox sync. I haven't seen anyone have ALL notes deleted yet, but I've had several disappear while I was editing. Working on it as quickly as I can.

[dfreire]

You're basically telling me that my notes are not recoverable, right?

[juranta]

The version 0 is added 12/31/2012 and the other changes have happened within the last hour, I wonder where the changes from the whole January are. Those "Interim Note-Changes" are 0 bytes and the biggest "Notes & Settings" is 3,74 KB. I don't think your notes are in those files. Is it possible that the actual notes database is now in some other place?

Sometimes when I've moved my notes database to some other folder outside of nvALT from Finder or Terminal and then went to nvALT to configure the new location, it had already found the new location all by itself. Huh? How did it do that?

In any case, this perhaps won't help you at this point, but I think it's a lot better idea to save the database as "Plain text files" in Dropbox instead of "Single Database". With plain text files, all of your notes are very quickly written on the disk. You can restore older versions of single files, compare them, open them with other programs outside of nvALT (even from iPhone with such programs as Notesy), etc, etc. You can actually see your notes on your disk instead of inside some mystical database! Also, Dropbox conflict problems are far less likely. The only major thing is that you can't encrypt the plain text files directly from nvALT. If you really need to encrypt the plain text database in Dropbox, it's maybe reasonable to create an encrypted volume with BoxCryptor, sparse bundle or something and store the notes there?

If you use plain text database, nvALT still creates those two files but they're no longer as important. I still wish that with plain text database, those files would reside inside ~/Library to prevent conflicts. nvALT supports only one database anyway.

[dfreire]

I was using "Single Database" (the default) unfortunately. The notes are not anywhere else, I have searched my entire disk for them. The only "Interim Note-Changes" was the dropbox one. There were two "Notes & Settings", one on dropbox, the other on "~/Library/Application Support/Notational Data", but both are empty.

It is possible that I have not restarted my computer for the entire month of January, so I haven't closed nvALT at any time. I suspect nvALT has not flushed to disk during this period, is that possible? (If not, how come Dropbox doesn't have any versions for the entire month?)

[juranta]

Yes, Dropbox should keep versions for 30 days. I just tested nvALT with a "Single Database" and when I created new notes, it seemed to flush them to "Interim Note-Changes" almost immediately. When I closed nvALT, it moved the stuff to "Notes & Settings", based on file sizes. Unfortunately, I guess it's possible that for some strange reason your notes were all the time only in the memory of nvALT, but this isn't normal behaviour of nvALT.

[anatomatic]

You should be able to recover them in Dropbox's web interface. Navigate to your Notes folder, and on the top there's a greyed-out trashcan. This allows you to restore deleted files up to 30 days old (also allows you to restore older versions, if you make changes you don't want.)

I have a very simple Automator workflow that zips my Notes folder (or it can point to a single data-base) and dates it in an "auto-backup" folder, also in Dropbox, that runs via launchd every couple of days. It's an additional failsafe. Download here. Make sure you open it in automator and point it to the file/folder you want backed up.

Notes have such a small file-size that it's easy to keep a lot of backups. And they're so important that you always want that failsafe, just in case.

[dfreire]

@anatomic, after clicking in the trashcan I get the same file history I showed in my opening comment, i.e. there aren't any versions in January. During this time nvALT probably kept the versions in memory and never flushed to disk.

[elasticthreads]

Sounds like users should be seriously discouraged from using Dropbox sync with notes stored in the database. If dropbox corrupts that one file... that's all the notes!

Please: everyone: this is a beta. zip up your notes database occasionally as a backup.

And if you store your notes in dropbox: don't store them in a single database.

[juranta]

Dfreire sent the whole history of those files and all modifications were from drome-mac, which means he wasn't using the database from any other computer. If there was a conflict, it would show up in history and there would be a conflict file in the same directory. Dropbox in my experience is very robust with conflict handling. And, after all, the Dropbox folder is after all just a plain normal folder, and if there is only one computer using it, then the Dropbox just uploads the changes on the background. So, I don't think this issue is related to those other Dropbox issues.

But yes, I think that it's usually better to use the text file database instead of the block database for many different reasons, whether you use Dropbox or not.

[ArmchairQuarterback]

I have not encountered this myself, but I would strongly recommend (as others have) storing the data as text files (instead of a single database), using Time Machine, and occasionally zipping stuff up (as recommended above). This is generally useful advice for any software, I think, especially stuff that is still in beta.

[ttscoff]

As a side note, storing as text files doesn't replace the database, it's an extra measure of security and searchability for your notes. The database still exists and nvALT runs just as fast.

On Wed, Mar 20, 2013 at 6:59 PM, ArmchairQuarterback < notifications@github.com> wrote:

I have not encountered this myself, but I would strongly recommend (as others have) storing the data as text files (instead of a single database), using Time Machine, and occasionally zipping stuff up (as recommended above). This is generally useful advice for any software, I think, especially stuff that is still in beta.

— Reply to this email directly or view it on GitHubhttps://github.com/ttscoff/nv/issues/135#issuecomment-15211712 .

[ArmchairQuarterback]

Thanks for the clarification Brett. In fact, this is a good thing to know, and I did not realize the data lived in both locations. There are a lot of features in nvALT that I like, but the ability to break out your notes this way and have them as plain text files right there in the folder is one of my favorites.

[ArmchairQuarterback]

@brett Any luck on the fix for lost notes?

[EDIT:] I am still not sure how I have lost note content. It may not be nvALT after all. I am going to keep looking into this, and I will post more if I come across a clear case.

[juranta]

It is not a good idea to even try to sync block database files such as Interim notes and that other file through Dropbox. Syncing is a complex problem and syncing block databases through Dropbox is surely not a good idea. nvALT/nv is very good at recognising changes made by other parties to those text files on the disk, it will pick up the changes almost immediately. Those Interim and Notes files absolutely shouldn't be inside Dropbox.

[elasticthreads]

I'm working on a fix to nvALT to keep this from happening and stabilize dropbox syncing. For now, that revolves around storing "Interim Note-changes" in ~/Library/Caches/ instead of the user's notes folder.

If you've experienced any lost notes, or are interested in testing this, please email me elasticthreads att gmail with the subject line "Test Dropbox sync" and I'll send out a beta in the near future. This is mainly for Dropbox sync so please only respond if you use that. Please include in this email the following information:

• what version of OS X you're running
• what hardware you use (i.e. "2012 Macbook Pro", or even better "Macbook Pro 5,1")
• what format notes database you use (i.e. single notes database or "individual text files with .txt extension")
• if you're using Dropbox + nvALT across multiple macs (and include the above 3 pieces of info for each mac you use)

thanks! David Halter aka elasticthreads

[juranta]

About the 0 byte issue, see this discussion:

http://konklone.com/post/dropbox-bug-can-permanently-lose-your-files

This may be a different issue than individual disappearing notes. I'm not sure.

[ArmchairQuarterback]

Thanks for posting a link to your blog post. I read through it and the comments section. I need to think through this a bit more, but at first glance it looks to me that Dropbox clearly deleted your data, and that is quite disconcerting. Although they were polite and generally helpful in their correspondence, they did not seem especially competent, and I wasn't clear that they accepted it was an issue on their end. If they don't recognize it as their problem, can we expect them to fix it?

I have Time Machine backups, and regularly copy my stuff onto a backup drive manually, so it is pretty rare for me to lose any data permanently. Yet, if I lost 25 random files, for example, I'm not really sure how I would deal with that. First of all, I'd have to know they were gone. Then, I'd have to wander through my backups and try to recover each one. It would be better if Dropbox just wiped out everything at once instead! I'll run the terminal command you suggested, and we'll see what happens.

By the way, I've been working with the new Dropbox beta mentioned a couple posts above by David. I have had a great experience with it, and in theory it seems like a much better way for nvALT to interact with Dropbox.

[juranta]

Thanks for the comment, ArmchairQuarterback. However, the blog isn't mine, I just found it when I was researching this issue again. I haven't got the 0 bytes problem but just a while ago another note disappeared from my nvALT, a version which stores Interim Notes in cache.

[elasticthreads]

yea, we haven't quite fixed the glitch. It's tricky to solve 100% of the time from nvALT. The only sure fire thing to do is not leave nvALT running on multiple macs. When you leave a mac, quit nvALT. Because, as far as I can tell, this only happens when nvALT is also running on another mac than the one you're using.

This isn't just an nvALT bug, though. This never happened until sometime in the last 6 months despite thousands of users (of both nvALT and Notational Velocity, which also has this issue) using multiple macs and syncing thru Dropbox. Dropbox made a change that's part of the issue.

on the other hand, Notational Velocity (and so nvALT), were written before Dropbox, syncing, cloud storage really existed. And the core codebase that manages your notes is not really made to handle some the complexities that come up when you're doing this. We're doing our best to fix this as best and fast as possible.