logseq is deleting the current journal file completely (dropbox sync related I believe)

[rluiten]

What happened?

I am seeing the current journal entry content get completely removed several times in a row.

By digging I can find the journal file in the folder under current database 'logseq/.recycle'.

But its gone I can restore it manually and I have done that today 3 times so far. But next time I come back to logseq and add something it is deleting the journal entry again.

I've seen this delete today's journal probably a dozen times in the last 2 weeks but didn't have enough context of what was happening and I couldn't describe the conditions well. This happened 3 times in row over an hour so today. I restored the content from .recycle by hand so far when I do the correct content is displayed in logseq again. So far today I navigate away and come back after some time it has deleted it again.

The folder that has my logseq database is in a folder that is synchronised to dropbox.

I have just discovered that if drobox client is not running the journal file is not deleted. So this is an interaction with dropbox client I believe.

Reproduce the Bug

1. Add a note in todays journal
2. Navigate to some other applications and such
3. Go back to logseq add another note, when I go straight back I haven't seen it delete.
   • If there is some time involved when I get to working on code or reading something then come back.
   • I usually am pasting something if it makes a difference.
   • Usually i come back after the edit and today the journal entry is gone
4. Watch logseq delete todays journal again

Expected Behavior

Don't delete today's journal.

Screenshots

I don't believe a screenshot is useful for this.

Desktop Platform Information

Windows 10 21H2 - fully patched Logseq 0.7.5 (have seen this delete behavior version since at least 0.7.0 possibly even earlier)

Mobile Platform Information

No response

Additional Context

No response

[cnrpman]

Likely not Logseq who delete your journals. If Logseq can't find the file in your file system, Logseq reads the data in its database and dump it to the recycle

The recycle won't be triggered if Logseq do the deletion on its own.

We are investigating the reports. It's about sync service and we are trying to reproduce the issue.

Can you report the file history of Dropbox on the directory?

[cnrpman]

Ref for tracking https://github.com/logseq/logseq/issues/5901

[rluiten]

@cnrpman given your information and some testing it does look like dropbox is deleting the file, but I don't understand why this started happening. I quit dropbox client and had no problems for at least 1/2 an hour. I started dropbox client again and I lost todays journal file a few minutes later after I modified. I don't believe I have seen this type of behavior before from dropbox client. I do have another dropbox client sync to same folder, that is how it was setup many months ago. It is entirely inactive when I am at home.

[cnrpman]

@rluiten Can you try open dropbox client when Logseq is closed and modify the files in text editor? Also we are interested in the file history of Dropbox

[IanGor]

Hi, just to be clear — pretty the same with Syncthing and Resilio Sync as well. I though it was bug on Android version (there I found that all data is deleted). But as I can see now — it's a common problem.

[cnrpman]

@IanGor There's another issue about the file watcher on Android..? https://github.com/logseq/logseq/issues/5841#issuecomment-1173267688 They have not the same behaviour.

[rluiten]

@cnrpman

1. exit dropbox client
2. start logseq
3. created a new journal entry for today put 2 small notes
4. quit logseq
5. start dropbox client, see dropbox uploads the file
6. wait a few minutes nothing happens
7. edit the file with notepad and save, see dropbox uploads the file
8. wait a few minutes nothing happens
9. edit the file with notepad and save, see dropbox uploads the file
10. wait a few minutes nothing happens

It does not appear that editing files with notepad with dropbox client running causes a problem.

After that I started logseq to see what it did.

1. it showed the latest modified content fine
2. waited a few minutes
3. edited the last note in the journal file added a few characters
4. waited a few minutes all ok
5. added a new note at end of notes and a blank new note then hit enter on the first new note
6. waited minutes switched to this editor for notes and saw file deleted
7. in dropbox web client I can see file as deleted now when I saw it deleted from file system
   • I had the journal folder in a explorer window and saw the file vanish
   • I cant tell tell what did the action

A copy of the currently deleted journal file from dropbox web file history.

Possibly related question what actions does logseq take when a journal is edited. What sequence steps? Options I can think of are

• it overwrites current journal file with new content
• it renames original file then goes and saves a new file with original name then delete renamed old file
• it writes current journal file to a new file name then remove original file and rename back to current file name There are probable other variants possible.

[cnrpman]

@rluiten Thanks for narrowing down the issue. We are investigating.

[cnrpman]

Still, I can't observed the deletion..

Do you have any plugin installed?

[rluiten]

I'd completely forgot I had any plugins. Installed months ago not really using. I have "Copy Code" v1.0.1 I have "Markdown Table Editor" v1.7.0 I also have theme "Dracula" installed.

I will disable 3 of them and see if I see any different behavior.

[cnrpman]

@rluiten Thank you. Seems the plugins are not related to the issue.

[rluiten]

Steps.

1. quit dropbox
2. start logseq create journal
3. quit logseq
4. start dropbox
5. start logseq
6. to add a new note and as i typed the cursor moved back then note deleted as i typed

This is getting weirder don't think I've seen it delete while I type before.

[cnrpman]

@rluiten It is just acting like file got overwritten.

[rluiten]

Generally there is no file on disk in the "journals" folder when it deletes. It shows me an empty journal entry on the day. No file is created till type something in again.

[rluiten]

note: given the evidence in dropbox maybe its being renamed?

[cnrpman]

@rluiten There's no such "move to recycle" API called by Logseq. https://github.com/logseq/logseq/blob/83569141fc1de294e9790067f54c81b5fb8678d1/src/main/frontend/fs/nfs.cljs#L113-L116

Have you noticed anything in the console when deletion happens? To open the console, Press alt, then click view - Toggle Developer Tool and switch to the Console tab

[rluiten]

Loaded logseq added a note. Got this almost immediately when stopped typing and switched back to another app

[rluiten]

Is there a keyboard shortcut to delete page? Not that I think I'm typing it but asking anyway.

[cnrpman]

@rluiten So it's the file watcher captures the file is deleted: https://github.com/logseq/logseq/blob/658d1d315a04ecff499b7458e479f94baee8b18b/src/main/frontend/fs/watcher_handler.cljs#L87-L93

[rluiten]

Do you know what Process Monitor is? https://docs.microsoft.com/en-us/sysinternals/downloads/procmon

I captured it for the specific file I'm seeing removed and tried to make it delete it took 5+ times of edit and go somewhere else this time but I captured a log that covers the deletion

At 9:03:59:0806077 I first see Dropbox reading the journal copy in the .recycle folder so I assume its just before then "journals_2-22_07_04.md". At 9:04:05.2076029 I see the first "NAME NOT FOUND" for the journal file. "2022_07_04.md"

I've attached a screen shot for now I can provide a 1.3MB compressed to zip .PML file if its useful. It unpacks to 15MB or so. I didn't attach in case large file attachments might be a problem I can put it somewhere else if it's useful.

[cnrpman]

@rluiten May send the attachment to junyi@logseq. Many thanks for your help!

[cnrpman]

@rluiten Good morning! We have released 0.7.6 , which contains a fix toward the unlink behavior of 3rd party sync service on Windows :)

[rluiten]

@cnrpman New version 0.7.6 has deleted new today's journal when I repeated the edit wait, edit wait twice already. So this is 0.7.6 doesn't make much difference to me that I can see so far.

It still logs when it happens.

Delete page: Jul 5th, 2022 , file path: C:/Users/rluiten/Dropbox/Logseq/DB1/journals/2022_07_05.md .

[cnrpman]

@rluiten The log is for receiving "unlink" event. In 0.7.6 it won't delete file on disk when the log shows up: https://github.com/logseq/logseq/commit/cc643f84875588852adac53576935e9dd77ec3c4

Sorry that the release doesn't help. We are doing more investigation.

[tiensonqin]

@rluiten Can you record a video? In 0.7.6 a page will be deleted from Logseq if the file watcher detects that the corresponding file has been deleted on the disk. But Logseq will not delete the file.

[cnrpman]

@rluiten Just made a video for simulating the delete action triggered by the "unlink" event:

https://www.loom.com/share/81a8c91932ae43848902e1bc956d1cda If no following-up add-file action is fired by the 3rd party sync service, Logseq's db of the page will keep empty. Finally a conflict modal will show up when you walk into that page, as the data in db (empty page) is mismatched with the file on disk (page with content)

Besides, @andelf is testing Dropbox on multiple Windows machines. We are validating the fix in real environment.

[rluiten]

@cnrpman I have just spent over an hour trying to capture a video of losing a journal file and can't get it to happen at the moment. I've been able lose a journal file in general in less than 15 minutes in the last week or so.

Yesterday when I reported that a journal file was lost I had just updated to 0.7.6 and I had checked settings shows 0.7.6 before trying however it's starting to look like maybe I was running old code somehow?

At the moment I see no problems. I shall keep an eye on things and add more information if/when I can get some.

[cnrpman]

@rluiten Maybe there's some latency for code to apply .. Sounds good that no more happening after. The formal 0.7.6 beta release is released few hours after your comment. There was a 0.7.6 nightly release before then. Thank you again for your great help!

[rluiten]

@cnrpman I have not updated a second time so still on the Logseq that identified as 0.7.6. I just downloaded the 0.7.6 again and its byte for byte the same as the previous 0.7.6 install I downloaded.

However that isn't important, I've been in and out of Logseq a dozen plus times both work and home and have not had any journal days vanish.

So I think you have fixed whatever caused my problem at the very least. Do you wish me to close this issue? Thank you!