Wrong configuration caused multiple notes in LogSeq not written to disk

Coda-Coda commented 2 years ago

What happened?

Hi, first of all, I love LogSeq very much and appreciate greatly the work being done on it!!

This issue is highly important to me as it might cause me (or others) to lose note content.

I have noticed that with multiple notes (stored locally) their contents have not always been written to disk. They persist when LogSeq is closed and re-opened but only a few lines of the note (the first few lines I would have written) appear on disk when I open it in VSCode (or inspect the files uploaded via git to GitHub - which is how I noticed this - when I was trying to look at a notes' contents on github.com on my phone). I do have git auto commit enabled and push regularly and I had thought it was working fine, giving me a good backup of my notes but this is not working as planned.

I am not sure how to reproduce the main bug, but one other thing I have noticed that may be related is this warning message that pops up when I make a new page (within LogSeq):

To reproduce the above, possibly related issue:

Go to LogSeq
Create a new page and type a few lines
Wait a moment
The error should pop up

I have wondered whether this occurs due to the notes being stored in a DropBox folder, but I am not sure. I thought it was relatively harmless as once dismissed LogSeq stops complaining. But now I wonder if it's related to the main issue of the notes not being written to disk as what is written to disk successfully seems like it could be the same as what was already written at the time of the error message popping up.

I am now concerned that I am going to lose a range of content I have written as I don't know all the files which are affected by the main issue. I am also wary of taking certain actions to debug things in case I lose my notes that are not saved to the filesystem in the process, as I'm sure you can understand.

Please help! :slightly_smiling_face: Please let me know you have any insights into how to best fix this for my situation. Thanks very much.

I am running LogSeq 0.5.1 on NixOS (a Linux distro).

Reproduce the Bug

I have included how to reproduce (as best as I can) in context in the previous answer.

Expected Behavior

When the LogSeq editor shows that content is there, it should immediately (or very soon) be written to disk, or at the very least on the close of LogSeq it should be written to disk.

The git auto-commit should be committing content that I write in notes at the specified interval and not missing large portions of content.

When I write a new note, I should not get the error pop-up that I mentioned.

Screenshots

The one relevant screenshot is

Desktop Platform Information

OS: NixOS
Browser: (I am using the LogSeq Desktop App)
Version: 0.5.1

Mobile Platform Information

No response

Additional Context

I use LogSeq to take notes related to my PhD study (and for aspects of my personal life), I really would not like to lose content...

Moilleadoir commented 2 years ago

I just struck something similar on restarting without exiting logseq. Items added earlier to a file & embedded in today’s journal disappeared, but a later added TODO in the journal was saved. Data in a local (Nextcloud) folder. I just assumed logseq regularly saves content, but this doesn’t seem to be true.

Edit: when I look at the actual files, the content is there though, so fixed by a re-index.

Coda-Coda commented 2 years ago

I did just try "Export graph" > "Export as JSON" and the resulting json file does appear to have my missing notes (though I am unable to check exhaustively for sure, but it seems to have them). This is helpful, but I am still unsure how best to proceed from here.

I could do a re-index, but I am still wary about if this safe to do in my situation.

Moilleadoir commented 2 years ago

You should check & backup the actual files, then try a re-index.

Coda-Coda commented 2 years ago

Perhaps, do you know what re-index does exactly? I am concerned that it reloads all notes in to LogSeq based on what they are in the filesystem. I don't know exactly which of my notes are missing parts, in the file-system.

manhtai commented 2 years ago

Logseq desktop app on my MacBook sometimes doesn't flush content into local files, restarting the app doesn't help. It still shows the latest content but the local files contain no content (I open local files with a different text editor).

I have to re-index and then paste the content again.

More info:

I use iCloud drive to sync up notes
Logseq version 0.5.9
macOS Monterey 12.2 on MacBook 14" M1 pro

pmz commented 2 years ago

I've actually experienced this problem for the first time this week (on iOS, with iCloud). Very scary as I actually lost some data.

Previously I was using Logseq on macOS only, with Dropbox. Never encountered the problem.

Coda-Coda commented 2 years ago

In order to guarantee that I don't lose any notes due to this issue, I have tried exporting. I have looked at the import options and I see "Import JSON from Roam Research" and "Import OPML" neither of which I've tried before. I am concerned that even switching to a different graph and back might cause LogSeq to reload notes from the filesystem (and so lose data). I have successfully backed up to JSON, EDN, OPML, (and standard Markdown, though I do care about block properties). The "Export as Roam JSON" button did not work for me.

My current plan is:

Create a new Graph
Use import from OPML to try recover my entire graph
Make sure this new copy is saved locally to my filesystem, add to git.
Now (hopefully) with the assurance that my data is backed up nicely as markdown files (as I had hoped was always the case...)
Try a re-index
Try updating LogSeq
Keep trying things (like moving the Graph out of DropBox etc.) until the original issue appears to be resolved.

My main concern is that by switching graphs in LogSeq I run the risk of LogSeq losing all the notes it has that are not yet saved to the filesystem - I do have these as .json .end and a folder of .ompl - so I am not overly concerned, but it would be a pain if I couldn't import these back into LogSeq properly.

If anyone with more LogSeq experience than me (especially with round-tripping an import and export of JSON or OPML) has any thoughts or suggestions please do share :)

Lastly, I did know all along that using LogSeq could come with risks like this, and if something goes wrong with my notes, it's ok :) I'd still happily like to hear any well-intentioned advice even if it turns out not to help (or goes wrong).

Thanks all who have interacted with this issue so far :)

sandrofsousa commented 2 years ago

I'm having the same issues with google drive on two macs. Indeed it appears to be a file writing issue, it can be replicated by:

Open logseq in source computer while it is closed on the other.
Make changes in the journal on the source computer.
Refresh the app, google drive will pick stuff and update local files.
Open logseq on the target computer, changes are not there, there's not even a file for the day
Check google drive sync log, the file for the day was synced, but somehow logseq got rid of it
Changing the file again on the source computer synced and logseq picked it, but this completely defeat the aim of seamless writing and focus.

I really love this app and I'm totally game on the journalling strategy, but it is unusable at its current state. I keep missing data, which makes me constantly worry about my notes instead of focusing on writting them. I really hope the devs can put some priority on this sync issue, which is very critical and I can't think about anything else with higher priority. I'd be happy to subscribe and support the devs when the product becomes more stable.

gorgabal commented 2 years ago

Same issue on MacOS monterey (12.2.1) and Logseq (0.6.3). I do not use any sync software that I know off. Find the content in logseq to be different from the one on-disk.

I really love the UI, so if I can be of any help in fixing this please let me know.

yang-gu commented 2 years ago

I love the app/UI too. I would love for it to just continuously save like with Android Studio.

Coda-Coda commented 2 years ago

This is not easy to fix. My troubleshooting has led me to the following conclusions.

Re-index is almost certainly unsafe (I can't find a safe way to even test it out).
Updating the app to the latest version doesn't fix this issue for existing discrepancies between LogSeq and the file system. It remains to be seen if it fixes it going forward.
Moving the Zettelkasten out of Dropbox doesn't seem to help in the immediate term.
"Refresh" seems safe, but doesn't fix the situation.
You can't round trip an export+import of OPML or JSON without losing some (or a lot) of info.
One thing you can do is Export public pages this, with the all pages public setting, gives you at least a usable snapshot of your current files.
Editing a file in LogSeq that has a discrepancy between it and the filesystem, does cause the file system to update (check via external software). This seemingly must be done one at a time.
There appears to be no other way to save the content onto the file-system from the state my LogSeq graph is in.

This leaves me with the task of manually opening each of the files in my graph (over 600) and making a brief edit in LogSeq then deleting the edit, if the contents of the file appear to differ from the file system. This is not impossible to do but is quite a pain.

I'll try streamline the process a little by doing a markdown export (no block properties) and then using git to help make it clear when files differ between LogSeq and the file-system. But after looking briefly this doesn't look like it'll help that much as there are plenty of differences between the file-system and a file from a markdown export (no block properties) even when they actually is no discrepancy.

My notes are valuable enough for me to go through this tedious process, but it would be much nicer if some update was pushed to LogSeq which was essentially Save all and would write the LogSeq version to disk for certain, or some other solution.

But aside from that, I would love to know if someone else finds an easier way...

Coda-Coda commented 2 years ago

Also if it is possible to do something via the Javascript console available due to LogSeq being an electron app, I'd be willing to try suggestions :smiley:

sawhney17 commented 2 years ago

@Coda-Coda If you'd like to batch edit every single file, you can run these two commands

First run this

let array = logseq.api.datascriptquery(`[:find (pull ?b [*]) :where [?b :block/uuid ]]`)

Then run this

for (const x in array){
console.log(array[x][0].uuid.$uuid$)
if (array[x][0].content != undefined){
logseq.api.update_block(array[x][0].uuid.$uuid$, array[x][0].content+" wxyish")
logseq.api.update_block(array[x][0].uuid.$uuid$, array[x][0].content)
}
}

sawhney17 commented 2 years ago

I chose to use wxyish because it's a random formation of letters adn in case something goes wrong, you can use the find and replace plugin to remove those items easily.

sawhney17 commented 2 years ago

Upon quitting, this is what shows up.

Coda-Coda commented 2 years ago

So, following @sawhney17's suggestion, I have managed to restore to disk as much content as was still in LogSeq without a tedious process - thanks!! :smiley:. A number of my files where I could see notes on LogSeq but not reflected on disk are now safely on disk (and in version control). Unfortunately, I see evidence in some of my files that I have lost data in the past due probably to this issue :cry:. For example, one note titled "gparted" simply says "To run in NixOS with Waylan" (I'm sure I didn't stop there when writing). This truncation makes me wonder if the issue is associated with the persistent "Page already exists" error that I mentioned originally (though if so, it would be intermittent or only affecting past versions, based on my troubleshooting). The reason I wonder this is because the amount time it takes to type "To run in NixOS with Waylan" seems similar to the length of time it takes for the "Page already exists" error to pop up for me, suggesting to me that perhaps LogSeq was saving my note onto disk until the error popped up, and then it didn't save it do disk after that for that session - and then perhaps between then and yesterday something happened (e.g. a Re-index) which caused it to lose that data forever. However everything I had in still in LogSeq when I ran @sawhney17's suggestion has now been written to disk.

Lessons learnt:

At least while LogSeq is in beta, do not trust it to write your content to disk.
Use git but turn off git auto-commit and very regularly manually review what is being committed to ensure everything you expect is on disk. (Edit: Even better, keep git-auto-commit on and use tags to help with regularly manually reviewing what's on disk, see https://github.com/logseq/logseq/issues/4125#issuecomment-1095656664).
To avoid the "Page already exists" pop-up use the work-around I suggest below. This might reduce the chances of LogSeq not writing notes to disk.

"Page already exists" pop-up workaround To be safer from one way of losing data if the connection I mentioned above is real, the following may work: Idea: Don't use LogSeq to create new files, use a script instead.

Create a script that makes a new markdown file in your LogSeq directory with the name from clipboard such as:

NAME=`xclip -o`
FILE="/home/user/Documents/Zettelkasten/$NAME.md"
if [ ! -f "$FILE" ]; then
echo "- " > "$FILE"
fi

In your OS, make it run via some very convenient keyboard shortcut.
Whenever you'd click on a new link to create a page in LogSeq, or type it in and create a new page, instead...
Type it in, copy it to clipboard, press the keyboard shortcut you made, then navigate to the new page.
After a very brief moment where anything you type will be lost, the new page will be open and the error won't appear.

(This isn't always possible, e.g. when annotating a pdf, pages get created automatically. Hopefully this workaround is actually irrelevant to the file loss issue and another explanation such as Dropbox interference was the problem).

This works for me on Linux (NixOS). Together with the lessons learnt, I'm hoping I can feel able to trust LogSeq not to lose my notes again. I look forward to a non-beta version where these issues are robustly resolved.

An additional note: I continue to be grateful for the awesome community of fellow LogSeq users and especially the contributors and developers who are great!

sawhney17 commented 2 years ago

That's good to hear. I think it may be possible to recover some of this data. Try going to the logseq/bak backup folder

sawhney17 commented 2 years ago

I'm personally a huge fan of the git auto commit feature and that's a great way to get version history. Are you sure that's the problem?

Coda-Coda commented 2 years ago

@sawhney17 thanks for your ongoing ideas :smiley:

That's good to hear. I think it may be possible to recover some of this data. Try going to the logseq/bak backup folder

I have had a look through every file in my logseq/bak folder and unfortunately it holds no content I don't already have.

I'm personally a huge fan of the git auto commit feature and that's a great way to get version history. Are you sure that's the problem?

I didn't think git auto commit was causing the issue, my intention was to have an easy way to keep an eye on what actually is getting written to disk, and turning off git auto commit and inspecting the uncommitted changes regularly seemed like a way to do that. However, your comment has made me realise this approach is not ideal and so instead what I've started doing is turning git auto commit back on, and using git tags e.g. git tag -f last-check to make comparing with when I last checked do-able (e.g. via GitLens: Compare HEAD with...).

Since I've been doing this, LogSeq doesn't seem to be losing any data. For now at least, I am only creating new pages in the roundabout way I mentioned.

Thanks for your help. I feel like for practical purposes I've solved this problem for myself. However the underlying issue is still unclear and probably not resolved, so I'll leave the issue open.

zionchan00 commented 2 years ago

let array = logseq.api.datascriptquery(`[:find (pull ?b [*]) :where [?b :block/uuid ]]`)

I'm an absolute noob but thanks for writing this, where do I put in these commands?

zionchan00 commented 2 years ago

@tiensonqin @xyhp915 @weihua-lu

sawhney17 commented 2 years ago

Type command option I or ctrl shift I on windows to bring up the dev console. Then click console at the top of the screen and then at the bottom you should see a textbox tl enter commands. On 19 Apr 2022, 3:32 AM +0400, zionchan00 @.***>, wrote:

let array = logseq.api.datascriptquery([:find (pull ?b [*]) :where [?b :block/uuid ]]) I'm an absolute noob but thanks for writing this, where do I put in these commands? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

Coda-Coda commented 1 year ago

Hi, I think I figured out part of what I thought was causing this issue. I keep my notes in the root directory of the graph folder (due to my usage before I used LogSeq) and so I had the following line in my config.edn. :pages-directory "." I think this is what caused the error I reported above that said:

I've now changed that line in my config.edn to: :pages-directory "" And that specific error in the image no longer seems to occur, and things seem to be working fine.

Just thought I'd point this out in case it helps :)

cnrpman commented 1 year ago

@Coda-Coda Sounds to be the crux of the issue. Thanks for your effort on debugging it!

logseq / logseq