"scan and serialize all dataview queries" takes a long time and seems stuck

[tijmenvdk]

Whenever I kick off this command, Obsidian becomes very sluggish until I close and restart it. Wanting to dig deeper, I ran FileActivityWatch (NirSoft tool on Windows), and I can see that Obsidian keeps writing to .md files. There is no feedback from within Obsidian that it is doing so, and there are no real cpu spikes (although my laptop fan does spin up more), but FileActivityWatch shows a lot of activity, and it appears that Obsidian keeps touching the same files.

I saw that the code has logging in it, but I do not know how to switch it to "debug" level. I am working on a 550 note vault where almost every note has at least 1 and often more dataviews, all now converted to serialized dataviews. There is a fair bit of complex queries, but everything uses standard DataView syntax.

There are no messages in the Developer console (Ctrl+Shift+i), but it looks like somehow the serializing gets stuck. Running on a modern/fast laptop, I have left it running for about 15 minutes now, will check if it stays like this indefinitely, or that it is going to snap out of it at some point.

[tijmenvdk]

OK, after 45 mins it sits at 35% cpu load on a 16 core machine, look forward to hearing how I might debug this. It might well be that one of the queries is causing this. I also have no way to tell if the process is actually finished or not.

[dsebastien]

I will need to try and reproduce this. I probably need to add more debounce time to file updates, and maybe a way to break cycles. If the results of updating a note imply changed to 1n other notes.

I don't have many complex queries in my vault yet. Do you think you could create a sample vault to help troubleshoot this?

[tijmenvdk]

I will try to create something that represents the current structure, without having 500+ notes 😊 in the meantime, is there a simple way to turn on debug level logging? Or do I need to set up an entire dev environment for that? Would be good to know what file the code is working on while updating.

[dsebastien]

No, but I think I'll add a debut setting. The Obsidian team specifically asked me to reduce logging :p

Le jeu. 20 juin 2024 à 21:14, Tijmen van de Kamp @.***> a écrit :

I will try to create something that represents the current structure, without having 500+ notes 😊 in the meantime, is there a simple way to turn on debug level logging? Or do I need to set up an entire dev environment for that? Would be good to know what file the code is working on while updating.

— Reply to this email directly, view it on GitHub https://github.com/dsebastien/obsidian-dataview-serializer/issues/10#issuecomment-2181363278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAV6H6KHKKKS42KWAICVNTZIMS2VAVCNFSM6AAAAABJUN3VJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBRGM3DGMRXHA . You are receiving this because you commented.Message ID: @.***>

[tijmenvdk]

Have made a very small vault, cannot reproduce the long runtime, but what I do see is that "serialize all" only picks up changes if I first turn the plugin off and then back on again. No feedback from the plugin or elsewhere, but I can see that the files do not get updated unless I turn the plugin off and on: not the same issue, but possibly related?

Would it be possible to add a single message indicating that serializing is done? That way I can tell that it is not finishing at least.

[tijmenvdk]

Will also try the "serialize all" directly after opening the "big" vault to see if that makes a difference.

[dsebastien]

Can you upload it here? I'll take a look tomorrow

Le jeu. 20 juin 2024 à 21:51, Tijmen van de Kamp @.***> a écrit :

Will also try the "serialize all" directly after opening the "big" vault to see if that makes a difference.

— Reply to this email directly, view it on GitHub https://github.com/dsebastien/obsidian-dataview-serializer/issues/10#issuecomment-2181417272, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAV6H4WI35VZXWX4IOKAYLZIMXEZAVCNFSM6AAAAABJUN3VJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBRGQYTOMRXGI . You are receiving this because you commented.Message ID: @.***>

[tijmenvdk]

I zipped it. To reproduce: change the "isPartOf" attribute in "offering 2-1.md", this should update the "solution 2.md" file when serializing all, but it does not. Zip also includes a link to a video showing this in action.

Again, not sure if this is related to the plugin getting stuck.

serializer-test-vault.zip

[dsebastien]

I think the problem is due to the fact that the delay between allowed file updates is too short. I've noticed a similar loop in my vault now that I have more queries. The files get processed once, which triggers a series of file modification events. Then, those modification events trigger a re-processing of the files, and so on.

[dsebastien]

I've released version 1.1.5 a few moments ago. Hopefully it should improve the situation. Could you give it a try?

[dsebastien]

I've also skipped making file modifications when the content doesn't change after processing all queries. Previously, the plugin would always write to disk, even if the file content did not change, leading to additional file modification events. This should help a lot

[tijmenvdk]

It works! Really fast on my (by now) 600+ notes vault, it no longer gets stuck, perfect! I do still notice different behavior when running the command multiple times, but that always gets resolved by turning the plugin off and on again. Will test that further and if the issue persists I will submit a new issue, but this one can be closed. Happy user, thanks a lot!