Error: org-node worker failed to scan files

[oatmealm]

I'm seeing lots of messages like "An org-node worker failed to scan files, not producing /tmp/org-node/demands-xxxxx.eld", for example after creating a new node (with vanilla org-roam-find).

[meedstrom]

Any output in the *org-node errors* buffer?

[oatmealm]

I actually forgot to mention. I did look it's empty with only this message: This output printed at 18:16:46

[meedstrom]

Hm, that means that none of the processes returned successfully, but they also didn't signal an error. Odd.

I've been meaning to write a test suite, but I won't have a lot of time this month. :/

Are you only seeing this after you create a new node? Or every time you do M-x org-node-reset?

[oatmealm]

I didn't invoke reset yet. Should I? I'm seeing this message quite a lot.

[meedstrom]

Do you mean it works, i.e. you can find nodes anyway? Hmm that is your risk to take then :)

You can always fall back on org-roam or quickroam commands!

[oatmealm]

It works, sort of...

Seeing a lot of these, and it often hangs while producing the errors:

An org-node worker failed to scan files, not producing /tmp/org-node/demands-2.eld.  See buffer *org-node errors*.
An org-node worker failed to scan files, not producing /tmp/org-node/demands-0.eld.  See buffer *org-node errors*.
An org-node worker failed to scan files, not producing /tmp/org-node/demands-8.eld.  See buffer *org-node errors*.
error in process sentinel: apply: Buffer name ‘*org-node errors*’ is in use
error in process sentinel: Buffer name ‘*org-node errors*’ is in use
An org-node worker failed to scan files, not producing /tmp/org-node/demands-6.eld.  See buffer *org-node errors*.

This output printed at 11:13:05

[meedstrom]

Still happens?

[oatmealm]

Still happens?

Hi there. Disabled for a while since it was very noisy, sorry.

Will check again shortly and report back.

[oatmealm]

yes, still happens and org-node-find cant' see most files in org-node-extra-id-dirs...

[meedstrom]

I'm sorry to hear that! At this point, I think I'll just expand the test suites until something gives.

My best hypothesis is that org-node does not handle malformed Org syntax very well, and you have some (it's quite common, I suspect) malformed syntax in some files such as these examples: https://github.com/meedstrom/org-node/issues/8#issuecomment-2101316447

[oatmealm]

I'm sorry to hear that! At this point, I think I'll just expand the test suites until something gives.

My best hypothesis is that org-node does not handle malformed Org syntax very well, and you have some (it's quite common, I suspect) malformed syntax in some files such as these examples: #8 (comment)

Wow, looking at list of possible problems, I pretty sure there are plenty... :( also possibly left over from partial migration from v1 etc.

Is there a way to force the parser to ignore them and made more robust? It's not org-node's problem to solve this, I'm sure...

I'm sorry to hear that! At this point, I think I'll just expand the test suites until something gives.

My best hypothesis is that org-node does not handle malformed Org syntax very well, and you have some (it's quite common, I suspect) malformed syntax in some files such as these examples: #8 (comment)

[oatmealm]

I think you might need to look elsewhere... I created a new empty roam directory and db, and org-node is still having similar isses:

Directory ‘/home/XXX/Nextcloud/roam2/’ does not exist; create? (y or n) y
Wrote /home/XXX/Nextcloud/roam2/20240602105717-hello_world.org
An org-node worker failed to scan files, not producing /tmp/org-node/demands-0.eld.  See buffer *org-node errors*.
An org-node worker failed to scan files, not producing /tmp/org-node/demands-1.eld.  See buffer *org-node errors*.
An org-node worker failed to scan files, not producing /tmp/org-node/demands-3.eld.  See buffer *org-node errors*.
An org-node worker failed to scan files, not producing /tmp/org-node/demands-2.eld.  See buffer *org-node errors*.
An org-node worker failed to scan files, not producing /tmp/org-node/demands-4.eld.  See buffer *org-node errors*.

[meedstrom]

It's not org-node's problem to solve this, I'm sure...

Yea, the best solution is to lint en masse.

There is a builtin org-lint, but I haven't yet figured out how to run it for many files at once. I'm currently relying on some old linting code I wrote for publishing my blog, which works well enough for me so it hasn't been a priority.

It's true that I could look for ways to make org-node fail more gracefully.

I think you might need to look elsewhere... I created a new empty roam directory and db, and org-node is still having similar isses:

Good test, thanks! But hey, org-node may still be trying to scan the same files as before, since it consults org-id, not directly org-roam-directory. You could verify this by checking if org-id-files contains stuff from your old org-roam-directory.

Maybe you'd like to try again, but fully wiping org-id:

(progn
  (org-id-locations-save)
  (copy-file org-id-locations-file "~/.org-id-locations.bak") ;; backup
  (delete-file org-id-locations-file)
  (setq org-id-locations nil)
  (setq org-id--locations-checksum nil)
  (setq org-agenda-text-search-extra-files nil)
  (setq org-id-files nil)
  (setq org-id-extra-files nil))

Don't forget to change org-node-extra-id-dirs along with org-roam-directory when you test :) And also wipe org-agenda-files since org-id also looks there...

Afterwards, you could restore your org-id database with

(progn
  (delete-file org-id-locations-file) 
  (copy-file "~/.org-id-locations.bak" org-id-locations-file)
  (org-id-locations-load))

[meedstrom]

The latest commit should now fail a bit more gracefully, and show you which files that failed.

[meedstrom]

Now I added a command M-x org-node-lint-all-files!