Bug report - Dataview not returning all results

just-a-goat commented 2 years ago

What happened?

Since updating to 0.5.26, dataview no longer fully indexes my vault. I have the same vault working correctly on 0.4.26. A simple table that should select all, will only return around 70 out of 1100 notes. All query blocks in the vault are affected. I can't see the difference between a note that will showup, and a note that wont.

When using the inspector, two errors appear (on 0.4.26 there are none). dataview-error

My use-case tends to be using one or more multi-level tags in inline fields.

Please let me know if there is anything else I can provide to help troubleshoot.

DQL

Table file.size

JS

No response

Dataview Version

0.5.26

Obsidian Version

0.14.6

OS

Windows

blacksmithgu commented 2 years ago

Out of the large number of files that are not indexed, can you pick one at random for me to try locally? There seems to be a parser failure. I have a few small fixes I've just added to improve this situation which will come in 0.5.27, though I have no idea if they will actually fix your problem.

just-a-goat commented 2 years ago

Sure. I've included a couple below for comparison. Since re-opening the vault today, I've gained another ~30 indexed files.

Is there a way to get a read-out of which file is being indexed at the time that the error occurs? The console only displays the error when the plugin switched to enabled, or on opening the vault with it already enabled. Dropping the cache or forcing a refresh doesn't prompt the error.

Aarakorca isn't indexed, but Bandit is, then Bandit Captain isn't. This is from \VaultDir\Campaigns\POTA+ - Princes of the Apocalpyse\Creatures\Humanoid\ Not Indexed: Aarakocra.md Indexed: Bandit.md

On the left is the directory / folder structure, and the right is the indexed files from that folder.

DQL: Table file.path SORT file.path, file.name

mitchmeister commented 2 years ago

Is there anyway of going back to 0.4.26? I am also missing significant number of files in dataview queries since upgrading just now to 0.5.26

just-a-goat commented 2 years ago

I'm not sure if this will help debugging this issue, but I spent my lunch break testing things out.

Mixed Results:

I set up a new folder, copied over the .obsidian dir from the known-issue folder.
Open the new folder as a vault and disabled safemode.
Copied the files & folders over slowly. I hit the error on a folder "ROM" that contained ~170 notes.
After the error, DV was unresponsive. I deleted the "ROM" folder from the new folder, closed obsidian and reopened it. There was no error.
I copied the files from the "ROM" folder over even more slowly 20-30 files at a time, and watched a simple DQL to see them all appearing. No errors.
I moved on and copied the rest of the files in the same manner. No errors.
I then deleted the files from the new folder(this prompted DV to crash, though it's a different error (trying to read tags from a file that no longer exists.. makes sense).

No Error, and works correctly:

With a clean cache, obisidian open
Copying all files/folders over to the new folder.
Re-opening Obisidan after above step.
Closing obsidian, deleting all files
Re-opening Obsidian.

Repeatable Error:

With a clean cache, obisidian closed.
Copy the files and folders into the new folder.
Open Obsidian. Parse Error and DV crashes!

Repeatable Error: edit

With a clean cache, obisidian open
Moving all files/folders to the new folder.
Parse error and DV crashes!

Conclusion: ... Well, I'm not sure. It seems to be related to the way that it builds the cache on first-run? That doesn't explain why my first attempt got mixed results though.

just-a-goat commented 2 years ago

That looks like it's worked - I've compiled the latest commit and the vault console has no errors, and all results are being displayed. Cheers! :D

just-a-goat commented 2 years ago

Is there anyway of going back to 0.4.26? I am also missing significant number of files in dataview queries since upgrading just now to 0.5.26

I don't think there is a simple way, if you don't have a backup of an older version. The less simple way is to pull the older version from github and re-compile it. I have only enough knowledge to scrape by on that front :/

blacksmithgu commented 2 years ago

Great to hear - thanks for trying the debugging, I added a few more fixes based on potential issues with the first-time cache loading and properly handling some undefined edge cases and I think it should be much more robust now.

blacksmithgu / obsidian-dataview