Throwing 80,000 files (180 GB) into a vault makes dataview do random stuff

laurastephsmith commented 2 years ago

What happened?

Dataview Version Affected

Issue exists in 0.5.16 but does not exist in 0.4.26.

The Problem

When adding about 80,000 files (180 GB) into a vault, Dataview basically stops working. Queries can only return a small number of notes in the vault. Any query that's expected to return any other notes returns nothing.

This is not a stress-test by the way! I've moved my Obsidian vault to be the root of my Dropbox. I want to be able to access any of my Dropbox files from within Obsidian.

To Reproduce

Install Dataview beta with BRAT.
Test Dataview: everything works as normal. A DQL test TABLE returns all notes in vault.[^1]
Close Obsidian.
Copy 80,000 files (180 GB) into vault.
Open Obsidian.
Test Dataview: everything works as normal.
Observe in system monitor that Obsidian is using loads of CPU: I assume it's caching the new data.
Once CPU goes back to idle, Dataview only returns about 28 notes out of about 800 notes in the test query. I may have closed and reopened Obsidian before this bullet point.
Delete some notes. The number of notes returned by the query goes down: it doesn't pick up extra notes in place of the ones deleted.
Add additional blank notes, which should be picked up by the query. They are not. The number returned by the query stays teh same.
Delete beta version of Dataview.
Install Dataview 0.4.26 from normal Community Plugins repository.
Close and reopen Obsidian.
Observe that the test query now returns every note in the vault (about 800 notes).
Check CPU use: it is idling.
Close Obsidian, open Obsidian.
Observe that Dataview is still working properly: all 800 notes are returned by the test query.

System Info

The bug was still present with all plugins disabled except for BRAT and Dataview. But this output here is with all my plugins enabled.

SYSTEM INFO: Obsidian version: v0.14.6 Installer version: v0.14.6 Operating system: #1 SMP Debian 5.15.15-1 (2022-01-18) 5.15.0-3-amd64 Login status: not logged in Insider build toggle: off Live preview: on Legacy editor: off Base theme: light Community theme: Typomagical Snippets enabled: 12 Safe mode: off Plugins installed: 48 Plugins enabled: 48 1: Divide & Conquer 2: Advanced Mobile Toolbar 3: Obsidian Big Calendar 4: Breadcrumbs 5: Calendar 6: Convert url to preview (iframe) 7: Customizable Sidebar 8: Cycle through Panes 9: Editor Commands Remap 10: Format Hotkeys 11: Highlightr 12: Macros 13: MetaEdit 14: Note Refactor 15: Apply Patterns 16: Auto Link Title 17: Collapse All 18: Code Editor Shortcuts 19: Excalidraw 20: Excel to Markdown Table 21: Filename Heading Sync 22: Footnote Shortcut 23: Frontmatter Tag Suggest 24: Global Hotkeys 25: Hider 26: Hotkeys for specific files 27: Kanban 28: Link Archive 29: Linter 30: Local File Interface 31: Pandoc Plugin 32: Pocket 33: Shell commands 34: Smart Typography 35: Smarter Markdown Hotkeys 36: Sort & Permute lines 37: Style Settings 38: Task Collector (TC) 39: Tasks 40: Text Format 41: Tidy Footnotes 42: Version History Diff 43: Zotero Desktop Connector 44: Obsidian42 - BRAT 45: Omnisearch 46: Snippet Commands 47: Tag Wrangler 48: Dataview

[^1]: Test query in its entirety is TABLE o, where o is a field I'm using in many notes.

DQL

No response

JS

No response

Dataview Version

0.5.16

Obsidian Version

0.14.6

OS

Linux

laurastephsmith commented 2 years ago

I've tested in a clean vault, with 100 blank notes. Adding the 80,000 files does not cause the problem to appear. But adding copies of my 800 notes from my production vault causes the problem to appear.

So I suppose either:

The quantity of notes is causing an issue; or
~~One or more of my notes is causing the thing to foul up (but only when the 80,000 other files are present).~~

laurastephsmith commented 2 years ago

Problem occurs in test vault with 800 blank notes (when the 80,000 other files are present).

blacksmithgu commented 2 years ago

The additional 180Gb of files you are adding are not markdown files, right? I wouldn't expect Dataview to crash and burn on this since it ignores non-markdown files. Do you get any errors in the developer console?

laurastephsmith commented 2 years ago

Correct - photos, videos... 30 years of junk! No errors in the console. And to reiterate, the problem doesn't exist in 0.4.26.

I've decided against putting all my files inside my vault directory now anyway, so this bug won't affect me. The quick switcher just... wasn't very quick any more :laughing: - but I'm very happy to test for you if you need me to!

blacksmithgu commented 2 years ago

I made a few small fixes to improve this a little more, though I'm still not sure what changed between 0.4.x and 0.5.x to break this; I'll need to add some more metrics tracking / logging to narrow this down.

laurastephsmith commented 2 years ago

Okay - if you want me to test it when you've done that, let me know.

blacksmithgu / obsidian-dataview