Notes hierarchy directory tree does not always scan completely

[CarlMangno]

Description

Intermittent fault that includes or excludes certain directories in a notes hierarchical tree.

With a somewhat complicated hierarchical directory tree of notes, FSNotes does not always include all directories in the sidebar. If a directory is not visible in the sidebar, then it is not possible to find the note in a search of all notes using FSNotes.

If I locate one of these "excluded" notes manually in the filesystem and open it with FSNotes, then FSNotes will created a copy of the note at the root level directory, rather than editing the original note passed to it on the command line.

To Reproduce

1. Fault is intermittent so I have no guaranteed way to reproduce it, though with my full notes tree the fault always occurs at parts of the tree that change each time.
2. To reproduce the fault a somewhat complicated directory tree of notes is required.
3. Completely uninstalled FSNotes and all associated configuration files.
4. Reinstalled FSNotes. The tree is never completely scanned correctly. On different occasions, different directories are included and excluded. Once a directory is included, it remains included, and similarly once a directory is excluded it remains excluded.
5. Renaming an excluded part of the tree can on some occasions cause the the directory to appear in the sidebar, (but other directories may disappear on these occasions - I think).

Expected behavior

The full notes tree to be scanned and all notes directories appear in the hierarchical sidebar.

FSNotes version

6.0.0 (542)

macOS/iOS version

Monterey 12.6.1

Additional context

This problem has existed with all previous versions of FSNotes as well.

[glushchenko]

I don't think everyone understands the meaning of the Default storage. You should not connect the whole hard disk to FSNotes, it is a very expensive operation, each run to scan large amounts of data.

Plug in your notes folder and you should be fine. I do not recommend the use of large and complex hierarchies.

[CarlMangno]

Hi glushchenko. Thank you for responding.

My notes repository is about 48 MB in size and has about 1,800 notes. It probably is at the larger end of the scale, but for searching purposes it is convenient to have all the notes in one repository. I am using the default iCloud storage location.

Regards, Carl

[anthonyesau]

Another data point for this issue:

This issue was affecting an external folder that I added to the sidebar as well. A number of directories within the main external folder are not shown in the sidebar and the notes within them cannot be accessed from FSNotes. After renaming a missing directory from Finder, it shows up in FSNotes, but after closing and opening FSNotes it disappears again. Some of the directories are missing some times after opening FSNotes but other times they show up.

Examining the files, the whole folder contained over 400 megabytes and about 600 files. This seemed excessive for primarily text notes with a few images so I tracked down some non-note types of files and folders and moved them out. This reduced the overall size to 30 megabytes and about 560 files. And now all the directories show up consistently, albeit with limited testing so far. So this seems to have fixed the issue for me.

[xor-gate]

I have almost the same issue with fsnotes Version 6.3.3 (578). I tried to remove the external storage from the sidebar. Re-add it and the folder structure is not complete. I only have like 10 subfolders and 9 markdown files in the toplevel external storage. It did work before. Restart of fsnotes did work once a time ago, but now it is completely broken for some unexplainable reason.

No i'm not adding my whole disk for scanning, I know this won't work properly.

[glushchenko]

Yes, it is limited. You can not attach hard disk with unlimited files quantity, because all files loading on start.

[xor-gate]

Yes, it is limited. You can not attach hard disk with unlimited files quantity, because all files loading on start.

But since when is the sidebar limited to displaying only 4 subfolders out of 10? I'm not adding my whole harddisk to the external storage. I understand this implication...

[xor-gate]

I just found out when copy the small folder to Default path. At "Documents". It displays all subfolders as expected:

So basicly we are limiting the scan depth of the external storage?

[glushchenko]

Yes, it is limited. You can not attach hard disk with unlimited files quantity, because all files loading on start.

But since when is the sidebar limited to displaying only 4 subfolders out of 10? I'm not adding my whole harddisk to the external storage. I understand this implication...

There not 4 directories, i see minimum 3 contains sub directories too. You should reconsider how you use fsnotes, this is not the place for an infinite number of files.

[CarlMangno]

Dear Gulshchenko,

Infinitie and whole hard disk is a vast overstatement. The problem can be triggered with a small collection of notes in a tree structure. It does seem to depend on the depth of the tree structure. The problem is intermittent and so hard to reproduce, but, I believe, that if a tree structure is permitted, the depth of the tree should not be limited to only one or two levels.

Regards, Carl

On Jul 17, 2023, at 18:27, glushchenko @.***> wrote:

Yes, it is limited. You can not attach hard disk with unlimited files quantity, because all files loading on start.

But since when is the sidebar limited to displaying only 4 subfolders out of 10? I'm not adding my whole harddisk to the external storage. I understand this implication...

https://user-images.githubusercontent.com/1050166/253562258-56d54094-ba48-44f0-b986-de6c911aacb1.png There not 4 directories, i see minimum 3 contains sub directories too. You should reconsider how you use fsnotes, this is not the place for an infinite number of files.

— Reply to this email directly, view it on GitHub https://github.com/glushchenko/fsnotes/issues/1418#issuecomment-1638910029, or unsubscribe https://github.com/notifications/unsubscribe-auth/AURHNZ3MU57PHHZ3KUXTPADXQWU4HANCNFSM6AAAAAARUUC3JI. You are receiving this because you authored the thread.

[glushchenko]

Dear Gulshchenko, Infinitie and whole hard disk is a vast overstatement. The problem can be triggered with a small collection of notes in a tree structure. It does seem to depend on the depth of the tree structure. The problem is intermittent and so hard to reproduce, but, I believe, that if a tree structure is permitted, the depth of the tree should not be limited to only one or two levels. Regards, Carl …

There may not be a lot of notes, but there are a lot of other files, that's the problem. Long scan = long run, so limit.

[glushchenko]

Dear Gulshchenko, Infinitie and whole hard disk is a vast overstatement. The problem can be triggered with a small collection of notes in a tree structure. It does seem to depend on the depth of the tree structure. The problem is intermittent and so hard to reproduce, but, I believe, that if a tree structure is permitted, the depth of the tree should not be limited to only one or two levels. Regards, Carl …

There may not be a lot of notes, but there are a lot of other files, that's the problem. Long scan = long run, so limit.

[CarlMangno]

Yes, very true! But I generally only start up FSNotes once, at boot time on my computer, so I would not mind a long start up time.

What do you think of the following? Shall I put in a feature request to have the start up time configurable, and it could default to the current limit?

Take care, Carlo

On Jul 17, 2023, at 18:46, glushchenko @.***> wrote:

Dear Gulshchenko, Infinitie and whole hard disk is a vast overstatement. The problem can be triggered with a small collection of notes in a tree structure. It does seem to depend on the depth of the tree structure. The problem is intermittent and so hard to reproduce, but, I believe, that if a tree structure is permitted, the depth of the tree should not be limited to only one or two levels. Regards, Carl … <x-msg://5/#> There may not be a lot of notes, but there are a lot of other files, that's the problem. Long scan = long run, so limit.

— Reply to this email directly, view it on GitHub https://github.com/glushchenko/fsnotes/issues/1418#issuecomment-1638929763, or unsubscribe https://github.com/notifications/unsubscribe-auth/AURHNZ5G5BS3ERP5Y4J3PXDXQWXBZANCNFSM6AAAAAARUUC3JI. You are receiving this because you authored the thread.

[glushchenko]

We can make it configurable, but there is a risk that the user will specify a very large directory and the application will not load. What if we make activation available only through the command line?

[CarlMangno]

Fantastic! That would be excellent! Those who can use the command line would generally know enough to keep the parameter to a reasonable size.

I would be most happy with that change if you can find time to do it. I realise that conditions for you must be extremely difficult at the moment.

Take care, Carl

On Jul 17, 2023, at 18:58, glushchenko @.***> wrote:

We can make it configurable, but there is a risk that the user will specify a very large directory and the application will not load. What if we make activation available only through the command line?

— Reply to this email directly, view it on GitHub https://github.com/glushchenko/fsnotes/issues/1418#issuecomment-1638942555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AURHNZYV2PTABLF2RNCY3B3XQWYPDANCNFSM6AAAAAARUUC3JI. You are receiving this because you authored the thread.

[xor-gate]

Thanks for having this conversation and reopening this. I'm the author and maintainer of Syncthing macOS. I have a FAQ in the README docs about some "hidden knobs", like disable the Sparkle auto updater. Using the Apple defaults key-value store. This case for fsnotes should also be documented and then implemented to finetune the scanner external storage scanner.

Something like:

$ defaults read com.github.glushchenko.fsnotes
{
   "FSNotesSidebarExternalStorageScanDepth" : 5
   "FSNotesSidebarExternalStorageMaxParentDirs" : 10
}

Then control is easy:

$ defaults write com.github.glushchenko.fsnotes FSNotesSidebarExternalStorageScanDepth 3
$ defaults write com.github.glushchenko.fsnotes FSNotesSidebarExternalStorageMaxParentDirs 20

The com.github.glushchenko.fsnotes context depends on what is in the Info.plist entry. This is just an example.