How to use target-folder with folders matching a patern?

[pinuke]

Let's say I have several folders named "I am everywhere." Is there a target-folder:... entry I can use to match every instance of "I am everywhere"

Example file structure:

/
  I am everywhere/
    I am everywhere/
      subfolder/
         I am everywhere/
  subfolder/
    I am everywhere/
      subfolder
  subfolder/
    subfolder/
      subfolder/
      subfolder2/
        subfolder/
          I am everywhere/

or say I have folders with a date suffix, how do I apply sorting rules to all subfolders matching "subfolder [moment.js-formatted-date-string]"?:

/
  subfolder - 01.01.0001/
    jesus christ was born I think.md
    ...
  subfolder - 01.01.0002/
    idk something biblical.md
    ...
  subfolder - 01.01.0003/
    and on this day Mary breathed I think.md
    ...
  subfolder - 01.01.0004/
    Baby jesus moment.md
    ...
  subfolder - 01.01.0005/
    Baby jesus moment2.md
    ...

[SebastianMC]

Hi @smartguy1196

the support for wildcards or multi-matching-names in target-folder: is intentionally kept simplistic and limited to the following cases:

• simple wildcard to match the folder specified by path and its immediate subfolders, indicated by the /... suffix in config
• simple wildcard to match the folder specified by path and the entire tree of its subfolders, indicated by the /* suffix in config

They work in cooperation with the other supported target-folder: cases:

• the obvious simplest one, direct match by exact folder path
  • if a target-folder: value does not contain any / infix it is always treated as a root-level sub-folder name
• multiple adjacent target-folder: lines before the sorting rules so that these sorting rules are applied to all of the listed target folders (this avoids the need to copy & paste the same set of sorting rules N times)
• the not obvious case when the target-folder: line is omitted in the spec and in result the sorting rules are applied to the folder, in which the sorting spec resides
• another not obvious case to exclude folder(s) from custom sort (via the sorting: standard sorting rule) and apply whatever order is selected in Obsidian UI toolbar
  • this makes sense only when used together with the two mentioned simple wildcard types

The primary design goal is to be able to detect duplicate (overlapping) sorting rules at configuration parsing time. And if duplication is detected, inform the user by raising an error. It is done by design to avoid confusion of type 'why the heck %$!@&*! the items in folder XYZ are sorted in a different way than I specified %$#@%$'

The support for regex-alike matching for target-folder: would prevent the sorting engine from detection of duplicate overlapping rules at parsing time. In turn, it could lead to confusion of which of the defined sorting rules was actually applied to a folder.

Regarding your specific example, maybe a combination of the supported target-folder: behaviors can fit your needs?

[pinuke]

Why not design it around rule overwriting, instead of around conflicts? Similar to how CSS rules overwrite each other?

In Cascading Style Sheets it's possible to have multiple patterns match the same HTML elements. The way CSS handles this is simply the rules loaded last, overwrite the previous rules.

Additionally, I do agree it would make more sense to prioritize rules written in local sortspec files.

So, let's say you do support regexing and have a regex pattern in the Vault Root folder. Let's say in that same sortspec file for the Vault Root, you also have another pattern that matches the same set of folders (or even a subset of them)

In CSS, the latter rules would overwrite the earlier rules.

To accommodate for multiple sortspec files, what would make most sense is that the most local sortspec rules should overwrite the rules from the parent sortspec files.

So the overwrite order, should look similar to this (obviously this is a simple model, but it is easy to scale):

Grandparent sortspec file
  Matching target-folder pattern
  Second Matching target-folder pattern
Parent sortspec file
  Matching target-folder pattern
  Another matching pattern
  Another matching pattern <<--if there was no local sortspec file, this pattern's rules should be selected
Local sortspec file
  has it's own set of rules. <<--These rules get selected

[SebastianMC]

Thanks for your great input!

The idea of smart logic of overwriting of rules crossed my mind initially, when I started creating the plugin. Yet, for the CSS you have a rich set of tools (e.g. inspectors) which allows you to easily identify (at run-time) which CSS rule is applied and to track the entire trail of overwritten rules. Thus, troubleshooting is easy, confusion is mostly eliminated.

That's why I originally decided to go with a simpler solution which is more friendly for a not-so-advanced users. It is the conflicts detection and not allowing any overwrites, which is currently present in the plugin.

Now, the plugin being mature and its codebase settled down, and because I appreciate your request and valuable and relevant feedback, I see the direction as follows, in order to implement this new feature:

1. Because the approach 'rules overwritting' is in direct opposition to 'conflicts detection' (from user perspective), a new plugin setting should be added. It should be like 'disable target-folder: conflicts detection' and under the hood, at technical level, it would enable the mechanics of rules overwritting. Maybe label it as experimental or advanced, if the implementation brings any risks
2. To avoid an overkill, allow a limited set of extensions to target-folder:, namely:
   • allow matching folders by the name alone, not by full path only (as it is implemented now)
   • allow a simple wildcard ... as a prefix or suffix
   • allow a limited set of simplistic-regex characters like \d, to represent a digit (and thus match basic date formats) BTW I intentionally don't want to allow a free-form user-entered regexp, because it could lead to performance problems in result of unintentional backtracking. And the blame would fall on me, the plugin author
3. Challenge - keep the extended (and relaxed) rules in line (backward compatible) with what is already supported (/... and /* logic) so that enabling of the rules overwriting doesn't change the pre-existing sorting behavior
4. Technically, parsing engine would be freed of the need to detect conflicts. At the same time it should collect all the target-folder: entries for consumption at run-time. Syntax checking of the new cases needed.
5. New run-time (sorting-time) logic to match each folder in the vault against the hierarchy of collected target-folder: specs
6. A basic form of run-time inspector is a must, even for me at development time. For example a context menu on folders in File Explorer saying 'Show the sorting rule for the folder', and generating the information output to console
7. New set of unit tests with high coverage to guarantee quality

Looks like a non-trivial thing, yet I'm tempted to create a PoC version to at least validate some of the above ideas and try to estimate the effort. This is my hobby after-hours project, anyways. If the above idea turns out to be too big, it is always possible to consider a simpler variant. After completing the PoC I should gain a more detailed insight of what it takes.

[pinuke]

I think it might be a good effort to start with developing or researching an "inspector." It may even be worth to allow the inspector to use full extended regex separately (but limit regex usage in the custom-sort code) for scalability.

I can't imagine that writing one would be too complex. I'm sure Obsidian's Plugin API has methods for scraping the directory tree.

For performance, I would explore caching (inside of the inspector). If:

• a folder doesn't change names or
• pre-existing target-folder: patterns don't change, ...then the inspector should not have to re-inspect the folder.

However, code to detect when to trigger re-inspection would need to be studied. I can think of a few cases where re-inspection code would need to be triggered:

• Folder gets renamed or moved.
• Folder gets added
• target-folder: pattern gets modified/added
• a local sortspec file gets added somewhere in the parent/grandparent folder tree

So, I think a plan of action would be to:

• [ ] separate a project for folder-tree inspection
  • [ ] before writing any code, brainstorm/research all possible cases where re-inspection would need to be triggered
  • [ ] design the folder-tree inspector
  • I do think the inspector should be kept as simple as possible for scalability and support.
    • the inspector should not be used to determine rule overwriting, just folder matching.
• [ ] work on adding the inspector to custom-sort:
  • [ ] design rule-overwrite logic on the custom-sort side.
  • [ ] integrate the inspector's folder-matching into custom-sort
  • [ ] then, (to increase performance and limit edge cases) implement the limited set of extensions to target-folder: (from "2. [...]" of your last comment)

[pinuke]

It's also possible, such an inspector has already been developed. I'm sure there's an npm package or git repo for inspecting file trees using regex.

[pinuke]

or to go even further on 'experimental', you could use a settings flag to use a new target-folder pattern using keywords like:

• /:regex(...)
  • to match an entire directory path
• /:inspect(...)
  • to send commands to the new experimental inspector

[pinuke]

actually, instead of /:[command]() it may be best to use characters that Obsidian doesn't allow in file paths like :,?, or [/]

perhaps, model the command off of URL queries like /:[command]?argname1=value1&argname2=value2

• important to note that obsidian does support the & and = in file paths

for single argument commands, you could probably do something like this: /:regex:[regular expression]

Looking forward to new commit messages!

[SebastianMC]

Hi @smartguy1196,

thanks for a lot for the food for thoughts which you supplied two weeks ago!

After digesting it and considering various trade-offs between flexibility, KISS-principle, human-friendliness, here goes the final implementation. A 1.6.0 release is underway, full documentation was added in the section Options for target-folder: matching of manual.md

To address the examples which you brought up in the beginning of the conversation you can take advantage of the extended functionality:

1)

The specification:

target-folder: name: I am everywhere

will match any folder named I am everywhere regardless of its location in the vault. No regexp-s involved, feature is friendly also to not advanced users.

2)

The specification:

target-folder: regexp: ^subfolder - \d\d\.\d\d.\d\d\d\d$

or equivalent

target-folder: regexp: ^subfolder - [0-9]{2}\.[0-9]{2}\.[0-9]{4}$

or a trickier one:

target-folder: regexp: ^subfolder - [0-3]?[0-9]\.[01]?[0-9]\.[0-9]{4}$

will handle the second example from your original post. Obviously, as in the regular expressions world, you can express the same intention in many ways using regexp. It is a matter of personal preference, requirements and the readability ("in two months will I need a second or a minute to understand my intention for this regexp?")

Thanks again for your valuable input, ideas and interaction!!! It resulted in adding of a new feature to the plugin, giving more flexibility to advanced users and / or for some specific vaults

[pinuke]

Ha aha. I forgot about this gem:

/
 subfolder - 01.01.0001/
   jesus christ was born I think.md
   ...
 subfolder - 01.01.0002/
   idk something biblical.md
   ...
 subfolder - 01.01.0003/
   and on this day Mary breathed I think.md
   ...
 subfolder - 01.01.0004/
   Baby jesus moment.md
   ...
 subfolder - 01.01.0005/
   Baby jesus moment2.md
   ...

You're welcome. I'll have to get around to testing this out. I'll leave this notification open for me until I can get it tested. I'm currently working on other projects at the moment, and I'm too preoccupied to test it now.

[pinuke]

Hey, I don't want to keep you from closing the issue. If I end up testing this and finding something broken, I'll open another issue.

Right now, I have my hands tied up in school work, work, and a few other projects (obsidian-selenium and chromebrew). I'll likely revisit this later though

[SebastianMC]

Hi @smartguy1196,

thanks for the update! I will close the ticket then.

BTW I took a look at the obsidian-selenium project, which you mentioned (and authored). Do I get the idea correctly: the obsidian-selenium should allow to write and execute end-to-end tests using the selenium tool(s) against Obsidian? Sort of: I prepare one or more more test vaults, with their internal configuration, plugins, set of notes, etc. and I can execute (as in 'regular' selenium) tests against these vaults (live in Obsidian)? Sounds great! Having the ability to use selenium against Obsidian would allow - on top of unit tests - cover the scenarios, which are not unit-testable at a plugin level. Fingers crossed!

[pinuke]

Well not just Obsidian, but anything browser based. I've just started working on it, so it's far from ready for use.

I have a private repo that I'm working on rn to make a package manager for webdriver (almost all current webdriver managers don't have support for all webdriver tools, and I want to fix that)

I'm designing a static-file package repository that anyone will be able to use to create their own webdriver manager. It will contain json files that provide instructions that a manager can use for installing driver tools. It works by using GH actions to maintain and add these static files.

This way, anyone can just create a gh action on the repo, and their tool can automatically be added/maintained (no need to manually upload packages)

This also allows the tools to be provided in an open and consistent way.

[pinuke]

The main purpose is to automate the browser, but I am heavily designing it around unit Testing.

I started this, because I got tired of having to manually transfer my school assignments to Obsidian. I wanted to find a way to do this with automation. Perhaps write an API that builds on dataview and templater.

[SebastianMC]

Aaaaaaah, got it! Obsidian automating the browser by the means of selenium, not the other way round :-) Clear.