[Feature Request]: support sorting rule with RegExp

[edentsai]

Hi, Hopes it can support sorting rule with RegExp

for example: I have monthly, weekly and daily notes and other note in the same folder:

/daily-notes
    2022-11.md     # monthly note
    2022-W46.md    # weekly note
    2022-11-11.md  # daily note
    2022-11-11-122313-note-something.md # date: YYYY-MM-DD-HHmmss-...
    2022-11-11-173513-another-note.md
    2022-11-12.md  # daily note    
    2022-11-12-135312-first-note.md
    2022-11-12-182112-second-note.md
    2022-11-13.md  # daily note    

I want to sort the notes in following order:

1. monthly notes
2. weekly notes
3. daily notes
4. other notes

The sort result I want:

/daily-notes
    2022-11.md     # monthly note
    2022-W46.md    # weekly note
    2022-11-11.md  # daily note
    2022-11-12.md  # daily note    
    2022-11-13.md  # daily note   
    2022-11-11-122313-note-something.md # date: YYYY-MM-DD-HHmmss-...
    2022-11-11-173513-another-note.md
    2022-11-12-135312-first-note.md
    2022-11-12-182112-second-note.md

If the sorting rule support RegExp will be better, for example:

---
sorting-sepc: |
    target-folder: /*
    // monthly notes: YYYY-MM.md
    /:files regexp: [0-9][0-9][0-9][0-9]-[0-9][0-9]\.md
        < a-z
    // weekly notes: YYYY-[W]WW.md
    /:files regexp: [0-9][0-9][0-9][0-9]-W[0-9][0-9]\.md
        < a-z
    // daily notes: YYYY-MM-DD.md
    /:files regexp: [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-3][0-9]\.md
        < a-z
    // other notes
    ...
        <a-z
---

thanks.

[SebastianMC]

Hi @edentsai,

good point! There is some rationale behind not supporting sorting rules based on user-entered regexp-s. Namely: it is easy, even for an advanced regexp user, to accidentally introduce the JS regexp backtracking. At best, the observable effect would be some slowdown of Obsidian app. At worst, the Obsidian app would become unresponsive, the regexp engine eating 100% of the processor for significant amount of time. In result, the blame would fall on the custom-sort plugin, not on the badly constructed killer-regexp.

This is why the scenario is avoided by design by not supporting user-entered regexps.

And that's why I'm very very hesitant with adding the explicit-regexp-based sorting rules (even if internally regexps are already in play).

At the same time, there are some options for exposing this functionality. The implementation is trivial, regexps are already used internally, the Roman-numbers ordering is based on regexps, for instance. The feature can be exposed as an undocumented use-at-your-own-risk and disabled-by-default feature, but is it worth it?

An example of a hidden feature is the ability to hide folders and files from File Explorer. It would be convenient for some users to hide the sortspec.md files (and some users do it). At the same time, it is not documented because it has some side-effects which can confuse users.

Please let me know what you think.

BTW with the newly added support for metadata-based sorting of notes and folders, you can impose the required order by introducing a new dedicated metadata. For example: note-type: with set of values like:

• note-type: 01 monthly
• note-type: 02 weekly
• note-type: 03 daily (numbers impose the order when sorting by metadata value)

And then configure sorting as:

---
sorting-spec: |
    target-folder: /*
    with-metadata: note-type
      < a-z by-metadata: note-type
    // other notes
    ...
      < a-z
---

Yes, this would be probably more a workaround rather than a dedicated solution, every note would need to be stamped with metadata explicitly (which could be not feasible or an overkill or simply unwanted, etc.) I've taken the opportunity to show you some different (maybe useful?) perspective.

At the same time please share your thoughts now, when you know why the explicit-regexps are not already supported by the plugin (even if the implementation is trivial and partially in place). I'm open to hear your suggestion(s) and happy to expose this feature as hidden-undocumented if you think it would help you (and not seeing better options)

thanks, Seb

[edentsai]

Yes, I agreed the user-entered RegExp will easily cause many issues which are more dangerous then it's flexibility.

I had found the usage /files: with-metadata: <key> is one of solution for me, but it is not refreshed automatically to avoid it's performance issue:

> The grouping and sorting by metadata is not refreshed automatically 
> after change of the metadata in note(s) to avoid impact on Obsidian performance.

and less flexible if someday I want to re-arrange the order of note, I must to update many notes.

---

I think I might not need the user-entered RegExp feature... here is my need / usage scenario:

• I want to keep my daily notes or important notes be obviously visiable, so I can find and open them quickly from mess notes in same folder.

my current solution is just put the mess notes into a subfolder to isolate them for keep my daily notes are obviously visible :)

I also tried the explicit expression \-d+ for match numbers but sadly it can be used only once per line... \-d+ is a better alternative to me if it be used many times in same line,
such as:

# files {YYYY-MM}.md:  
/:files \-d+\-d+\-d+\-d+-\-d+\-d+.md

(sorry for my weak english)

thanks.

[SebastianMC]

Hi @edentsai

allowing a wider usage of controlled-regexp-alike expressions seems to be the way to go here. While avoiding the risk of the dangerous regexp backtracking, it would allow more flexible sorting grouping rules

Putting it as my next to-do item for the plugin

BTW I don't see any reason for you to be sorry for your English ;-)

[SebastianMC]

Hi @edentsai

the feature is completed in the dedicated branch, awaiting merge to master and release. I need to sleep with it for some more time, to feel safe with the backward compatibility. The unit tests coverage is high, yet not full. I need to decide if to add more unit tests for untested (and modified) areas or if manual tests of representative scenarios are enough.

A new section in the manual.md was added to describe the new feature

When this feature is released, you will be able to use the following syntax (based on your example):

---
sorting-sepc: |
    target-folder: /*
    // monthly notes: YYYY-MM.md
    /:files \[0-9]\[0-9]\[0-9]\[0-9]-\[0-9]\[0-9]
        < a-z
    // weekly notes: YYYY-[W]WW.md
    /:files \[0-9]\[0-9]\[0-9]\[0-9]-W\[0-9]\[0-9]
        < a-z
    // daily notes: YYYY-MM-DD.md
    /:files \[0-9]\[0-9]\[0-9]\[0-9]-\[0-9]\[0-9]-\[0-3]\[0-9]
        < a-z
    // other notes
    ...
        <a-z
---

The \[0-3] expression is undocumented yet supported (just for your specific case), even if the newly added syntax is not fully generic

[SebastianMC]

Feature completed, surprisingly, a substantial refactoring was needed. At the same time it made the core more generic and more friendly to further updates of this type. The initial set of regex-alike expressions is simple, sufficient to handle Your @edentsai example. Will is if there is demand for more