search

pashashocky / obsidian-note-gallery

A masonry note gallery for obsidian.
GNU General Public License v3.0
192 stars 6 forks source link

[FR] Support for Native obsidian search query, dataview key and frontmatter for sorting #6

Closed wenbopeng closed 9 months ago

wenbopeng commented 11 months ago

Thank you for developing this amazing plugin, I think it may be as useful as dataview.

It would be perfect if local query and dataview key were supported for filtering.

query: "/regex/"(Support Obsidian's own search syntax, especially regex, that would be perfect!)
limit: 100      
recursive: true 
sort: desc      
sortby: frontmatter&dataview key
fontsize: 8pt
pashashocky commented 11 months ago

Think this makes sense to focus on next!

pashashocky commented 11 months ago

@wenbopeng was giving this some thought, so far it's turning out that both the native search functionality and dataview are possible but are non-trivial implementations. So I'd want to start simple.

Could you give me some more examples of what different queries you'd like to write, maybe specifically to dataview and sorting... I think I generally understand the native search + regex ask. This also is related to #4 with regards to dv...

wenbopeng commented 11 months ago

OK The biggest difference between native search and dataview is that the former can search the full text, while dataview can only search front matter or the dataview key in the content.

One application scenario of native search (a little complicated) is that I need to display notes where two words are in the same sentence. I can use regex:

[^\.\n]*(((word1)[^\.\n]*(word2)))[^\.\n]*

In fact, you can enable this regex using /regex/ in Obsidian's search bar.

Note gallery just needs to get the notes hit by this regex.

Another important thing is that many zettelkasten users do not divide their notes into different paths. All their notes are in one folder. So supporting the tag /text/regex is a more practical choice. (I have 3000+ notes, they are all in the same folder😂)

On the other hand, I will use created and modified and rating in frontmatter to record time and rating.

I must use created to fix the creation time of notes, because changing computers or restoring backups will cause ctime or cday to change, but created will never change. I want to use a custom dv key/frontmatter to sort.

pashashocky commented 11 months ago

Wanted to give you a small update.

Note gallery just needs to get the notes hit by this regex.

This sounds simple, but was very difficult to get done due to obsidian restrictions and closed source hacks. Check out discord. Native search support will be ready for testing soon...

I must use created to fix the creation time of notes, because changing computers or restoring backups will cause ctime or cday to change, but created will never change. I want to use a custom dv key/frontmatter to sort.

Good to know, I didn't actually consider this... DV keys might be a little more straightforward to implement, but this is likely a multi-release issue... We likely also want to implement filtering on dv keys to support #4.

Would like to hear opinions on code-block configuration structure:

Ultimately thinking of what would be a good UX? Let's brainstorm specific examples please for code-block...

~~~note-gallery
query: "path: atlas OR path: efforts"  # this is pretty self explanatory as can be tested in native search
dvQuery: "???"    # what options here? and default?
sortBy: "???"       # what options here? and default?
~~~
wenbopeng commented 11 months ago

I think it's best if query completely imitates the syntax of nativeQuery. And I think dvQuery is a proper subset of nativeQuery. So, dvkey and frontmatter are a good idea for sorting

pashashocky commented 11 months ago

@wenbopeng I'm not actually super familiar with the usage of dataview, could you maybe link me the docs on what you're talking about?

query is starting to work really well actually, the only issue I'm finding is dealing with escaping when parsing with YAML...

wenbopeng commented 11 months ago

@pashashocky Dataview, 2023-11-23 09:49

pashashocky commented 11 months ago

@wenbopeng - I know where the Dataview docs are, what specifically are you referring to as dvQuery and dvkey?

Separately - I've just released 0.0.34 - please try using the new query and debugQuery options, they are documented in the README. Native search should be working...

Need to implement correct results sorting next and I can start looking into dataview integration.

wenbopeng commented 11 months ago

@pashashocky dvQuery is the search syntax of dataview. What I mean is that Native search can almost cover the search of dataview, so dvQuery is not high priority , but it is very useful to use dvkey (or in a narrow sense, properties) for sorting.

pashashocky commented 11 months ago

@wenbopeng gotcha, I think that makes sense - the only thing that's currently not supported by native search is checkboxes status, but hopefully that should come soon to address #4.

I've pushed a few more fixes, let me know how the testing of native query goes with the latest version.

wenbopeng commented 11 months ago

@pashashocky

version: 0.0.38

image

pashashocky commented 11 months ago

@wenbopeng - current function is output matches from path + query, but it's good that it's rendering - try removing path for now to see if you receive results from query.

Can you please give a try to:

~~~note-gallery
query: '"knowledge"'
debugQuery: true
~~~
wenbopeng commented 11 months ago

@pashashocky Yes, it works, but even if you limit the number of notes, obsidian is very laggy(Can't even continue to use it). I think you need to consider the performance of the plug-in.

pashashocky commented 11 months ago

@wenbopeng - good, finding similar in #2, will try to make some improvements. Maybe come chat in discord for debugging.

BTW, here's how it is for me: https://dropover.cloud/5a48ed

wenbopeng commented 11 months ago

@wenbopeng - good, finding similar in #2, will try to make some improvements. Maybe come chat in discord for debugging.

BTW, here's how it is for me: https://dropover.cloud/5a48ed

amazing!

wenbopeng commented 11 months ago

@pashashocky

version: 0.0.38

image

I found that if I put double quotes around the string, it works

pashashocky commented 11 months ago

Still laggy?

query: test is best will translate to match mentions of "test" or "is" or "best" query: 'test is best' will translate to match mentions of "test" or "is" or "best" query: '"test is best" will translate to match mentions of "test is best"

wenbopeng commented 11 months ago

@pashashocky Tested again, they are all right. However, the notes must be closed and reopened before they are displayed correctly. This may be a cache issue.

pashashocky commented 11 months ago

@wenbopeng pushed a few more optimizations, bug fixes and cache updating to 0.0.41. Fixed an issue that was not de-bouncing the updates Please run command Drop all cache and re-initialize database... hope we're getting closer :)

wenbopeng commented 11 months ago

@pashashocky Yes, 0.0.41 is better, it is what I want, can I customize the display width/height/content ratio?

wenbopeng commented 11 months ago

@pashashocky attention~I think I've found a serious bug!

image

wenbopeng commented 11 months ago

@pashashocky attention~I think I've found a serious bug!

image

Maybe it's because other search result preview plugin is installed. I don't know if this will affect performance.

pashashocky commented 11 months ago

@wenbopeng yeah, how did you manage that? there's already a piece of code to prevent exactly that recursion problem :) What's your query in this case? Should definitely fix that - but going for a workout, maybe this afternoon

pashashocky commented 11 months ago

I tried replicating but couldn't: https://dropover.cloud/7688dc

SCR-20231124-lrfa

pashashocky commented 11 months ago

@wenbopeng Let me know if you can test this in isolation or it's my plugin, also, we're now 0.0.43, moving fast.

wenbopeng commented 11 months ago

@wenbopeng yeah, how did you manage that? there's already a piece of code to prevent exactly that recursion problem :) What's your query in this case? Should definitely fix that - but going for a workout, maybe this afternoon

This is due to another plugin ivan-lednev/better-search-views: Outliner-like breadcrumb trees for search, backlinks and embedded queries

pashashocky commented 11 months ago

Will take a look, I think it's the same issue as #8, because the patches aren't stacking. Will need to figure this out...

Edit: different issue to 8.

pashashocky commented 11 months ago

@wenbopeng this was a fun and useful bug to fix! thank you for reporting. Live in 0.0.46, please go test. Ultimately an issue on my side, so worthwhile implementing :)

I had a hunch that the patches might be the issue and conflict with the other patching plugins like obsidian-query-control and better-search-views... think both are compatible now.

wenbopeng commented 11 months ago

@pashashocky 0.0.47, Yes, the search results are hidden, that's good. A new issue, if you declare path and query at the same time, the search will be invalid. Try don’t use path:? Anyway, query can achieve everything.

pashashocky commented 11 months ago

@wenbopeng are you sure about that? i've tested that it works in the past and today..? What do you mean by invalid - results from path won't be covered in the debugQuery search window

path and query should be compatible until I make path deprecated, results get combined in union...

here: https://github.com/pashashocky/obsidian-note-gallery/blob/main/src/react/utils/use-files.ts#L179

pashashocky commented 9 months ago

Closing this as this was mainly focusing on enabling native obsidian search and we achieved that. We should continue the conversation about sorting on frontmatter properties and dv keys in #19.