Search does not search the text content of unloaded tiddlers

[willwray]

Quoting from tiddlywiki.com/dev/#LazyLoadingMechanism:

The browser-based search built into TiddlyWiki5 will only search the text of tiddlers that have been fully loaded. The expectation is that when lazy loading is used in a client-server configuration, then it is the server that really needs to handle search operations, because it is only the server that can "see" the text of all tiddlers. So, the plan is to integrate the built in search with TiddlyWeb's search API. The simplest approach is that any local search triggers an asynchronous server side search. The results of the search would be asynchronously loaded such that they would dynamically appear in the local search results.

TiddlyWeb's search API tiddlyweb/search

I guess that google's app engine datastore is easily searchable from Go code. Is asynchronous request->response feasible for dynamic search results?

Or, is there a workaround for now such as loading fat tiddlers instead of skinny tiddlers? (Surely for any truly tiddly tiddler worth its title, its text will be smaller than its metadata anyhow.)

Thanks for publishing this cool tiddly project :+1: :cat2: :blowfish:

[rsc]

I think that implementing search on the server would be fairly straightforward. That link you found about the tiddlyweb/search API is interesting, but I can't find any evidence that it's used. In particular, I have the jermolene/TiddlyWiki5 repo checked out and can't find any mention of the /search URL. There are some references to a special $:/temp/search tiddler, but no code that does a GET of /search on the tiddlyweb server.

core/modules/commands/server.js has what looks like a local tiddlyweb server (for node.js?), and it doesn't mention search.

plugins/tiddlywiki/tiddlyweb is the tiddlyweb client, and it doesn't mention search either.

Maybe I have an old copy? Or maybe the docs were written as a design but never implemented?

In any event, if you can figure out how to make TiddlyWiki use /search, it should be easy to implement.

[cdent]

plugins/tiddlywiki/tiddlyweb is the tiddlyweb client, and it doesn't mention search either.

I don't think you have an old copy. The functionality provided at /search by the python implementation of TiddlyWeb is one of many things in tiddlyweb that TW5 doesn't (yet?) support. There has been discussion at various times of extending search in either tw5 or tw classic such that a local search can failover to a remote search but as far as I know nothing other than proof of concepts existed. Another option considered (but mostly dismissed) was if a search was made to go ahead and load everything not already loaded.

What probably needs to happen is greater attention to the tiddlyweb plugin in TW5, but I'm not clear on who is actively involved in that.

[willwray]

Searching TiddlyWiki5 GitHub issues for 'search' hits 258 of 568 issues with 112 open. I only find one that explicitly mentions this search issue; TiddlyWiki5 #1194: TW5 and TiddlySpace / TiddlyWeb, opened three years ago with input from @cdent.

Unfortunately, due to the recent demise of TiddlySpace, this issue was revived by @pmario and closed by @tobibeer two days ago. As Chris notes, the issues are all still relevant to TiddlyWeb.

A replacement TW5 and TiddlyWeb issue is needed:

• without the TiddlySpace reference
• focused on what's needed for this and similar 'tiddly-server' projects using tiddlyweb plugin

Perhaps broader TiddlyServer issues should be separated out: There appear to be more general 'tiddly-server' architectural issues related to the separation of server-core index.html and to the serving of skinny tiddlers for plugins and macros , c.f. similar sync-adaptor issues.

I'm still interested in workarounds or hacks to get search working. Is the 'fat' parameter supported such that all tiddlers are fetched non-skinny?

Relevant TW5 'tiddlyweb' discussion: https://github.com/Jermolene/TiddlyWiki5/issues/2694 https://github.com/Jermolene/TiddlyWiki5/pull/2371

[rsc]

The "skinny tiddler list" is fetched frequently (every minute?), so you don't want to make it fat in general. Even expanding $:/tags/Macros-tagged tiddlers, as I did, is more than I'd like. Ideally I would like to distinguish the first skinny load from the others, so that the first one can be a little less skinny. An equivalent, probably better alternative would be to insert the needed fat tiddlers into index.html itself - that's the real first load, and there are a few tiddlers for which even the first skinny load is too late. If you got that right then I think for example it might be possible to add new modules without regenerating index.html served by this server (rsc/tiddly).

Either way, if you can identify the first load, then you could include all text tiddlers (maybe up to a certain size) in that load, and then the default search would work for them. One of the things I like about the on-demand loading is I can drop PDFs and other large files into my tiddlywiki and they don't slow down ordinary operation. Those shouldn't be fattened in this situation.

I don't see any code using fat=1 in the tiddlyweb plugin, so rsc/tiddly does not support it.

Best thing to investigate is probably how to munge index.html during the initial serve to include full tiddler content for tiddlers matching some condition.

[cdent]

An equivalent, probably better alternative would be to insert the needed fat tiddlers into index.html itself

This is what the tiddlywebwiki plugin for the tiddlyweb server does (for tiddlywiki classic, not 5; there's not yet been similar plugin for 5): rather than having a static index.html, it uses one that is templatized and loads it up with some or all of the tiddlers in a recipe. For large tiddlywikis this proved very slow, and led to a variety of lazy/dynamic loading systems. The one that worked best and might be easiest to translate into this system was for there to be a filter/tag on a suite of tiddlers so that some were labelled as being required at startup. These were injected into index.html when it was read. Everything else was loaded dynamically. If the label was passed as a query string, that could provide some configurability.

tiddlywebwiki: https://github.com/tiddlyweb/tiddlywebwiki

[tobibeer]

@cdent, was there some server side-search component in the frontend for anything lazy loaded on TiddlySpace / TWC?

[pmario]

There was no UI in TWc, but if you know the URL pattern, you can initiate a server side search.

[cdent]

@tobibeer not that I recall but it's probably worth mentioning that it was 3 years ago that I stopped paying attention to tiddlyspace, and for the last more than a year I didn't really pay that much attention to the tiddlywiki aspects, as the single page apps using aggregations of tiddlers was always more important to me.

When I implemented dynamic (not the same as lazy) loading for tiddlywiki classic in tank it was in this commt. I'm not sure if that is helpful in any way other than showing one of the strategies.

But yeah, basically what @pmario says: if there's something handling /search you can query it to get more tiddlers. rsc doesn't currently have that.