Support using external-js version of TiddlyWiki

[simonbaird]

I'm not sure how it works, but it seems like it could be useful for Tiddlyhost. See https://github.com/Jermolene/TiddlyWiki5/pull/5570 .

Update Dec 6, 2022: This has been demonstrated by @Telumire at https://tiddlywiki-5-2-4-prerelease-external-js.tiddlyhost.com/ and it works great. I changed the title from "Investigate..." to "Support...". See also the later discussion in this issue.

[pmario]

It splits the wiki into 2 elements. The index.html data section + plugins and the tiddlywikicore-.js which can be cached by the browser.

I think it can safe you some outgoing bandwidth and some harddisk space if the core is "hardlinked" into the wiki directory.

I use it with a local WebDav setup. ...

There is a possibility to move plugins to the core-file but that wasn't part of the PR. If the plugins are part of the core-file, it would be relatively easy to update the core, by swtching only one hard-link. + the reference in the index.html

[simonbaird]

Cool, thanks. It seems ideal for Tiddlyhost. One challenge will be making sure you can still download a working TW file. I guess Tiddlyhost would need to be able to reassemble a standalone TW file for downloading.

[pmario]

I think it should be enough to have a button similar to the "Download Empty" but with a filter that makes it a "Download Wiki" button.

[simonbaird]

Reading through https://github.com/Jermolene/TiddlyWiki5/pull/5570 it seems like the external js version isn't going to be supported long term. So this might be a dead end.

I'd still like to learn more about it though. Perhaps we could use an unsupported version. I'm interested in reducing the amount of data that needs to be sent for a Tiddlyhost save. Currently it's a lot because in includes all of core.js, so saving is quite slow.

[pmario]

The PR didn't introduce a technical problem. It did introduce a philosophical one. Jeremy didn't like, that the mechanism is "too visible" see: https://github.com/Jermolene/TiddlyWiki5/pull/5570#issuecomment-892127428 and it did add a "download" button to the PageToolbar section, which is off-by-default.

I did extract the whole thing into a plugin. But it's not published yet. I want more testing

Only a little change has to be made to the core as mentioned at: https://github.com/Jermolene/TiddlyWiki5/pull/5570#pullrequestreview-722426516

The rest can be handled by the a plugin. ... I'll post a link here, when it's available.

For your usecase, we would also need a version, which doesn't expose the "external-core" to the users.

• If they click the "save button" it will only save the data. .. Done
• If they click the "download" button, they will download the data + core and get a single-file wiki

[simonbaird]

It sounds interesting. Will look forward to reviewing the plugin once it's ready.

For Tiddlyhost downloads I'm thinking about https://sitename.tiddyhost.com/download urls, which would mean the assembly into a standalone empty file would need to happen in Tiddlyhost server side host code also.

[pmario]

For Tiddlyhost downloads I'm thinking about https://sitename.tiddyhost.com/download urls, which would mean the assembly into a standalone empty file would need to happen in Tiddlyhost server side host code also.

That's right. .. But if users create their own buttons, we still need to be sure, that it works.

[simonbaird]

https://tiddlywiki.com/#Using%20the%20external%20JavaScript%20template explains it in some more detail. And perhaps the existence of this page indicates that it will continue to be supported.

I'm pretty keen to try to make this work on Tiddlyhost since I expect it reduces the size down to about a quarter. (See this forum post.

[simonbaird]

@pmario Am I right to assume this is supported now without a plugin?

[pmario]

@pmario Am I right to assume this is supported now without a plugin?

Yes, it's part of the core now. .. I think it can reduce the amount of content that has to be shipped by the server quite a bit. Most TWs just have a view kBytes of content. So the TW core makes a big portion of the size.

The core can be cached by the browser which should also improve start speed. AND saving the wiki will also be sped up quite a bit, since the core hasn't to be transferred by the user.

---

With FF there is a bit of a problem if I create the external core-js file. For me FF always adds a .html extension to the file, which I have to remove manually. ... But I do have an unpublished PR to fix that.

[Telumire]

Hi @simonbaird, I used a plugin made by cdruan to externalise the core js + plugins of my "libro edition", uncompressed file size went from 3,24MB to 0,19MB ! Maybe this will be helpful for your investigation : libro-external-js.tiddlyhost.com vs libro.tiddlyhost.com

Relevant : https://github.com/Jermolene/TiddlyWiki5/issues/7091

Performances were also improved, from a score of 8 to 28 :

- https://pagespeed.web.dev/report?url=https%3A%2F%2Flibro.tiddlyhost.com%2F  - https://pagespeed.web.dev/report?url=https%3A%2F%2Flibro-external-js.tiddlyhost.com%2F 

According to pagespeed.web.dev, text compression could be enabled to improve even more the loading time, but you probably already now about that

[simonbaird]

@Telumire I just noticed it on https://tiddlyhost.com/hub?t=1 and came here to mention it. Brilliant! That's super cool, and it seems to work perfectly on Tiddlyhost. I'm pretty excited about exploring this option for Tiddlyhost.

Rough plan:

• Make it so Tiddlyhost can extract the "core" plugin from a given empty file into a standalone js file, possibly compressed. Likely to be done at build or pre-build time, rather than at run time, since it's static for a given Tiddlywiki release.
• Make it so Tiddlyhost can serve that file at a known url, cached as hard and as long as we can manage.
• Prepare an "empty" with external core js, similar to @Telumire's example and make that available when creating. Perhaps eventually it will become the default option.

Futher considerations:

• What about the "download" action, e.g. do we want to optionally support converting an external js TW to a self-contained TW on download by injecting the core plugin?
• Later maybe: Do we want to support converting existing sites to external js sites, on download or on the fly?
• Even later: Presumably other plugins could be externalized this way, is that a useful concept?

[pmario]

I do use a test-version at GitHub .. It works well there. https://github.com/pmario/test .. The repo contains a link to the test-page

[saqimtiaz]

• What about the "download" action, e.g. do we want to optionally support converting an external js TW to a self-contained TW on download by injecting the core plugin?

Having explored externalizing the core in the past, I never used it in practice because I found that users struggled with the shift from a self contained HTML file, to two files that need to be in the same directory (for offline usage). I suspect TH users may have similar issues, in that was it expected when downloading is a self contained and functional TiddlyWiki file.

[simonbaird]

...expected when downloading is a self contained and functional TiddlyWiki file.

The way I'm picturing it, the core js file will be referenced by an absolute url, i.e. it won't need to be in the same directory as the TiddlyWiki file. This means it would be fully functional, but it would require internet access to work (unless the browser caching was good enough to have the file fully available offline). This need for internet access is something I think we would need to explain, and perhaps provide an option to produce a fully self-contained version when downloading.

[pmario]

This need for internet access is something I think we would need to explain, and perhaps provide an option to produce a fully self-contained version when downloading.

I think it should be possible to create a single file wiki with a TH://index.html?downlad=singlefile.html URI or even a "download single" button in the wiki

[simonbaird]

Agreed.

[simonbaird]

I'm trying to figure out the best way to produce an external js compatible "empty" file and a matching "tiddlywikicore.js" file from the command line. This seems to be pretty close:

node tiddlywiki.js editions/server-external-js --verbose --output ./output/external-js --build index
ls -l output/external-js
google-chrome output/external-js/index.html 

Some questions for the experts:

• What's the difference between $:/core/save/all-external-js and $:/core/save/offline-external-js, and which one would be better for producing an external js "empty" file that could be used on Tiddlyhost, similar to @Telumire's example? IIUC the above is using offline-external-js since that's how it's defined in the relevant tiddlywiki.info build target.
• Would the files produced by this command differ from the files produced with @cdruan's plugin if it was used on a fresh empty file? And if so, how?
• Will I run into problems using the "server-external-js" edition's tiddlywiki.info and build targets, since I don't actually want to use the server with these files?

[cdruan]

Hi @simonbaird

What's the difference between $:/core/save/all-external-js and $:/core/save/offline-external-js, and which one would be better for producing an external js "empty" file that could be used on Tiddlyhost, similar to @Telumire's example?

$:/core/save/all-external-js is only applicable if you'd like to start up the TW server (node.js) by externalizing the core. E.g.

tiddlywiki YOUR_WIKI_FOLDER --listen 'root-tiddler=$:/core/save/all-external-js' use-browser-cache=yes

It replaces the default root-tiddler $:/core/save/all. So you'd want to use $:/core/save/offline-external-js template to produce the .html file. The offline version filters out client/server plugins.

Would the files produced by this command differ from the files produced with @cdruan's plugin if it was used on a fresh empty file?

At this point, the tiddlywikicore.js file should be the same; but there are some duplicate code that may become out of sync in the future... (I originally created the plugin to support earlier versions of TW that do not have working external-js functionality, hence the duplicates.)

The actual wiki file (e.g. empty.html) would be different. Wikis produced by the plugin would require/include the plugin either internally or externally. The plugin's saving UI is improved so that a pop-up appears to remind users to download the necessary JS file(s) in addition to the wiki file. The external-js standalone wiki rendered by the core version only pops up a dialog box to save the wiki. Unfamiliar users will have trouble starting up the wiki if they do not have the required JS file.

Will I run into problems using the "server-external-js" edition's tiddlywiki.info and build targets, since I don't actually want to use the server with these files?

No, I don't think so.

[simonbaird]

Thanks @cdruan, that's really helpful.

[simonbaird]

Well, I think it's working pretty well so far. See https://external-core-test.tiddlyhost.com/ for a demo, plus some explanations and warnings.

[Telumire]

Based on tiddlywiki-5-2-4-prerelease-external-js.tiddlyhost.com

Thanks for your work @simonbaird ! Do you want me to delete https://tiddlywiki-5-2-4-prerelease-external-js.tiddlyhost.com now that there is an official solution ?

[simonbaird]

I realized that the '$:/core' tiddler can come back as an internal tiddler if it is modified and then the site is saved. I guess it's unlikely someone would modify that, but still, it's a bug. There might be some more work needed here to figure out how to prevent that, perhaps by using the $:/core/save/offline-external-js when saving, or by setting '$:/core' to be always readonly somehow. (I'll confess I'm not super clear on how this all works.)

[simonbaird]

Do you want me to delete https://tiddlywiki-5-2-4-prerelease-external-js.tiddlyhost.com now that there is an official solution ?

Up to you, but I'd say let's keep it. The "official solution" is buggy and experimental and also I'm still using your demo as a reference, especially the $:/config/SaveWikiButton/Template which I'm having trouble understanding. (How is it a button?)

[pmario]

especially the $:/config/SaveWikiButton/Template which I'm having trouble understanding. (How is it a button?)

It's a bit tricky and the name may be misleading. .. I'll try to explain with the default setting at tiddylwiki.com and then show the difference between tw-com and the external-js

• The TW Save Button shown in the right sidebar activates the tm-save-wiki message
• tm-save-wiki needs a parameter param " Title of a tiddler to use as a template for rendering the wiki" like so:
  • <$action-sendmessage $message="tm-save-wiki" $param={{$:/config/SaveWikiButton/Template}} filename=<<site-title>>/>
• So it reads the content of the {{$:/config/SaveWikiButton/Template}} tiddler
  • which is: $:/core/save/all
• $:/core/save/all contains "the recipe" to build tiddlywiki.html file
• It does this by defining a filter macro, that's used by the $:/core/templates/tiddlywiki5.html template.

So that's how tiddlywiki.com builds a wiki if the save-button is clicked.

---

Since the external-core version doesn't need the same tiddlers, but also needs to click the same button we need a mechanism to "switch" the config.

• $:/config/SaveWikiButton/Template for the external-js version contains
• $:/core/save/offline-external-js (external-js version)
• which looks very similar to $:/core/save/all but without the -[[$:/core]] and the tiddlyweb related tiddlers-[[$:/plugins/tiddlywiki/filesystem]] -[[$:/plugins/tiddlywiki/tiddlyweb]] tiddlers.
• and it uses a different template {{$:/core/templates/tiddlywiki5-external-js.html}} that you know already

So that's the main difference.

Hope that makes it clearer.

[pmario]

@Telumire wrote Do you want me to delete https://tiddlywiki-5-2-4-prerelease-external-js.tiddlyhost.com/ now that there is an official solution ?

@simonbaird wrote Up to you, but I'd say let's keep it.

I think you should keep it, since I'm using it for my explanation from the last post too. So we will need it atm.

[simonbaird]

Thanks @pmario. It's starting to make more sense.

[simonbaird]

I realized the "empty" I created had the highlight-js plugin included for some reason. I've pushed a new version without that, and now a new "external core" site is just 100 KB.

(Still nearly twice the size of FeatherWiki, but 20 times smaller than a TiddlyWiki with $:/core included, which is a huge deal for Tiddlyhost.)

[simonbaird]

Here's the command I'm using now to create an external-core empty and a core.js:

  git reset --hard && git checkout v5.2.3 && \
  node tiddlywiki.js editions/empty --output ../TiddlyWiki5/output/external-core \
    --render '$:/core/templates/tiddlywiki5.js' '[[tiddlywikicore-]addsuffix<version>addsuffix[.js]]' 'text/plain' \
    --rendertiddler '$:/core/save/offline-external-js' 'index.html' 'text/plain'

[simonbaird]

I would very much like to use https://tiddlyhost.com/tiddlywikicore-5.2.3.js instead of tiddlywikicore-5.2.3.js for the coreURL.

I think I know how to do it now, but there is an easier way to do it coming in the next release of TiddlyWiki, see this.

[simonbaird]

Also todo maybe: Make "tw5-external-core" a different "kind", separate from "tw5".

[pmario]

Also todo maybe: Make "tw5-external-core" a different "kind", separate from "tw5".

I don't understand that. Can you be more specific?

[simonbaird]

Also todo maybe: Make "tw5-external-core" a different "kind", separate from "tw5".

I don't understand that. Can you be more specific?

The second column at https://tiddlyhost.com/sites shows what "kind" a site is. Currently there are three possible kinds: "TW5", "TW Classic" & "Feather Wiki". I think it will be useful to introduce a fourth kind, say "TW5 External", so it's possible to distinguish between the self-contained TW5 sites and the external core TW5 sites. As well as being useful info to show to users, there are places where Tiddlyhost itself needs to use that information, e.g. to do whatever special handling is needed for the external core sites compared to regular sites, such as injecting a custom coreURL variable definition.

[pmario]

I think it's OK to have a new "kind" .. On the other hand as a user I would expect to be no difference if I want to download a single file wiki.

There should be a download button, that just does that. In the end I don't really care how the stuff loads from tiddlyhost. It needs to work and the UX should be fast.

But the most important feature is. I need to trust it. Not about security about data-integrity. Content should never be lost. That's it.

[simonbaird]

There's a non-trivial number of potential "download" behaviors. Rather than get into it here, I'll make a new issue.

[simonbaird]

I think this can be considered done. There is a new "kind" now for the external core TiddlyWiki sites, and there's a unified "core url" for every site (which amplifies the "cache forever" benefits). I moved the "external core" option up the list of options when creating and added some explanations. There might be some follow up work, but I'll make new issues as needed.

Thanks very much @pmario @Telumire @cdruan for the inspiration and technical help.

[simonbaird]

There's a non-trivial number of potential "download" behaviors. Rather than get into it here, I'll make a new issue.

See #283 .

[simonbaird]

One more followup: #284 .