Could the homepage persist state rather than resetting each time?

[simonw]

As illustrated by this GIF, if you interact with the plugin and then refresh the page it goes back to the default Getting Started layout:

I can't figure out how to get it so refreshing the page keeps the new layout.

The new layout appears to be persisted - a tiddler called $:/StoryList is created and updated.

But... when the page loads from scratch, that information is forgotten.

[simonw]

I have a hunch that what is happening here is that the index.html page that is served by the server includes that layout, and hence that static version is rendered before any data is fetched from the JSON APIs.

My guess is that the Node.js version of tiddlyweb dynamically renders that initial HTML page - which it can do because it has the ability to run TiddlyWiki's code server-side.

My plugin only runs Python server-side, so I can't do that.

Am I right about why this is happening? And is there a workaround?

The index.html I am serving up is default TiddlyWiki from https://tiddlywiki.com/empty.html with the tiddlyweb plugin installed and used to generate a version that's default+tiddlyweb plugin.

[Jermolene]

Hi @simonw

As illustrated by this GIF, if you interact with the plugin and then refresh the page it goes back to the default Getting Started layout: I can't figure out how to get it so refreshing the page keeps the new layout.

The new layout appears to be persisted - a tiddler called $:/StoryList is created and updated.

But... when the page loads from scratch, that information is forgotten.

You're correct about the tiddler $:/StoryList containing the list of titles of the tiddlers that are currently displayed in the story river.

When it starts up, TiddlyWiki chooses the initial tiddlers to display as follows:

• If the URL location hash is not empty, the contents are URL decoded to give <title>: [<title> ]. The first title before the colon is the highlighted tiddler, and the list of space-separated titles after the colon is the story sequence
• Otherwise, the tiddler $:/DefaultTiddlers is evaluated as a filter expression to give the list of titles to be opened in the story sequence. So if you always want to open the same list of tiddlers, just list their titles in $:/DefaultTiddlers

The upshot is that $:/StoryList is overwritten on startup.

There are two ways to make the story sequence sticky so that refreshing the page retains the current ordering:

• Set $:/DefaultTiddlers to [list[$:/StoryList]] (this can be done from the default tab of the control panel with a button labelled "retain story ordering"). The filter expression then evaluates to the existing current story list, and thus overwriting it with its current value
• Select "Include the target tiddler and the current story sequence" under "Navigation Address Bar" in the list of options under the control panel "Settings" tab (equivalently, set the tiddler $:/config/Navigation/UpdateAddressBar to permaview

The advantage of the first approach is that URLs remain clean, while the continually updated address bar of the second approach means that ctrl-L/ctrl-C will always give a permalink to the currently open tiddlers.

[simonw]

Set $:/DefaultTiddlers to [list[$:/StoryList]] (this can be done from the default tab of the control panel with a button labelled "retain story ordering"). The filter expression then evaluates to the existing current story list, and thus overwriting it with its current value

I've done that but it doesn't seem to be working. Here's what my settings screen looks like:

Here's the record in the SQLite database:

Here's what /recipes/all/tiddlers.json returns:

[
  {
    "created": "20211223022941943",
    "creator": "me",
    "tags": [],
    "title": "This is my new wiki",
    "modified": "20211223022951953",
    "modifier": "me",
    "type": "text/vnd.tiddlywiki",
    "text": "OK, let's see how well this works.",
    "revision": 1
  },
  {
    "created": "20211223023023183",
    "creator": "me",
    "title": "$:/SiteTitle",
    "modified": "20211223023033081",
    "modifier": "me",
    "type": "text/vnd.tiddlywiki",
    "text": "Simon's Wiki",
    "revision": 5
  },
  {
    "created": "20211223023035690",
    "creator": "me",
    "title": "$:/SiteSubtitle",
    "modified": "20211223023038971",
    "modifier": "me",
    "type": "text/vnd.tiddlywiki",
    "text": "Running datasette-tiddlywiki",
    "revision": 5
  },
  {
    "created": "20211223023048219",
    "creator": "me",
    "title": "$:/DefaultTiddlers",
    "modified": "20211223023048219",
    "modifier": "me",
    "type": "text/vnd.tiddlywiki",
    "text": "[list[$:/StoryList]]",
    "revision": 1
  },
  {
    "created": "20211223023104671",
    "creator": "me",
    "title": "$:/config/PageControlButtons/Visibility/$:/core/ui/Buttons/new-journal",
    "modified": "20211223023104671",
    "modifier": "me",
    "type": "text/vnd.tiddlywiki",
    "text": "show",
    "revision": 1
  },
  {
    "created": "20211223023109702",
    "creator": "me",
    "title": "22nd December 2021",
    "tags": [
      "Journal"
    ],
    "modified": "20211223023118858",
    "modifier": "me",
    "type": "text/vnd.tiddlywiki",
    "text": "What does the journal do?",
    "revision": 1
  },
  {
    "created": "20211223023303755",
    "creator": "me",
    "title": "$:/config/Navigation/UpdateAddressBar",
    "modified": "20211223023303755",
    "modifier": "me",
    "type": "text/vnd.tiddlywiki",
    "text": "permalink",
    "revision": 1
  },
  {
    "created": "20211223023308969",
    "creator": "me",
    "title": "$:/config/Navigation/UpdateHistory",
    "modified": "20211223023308969",
    "modifier": "me",
    "type": "text/vnd.tiddlywiki",
    "text": "yes",
    "revision": 1
  },
  {
    "created": "20211223074205441",
    "creator": "me",
    "title": "$:/config/BitmapEditor/LineWidth",
    "modified": "20211223074205441",
    "modifier": "me",
    "type": "text/vnd.tiddlywiki",
    "text": "28px",
    "revision": 1
  },
  {
    "title": "$:/StoryList",
    "fields": {
      "list": "GettingStarted"
    },
    "type": "text/vnd.tiddlywiki",
    "text": "",
    "revision": 104
  }
]

@Jermolene any idea what's going wrong there?

Also, can you confirm that my hunch about the index page returning the default TiddlyWiki is NOT the reason I'm seeing this behaviour? Be great to not have to worry about needing to dynamically generating that 2MB of HTML.

[simonw]

... oh wait! Just spotted this in that /recipes/all/tiddlers.json output:

  {
    "title": "$:/StoryList",
    "fields": {
      "list": "GettingStarted"
    },
    "type": "text/vnd.tiddlywiki",
    "text": "",
    "revision": 104
  }

Looks to me like something is resetting $:/StoryList to just have GettingStarted in it - and revision 104 suggests that record has been updated a lot.

[simonw]

Confirmed: any time I run a fresh reload of the whole page, the $/StoryList record is reset to the default GettingStarted one.

I just edited my TiddlyWiki and my $:/StoryList record looks like this:

Then I refreshed the browser and the record updated to this:

Note that the revision incremented from 118 to 119 and the contents of the StoryList was reset.

[Jermolene]

Hi @simonw

Also, can you confirm that my hunch about the index page returning the default TiddlyWiki is NOT the reason I'm seeing this behaviour? Be great to not have to worry about needing to dynamically generating that 2MB of HTML.

I'm afraid that's exactly it. The tiddler $:/DefaultTiddlers is currently baked into the index.html file with the text GettingStarted. The sync process almost immediately replaces it with the proper value read from tiddlers.json, but the startup process has already got to work on the original value, hence the overwrite.

This problem has some other manifestations that may make it worth fixing – in particular TiddlyWiki plugins containing JS modules cannot be hot loaded, and must be baked into the HTML file.

Baking the tiddlers you need into a TiddlyWiki is fairly easy: the tiddlers are stored in a JSON array in a script tag. There are some docs here:

https://tiddlywiki.com/dev/#Data%20Storage%20in%20Single%20File%20TiddlyWiki

If you view source on a TW5 and search for <!--~~ Ordinary tiddlers ~~--> you'll see immediately.

[simonw]

Great! It sounds like I can solve this with a custom tiddlywiki.html build then.

I'm currently creating that file using the process described by the Go/AppEngine repo here: https://github.com/rsc/tiddly/blob/4c01b1863e7ca9f267cb922253950e396a429e63/README.md#tiddlywiki-base-image

I'd like to automate the process but that looks a bit fiddly - so I'm OK with doing this manually right now.

[simonw]

Aha! Yes I see this in the source now:

So it looks like another option is that I could dynamically generate just a small portion of that HTML to dump the tiddlers from the database into the page for the initial load.

[Jermolene]

I'd like to automate the process but that looks a bit fiddly - so I'm OK with doing this manually right now.

That makes sense.

[Jermolene]

So it looks like another option is that I could dynamically generate just a small portion of that HTML to dump the tiddlers from the database into the page for the initial load.

Exactly, that would be the most flexible approach if practicable.

[Jermolene]

Exactly, that would be the most flexible approach if practicable.

Given a copy of https://tiddlywiki.com/empty.html it should just be a matter of searching for the magic marker and splicing in your JSON.

Some of the interesting applications of the integration would involve stitching together wikis where the tiddlers that make up the mechanism (plugins etc.) come from a different database than the tiddlers making up the payload. Then one can mix and match things: eg making a generic viewer based on the plugins from https://tiddlymap.org/, combining it with different datasets for exploration.

[simonw]

I'm going to try this: on page load I'll server-side replace the "$:/StoryList" plugin in the default plugins with the one from the database, if one exists. I'll leave the other plugins in the list as they are.

[Jermolene]

That should work!

[simonw]

I wanted to check that it would be safe to use a Python regular expression to extract the contents of the <script class="tiddlywiki-tiddler-store" type="application/json">(.*?)</script> block - what happens if a Tiddler contains the text </script>?

So I created one and saved it to see, and it looked like this:

{"created":"20211223200014515","text":"What does a \u003C/script>...

Looks like this is where that happens: https://github.com/Jermolene/TiddlyWiki5/blob/d5d73e02e9080a4c6f779e94a4ab281cd8a2e7b6/core/modules/widgets/jsontiddler.js#L44-L47

    // Escape unsafe script characters
    if(this.attEscapeUnsafeScriptChars) {
        json = json.replace(/</g,"\\u003C");
    }

So that's great, I can use a regular expression in Python to extract the list of plugins from the default HTML.

[Jermolene]

So that's great, I can use a regular expression in Python to extract the list of plugins from the default HTML.

Excellent! You should do the same escaping ideally, for interoperability with other tools.

[simonw]

Haven't managed to get this to work with. Here's my work-in-progress:

diff --git a/datasette_tiddlywiki/__init__.py b/datasette_tiddlywiki/__init__.py
index 570cef7..b9438a9 100644
--- a/datasette_tiddlywiki/__init__.py
+++ b/datasette_tiddlywiki/__init__.py
@@ -2,11 +2,15 @@ from datasette import hookimpl
 from datasette.utils.asgi import Response, NotFound
 import json
 import pathlib
+import re
 import textwrap
 import urllib

 html_path = pathlib.Path(__file__).parent / "tiddlywiki.html"
-
+tiddler_store_re = re.compile(
+    r'<script class="tiddlywiki-tiddler-store" type="application/json">(.*?)</script>',
+    re.DOTALL
+)

 @hookimpl
 def startup(datasette):
@@ -32,10 +36,47 @@ def startup(datasette):
     return inner

-async def index(request, datasette):
+async def index(datasette):
     try:
         db = datasette.get_database("tiddlywiki")
-        return Response.html(html_path.read_text("utf-8"))
+        html = html_path.read_text("utf-8")
+        # https://github.com/simonw/datasette-tiddlywiki/issues/4
+        # If a tiddler for `$:/StoryList` exists, we need to replace it
+        story_list = (
+            await db.execute(
+                "select title, meta, text, revision from tiddlers where title = ?",
+                ["$:/StoryList"],
+            )
+        ).first()
+        if story_list:
+            # We need to make a small change to the tiddlers baked into that page
+            def replacer(match):
+                tiddlers = json.loads(match.group(1))
+                # Replace the StoryList tiddler
+                new_tiddlers = []
+                for tiddler in tiddlers:
+                    if tiddler.get("title") == "$:/StoryList":
+                        story_tiddler = tiddler_to_dict(story_list, "$:/StoryList")
+                        print("old")
+                        print(tiddler)
+                        print("new")
+                        if "fields" in story_tiddler:
+                            story_tiddler.update(story_tiddler.pop("fields"))
+                        # Copy created and modified
+                        story_tiddler["created"] = tiddler["created"]
+                        story_tiddler["modified"] = tiddler["modified"]
+                        # Drop type and revision
+                        del story_tiddler["type"]
+                        del story_tiddler["revision"]
+                        print(story_tiddler)
+                        new_tiddlers.append(story_tiddler)
+                    else:
+                        new_tiddlers.append(tiddler)
+                return '<script class="tiddlywiki-tiddler-store" type="application/json">{}</script>'.format(
+                    json.dumps(new_tiddlers).replace("<", "\\u003C")
+                )
+            html = tiddler_store_re.sub(replacer, html)
+        return Response.html(html)
     except KeyError:
         return Response.text(
             "You need to start Datasette with a tiddlywiki.db database", status=400
@@ -52,11 +93,7 @@ async def all_tiddlers(datasette):
     for row in (
         await db.execute("select title, meta, text, revision from tiddlers")
     ).rows:
-        tiddler = json.loads(row["meta"])
-        tiddler["title"] = row["title"]
-        tiddler["text"] = row["text"]
-        tiddler["revision"] = row["revision"]
-        tiddlers.append(tiddler)
+        tiddlers.append(tiddler_to_dict(row, row["title"]))
     return Response.json(tiddlers)

@@ -117,11 +154,7 @@ async def tiddler(request, datasette):
         ).first()
         if row is None:
             raise NotFound("Tiddler not found")
-        output = json.loads(row["meta"])
-        output["title"] = title
-        output["text"] = row["text"]
-        output["revision"] = row["revision"]
-        return Response.json(output)
+        return Response.json(tiddler_to_dict(row, title))

 async def delete_tiddler(request, datasette):
@@ -164,3 +197,11 @@ def menu_links(datasette):
     return [
         {"href": datasette.urls.path("/-/tiddlywiki"), "label": "TiddlyWiki"},
     ]
+
+
+def tiddler_to_dict(row, title):
+    output = json.loads(row["meta"])
+    output["title"] = title
+    output["text"] = row["text"]
+    output["revision"] = row["revision"]
+    return output

[Jermolene]

Looks OK through the very narrowed eyes of someone who doesn't know Python very well. I can't tell if it's a problem here, but one thing is that the JSON structure in the store area does not stash the meta fields into a sub-object called fields, as we do for the API calls.

[simonw]

Thanks to #6 I now have a hook in the main code for manipulating the default baked in tiddlers: https://github.com/simonw/datasette-tiddlywiki/blob/3d2f5180463ceaf4f65b392395e825577ac1899d/datasette_tiddlywiki/__init__.py#L78-L95

[simonw]

I think I want $:/StoryList GettingStarted as the default tiddlers.

[Jermolene]

You could use GettingStarted [list[$:/StoryList]] to open any previously opened tiddlers plus GettingStarted if it is not included.

[simonw]

I pushed my code to a branch here but I cannot get it to work: https://github.com/simonw/datasette-tiddlywiki/blob/cef0eebe25520c5100a1e2485ed8baf9a10c2766/datasette_tiddlywiki/__init__.py

No matter what I try the StoryList tiddler stored on the server keeps getting reset to this:

{"title": "$:/StoryList", "fields": {"list": "GettingStarted"}, "type": "text/vnd.tiddlywiki"}

[Jermolene]

I pushed my code to a branch here but I cannot get it to work: https://github.com/simonw/datasette-tiddlywiki/blob/cef0eebe25520c5100a1e2485ed8baf9a10c2766/datasette_tiddlywiki/__init__.py

No matter what I try the StoryList tiddler stored on the server keeps getting reset to this:

{"title": "$:/StoryList", "fields": {"list": "GettingStarted"}, "type": "text/vnd.tiddlywiki"}

Perhaps the splicing hasn't worked, and so there's no valid $:/StoryList tiddler in the JSON store area?

Could you post an extract of the spliced HTML file?

[simonw]

Sure, here's the full file I'm serving after applying that replacement: https://gist.githubusercontent.com/simonw/78ae3eed9eee6f0faa3cb0fd18308a04/raw/262631f790482494276b460dee90b82ed6600e32/tiddlywiki.html

Search for "title": "$:/StoryList" on that page to find the embedded Tiddler, it looks like this:

{"created": "20211222224039169", "title": "$:/StoryList", "text": "", "list": "Drawing GettingStarted", "modified": "20211222224039169"}

[Jermolene]

Hi @simonw the tiddler $:/DefaultTiddlers is set to GettingStarted which causes TiddlyWiki to automatically open that tiddler at startup, overwriting $:/StoryList. The fix should be to set $:/DefaultTiddlers to [list[$:/StoryList]] which will retain the current value in $:/StoryList.