datasette-tiddlywiki plugin, initial research

[simonw]

Based on code in https://github.com/Jermolene/TiddlyWiki5/tree/master/core/modules/server/routes

Twitter thread: https://twitter.com/Jermolene/status/1473764012160655371

Also useful: https://tiddlywiki.com/#WebServer%20API

[simonw]

Following README at https://github.com/Jermolene/TiddlyWiki5

~ % npm install -g tiddlywiki

added 1 package, and audited 2 packages in 4s

found 0 vulnerabilities
~ % tiddlywiki --version
5.2.1
~ % cd /tmp
/tmp % tiddlywiki MyNewWiki --init server 
Copied edition 'server' to MyNewWiki
/tmp % ls
MyNewWik
/tmp % tiddlywiki MyNewWiki --listen
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList 
Serving on http://127.0.0.1:8080
(press ctrl-C to exit)
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList 

[simonw]

OK I played with it and it's pretty amazing. And it would be so great to have a SQLite / Datasette backend for it.

Hardest part seems to be figuring out how to render that initial HTML page, since I think that may involve Node.js executing TiddlyWiki code to generate HTML, not just returning JSON?

But... on closer inspection of the source it looks like it might be boilerplate HTML and. then a big blob of embedded JSON, which should be a lot easier to produce.

Here's the code for that: https://github.com/Jermolene/TiddlyWiki5/blob/master/core/modules/server/routes/get-index.js

exports.method = "GET";

exports.path = /^\/$/;

exports.handler = function(request,response,state) {
    var text = state.wiki.renderTiddler(state.server.get("root-render-type"),state.server.get("root-tiddler")),
        responseHeaders = {
        "Content-Type": state.server.get("root-serve-type")
    };
    state.sendResponse(200,responseHeaders,text);
};

I think this is renderTiddler: https://github.com/Jermolene/TiddlyWiki5/blob/7d1f0ea8f4e9955606656d4c458c49e8708c67ac/core/modules/wiki.js#L1190-L1197 - but it looks like it's operating on a DOM?

exports.renderTiddler = function(outputType,title,options) {
    options = options || {};
    var parser = this.parseTiddler(title,options),
        widgetNode = this.makeWidget(parser,options);
    var container = $tw.fakeDocument.createElement("div");
    widgetNode.render(container,null);
    return outputType === "text/html" ? container.innerHTML : (outputType === "text/plain-formatted" ? container.formattedTextContent : container.textContent);
};

[simonw]

The Go code may have clues: https://github.com/rsc/tiddly/blob/master/tiddly.go - it has a main() method here https://github.com/rsc/tiddly/blob/4c01b1863e7ca9f267cb922253950e396a429e63/tiddly.go#L105 which does this:

http.ServeFile(w, r, "index.html")

And index.html is a 2.52MB file here https://github.com/rsc/tiddly/blob/4c01b1863e7ca9f267cb922253950e396a429e63/index.html - I think that's just TiddlyWiki itself.

So maybe all I have to do is implement a plugin that returns TiddlyWiki itself and sets up that small list of paths?

[simonw]

https://github.com/rsc/tiddly/blob/master/README.md says:

The TiddlyWiki downloaded as index.html that runs in the browser downloads (through the JSON API) a master list of all tiddlers and their metadata when the page first loads and then lazily fetches individual tiddler content on demand.

So it's a custom build of TiddlyWiki. In fact the details of that are here: https://github.com/rsc/tiddly/blob/master/README.md#tiddlywiki-base-image

• Open tiddlywiki-5.1.21.html in your web browser.
• Click the control panel (gear) icon.
• Click the Plugins tab.
• Click "Get more plugins".
• Click "Open plugin library".
• Type "tiddlyweb" into the search box. The "TiddlyWeb and TiddlySpace components" should appear.
• Click Install. A bar at the top of the page should say "Please save and reload for the changes to take effect."
• Click the icon next to save, and an updated file will be downloaded.
• Open the downloaded file in the web browser.
• Repeat, adding any more plugins.
• Copy the final download to index.html.

[simonw]

I downloaded empty.html from https://tiddlywiki.com/#GettingStarted

[simonw]

I followed the instructions to install the tiddlyweb plugin and saved a new copy, then I found this in the plugins pane:

Plus a link to this source code: https://github.com/Jermolene/TiddlyWiki5/tree/master/plugins/tiddlywiki/tiddlyweb

[simonw]

https://tiddlywiki.com/#Installing%20a%20plugin%20from%20the%20plugin%20library is interesting:

I wonder if I could do better than that? My tiddlywiki.info file could be persisted in SQLite.

When you restart the Node.js server does it install any missing plugins?

More clues in https://tiddlywiki.com/#Installing%20a%20plugin%20from%20the%20plugin%20library - one option is:

Place the PluginFolders containing the plugins in a plugins folder within the wiki folder.

[simonw]

There is a TINY Ruby implementation here: https://gist.github.com/jimfoltz/ee791c1bdd30ce137bc23cce826096da

And a fork here: https://github.com/korikori/tw5_server

Which says this:

curl https://tiddlywiki.com/empty.html >> folder/empty.html

Then, to start the server, run the following:

ruby tw5-server.rb folder/empty.html

I tried it and it worked! Each time you saved a tiddler it PUT the entire index.html file and saved it to disk again.

It looks like TiddlyWiki knows that it can PUT the file because of this bit: https://github.com/korikori/tw5_server/blob/d09aa95bf8afc3e3b336498dc60433afea66acd0/tw5-server.rb#L44-L48

         def do_OPTIONS(req, res)
            res['allow'] = "GET,HEAD,POST,OPTIONS,CONNECT,PUT,DAV,dav"
            res['x-api-access-type'] = 'file'
            res['dav'] = 'tw5/put'
         end

I think this is the client code: https://github.com/Jermolene/TiddlyWiki5/blob/7d1f0ea8f4e9955606656d4c458c49e8708c67ac/core/modules/savers/put.js

[simonw]

Lots of interesting savers in https://github.com/Jermolene/TiddlyWiki5/tree/master/core/modules/savers - including one that saves to the GitHub API: https://github.com/Jermolene/TiddlyWiki5/blob/master/core/modules/savers/github.js

[simonw]

https://tiddlywiki.com/#Using%20the%20read-only%20single%20tiddler%20view is in interesting feature.

http://127.0.0.1:8080/BobTheTiddler for the Node.js server returned this (I pretty-printed it):

<head>
  <meta content="text/html;charset=utf-8" http-equiv="Content-Type" />
  <meta content="TiddlyWiki" name="generator" />
  <meta content="<<version>>" name="tiddlywiki-version" />
  <meta content="width=device-width, initial-scale=1.0" name="viewport" />
  <meta content="yes" name="apple-mobile-web-app-capable" />
  <meta
    content="black-translucent"
    name="apple-mobile-web-app-status-bar-style"
  />
  <meta content="yes" name="mobile-web-app-capable" />
  <meta content="telephone=no" name="format-detection" />
  <link href="favicon.ico" id="faviconLink" rel="shortcut icon" />
  <link
    href="%24%3A%2Fcore%2Ftemplates%2Fstatic.template.css"
    rel="stylesheet"
  />
  <title>
    BobTheTiddler:My TiddlyWiki — a non-linear personal web notebook
  </title>
</head>
<body class="tc-body">
  <div class="tc-sidebar-scrollable" style="overflow: auto">
    <div class="tc-sidebar-header">
      <h1 class="tc-site-title">My TiddlyWiki</h1>
      <div class="tc-site-subtitle">a non-linear personal web notebook</div>
      <h2></h2>
      <div class="tc-sidebar-lists">
        <div class="tc-menu-list-subitem">
          <a class="tc-tiddlylink tc-tiddlylink-shadow" href="GettingStarted"
            >GettingStarted</a
          >
        </div>
      </div>
    </div>
  </div>
  <section class="tc-story-river">
    <div class="tc-tiddler-frame">
      <div class="tc-tiddler-title">
        <div class="tc-titlebar"><h2>BobTheTiddler</h2></div>
      </div>
      <div class="tc-subtitle">
        <a class="tc-tiddlylink tc-tiddlylink-missing" href=""></a>22nd December
        2021 at 2:32pm
      </div>
      <div class="tc-tags-wrapper"></div>
      <div class="tc-tiddler-body"><p>This is bob</p></div>
    </div>
  </section>
</body>

[simonw]

Looks like this code did that: https://github.com/Jermolene/TiddlyWiki5/blob/master/core/modules/server/routes/get-tiddler-html.js - again using state.wiki.renderTiddler() so that's clearly code that works in Node.js using a fake DOM.

Here's that fake DOM: https://github.com/Jermolene/TiddlyWiki5/blob/master/core/modules/utils/fakedom.js

[simonw]

I want to avoid anything where I'm replicating code from the Node.js server - so I think the Datasette plugin will need to do pretty much what the Go code does:

• Serve index.html which is a static build of TiddlyWiki with that tiddlyweb plugin baked into it
• Serve the various API endpoints that the Go code uses
• That's it

[simonw]

(It would be great if I could figure out a way to build empty.html with the plugin using just command-line tools, but that could be hard)

[simonw]

I tried using https://tiddlywiki.com/empty.html and then committing it to git, installing the plugin and running diff - but the difference was really unpleasant to work with. I couldn't even see it without doing this:

GIT_PAGER='less -r' git diff HEAD^ --word-diff

I think the best way to automate creation of this might be to use https://playwright.dev/ to automate the process of installing the plugin which is pretty crazy!

[simonw]

Got this error from a PUT to my prototype:

[simonw]

https://github.com/Jermolene/TiddlyWiki5/blob/master/core/modules/server/routes/put-tiddler.js generates an Etag like so:

    var changeCount = state.wiki.getChangeCount(title).toString();
    response.writeHead(204, "OK",{
        Etag: "\"default/" + encodeURIComponent(title) + "/" + changeCount + ":\"",
        "Content-Type": "text/plain"
    });

[simonw]

This looks like a more thorough Go implementation (found via GitHub code search for "recipes/all/tiddlers" which found https://github.com/opennota/widdly ) https://gitlab.com/opennota/widdly

Especially https://gitlab.com/opennota/widdly/-/blob/master/api/api.go

[simonw]

Here's how it saves tiddlers: https://gitlab.com/opennota/widdly/-/blob/master/api/api.go#L178-214

// putTiddler saves a tiddler.
func putTiddler(w http.ResponseWriter, r *http.Request) {
    key := strings.TrimPrefix(r.URL.Path, "/recipes/all/tiddlers/")

    var js map[string]interface{}
    err := json.NewDecoder(r.Body).Decode(&js)
    if err != nil {
        http.Error(w, "bad request", http.StatusBadRequest)
        return
    }
    io.Copy(ioutil.Discard, r.Body)

    js["bag"] = "bag"

    text, _ := js["text"].(string)
    delete(js, "text")

    meta, err := json.Marshal(js)
    if err != nil {
        internalError(w, err)
        return
    }

    rev, err := Store.Put(r.Context(), store.Tiddler{
        Key:  key,
        Meta: meta,
        Text: text,
    })
    if err != nil {
        internalError(w, err)
        return
    }

    etag := fmt.Sprintf(`"bag/%s/%d:%032x"`, url.QueryEscape(key), rev, md5.Sum(meta))
    w.Header().Set("ETag", etag)
    w.WriteHeader(http.StatusNoContent)
}

[simonw]

It's doing that thing where macro tiddlers are returned "fat" but regulars are returned "thin" too: https://gitlab.com/opennota/widdly/-/blob/master/store/flatfile/flatfile.go#L123-159

[simonw]

I need a schema.

Tiddlers have a revision which starts at 1 and gets incremented on every save.

Beside that they have a name and a meta which is the JSON. And maybe a text? Not sure about that, need to sniff the Node.js server to see what it does.

[simonw]

Weirdly when I play with the Node.js server it looks like everything always get a revision of 0, because any edits are handled by Draft notes at first.

[simonw]

Found another Go implementation that has a SQLite backend: https://github.com/roelrymenants/liddly/blob/master/repo/tiddler_repo_sqlite.go

[simonw]

The schema it uses is:

CREATE TABLE IF NOT EXISTS tiddlers ( 
    title TEXT, 
    meta BLOB, 
    text TEXT, 
    revision INTEGER,
    PRIMARY KEY(title, revision))

[simonw]

I like the source code of that one the best out of the Go examples I've seen.

https://github.com/roelrymenants/liddly/blob/0d97a4906f082cfcdddb44972234ec6db177be21/tiddlyweb/tiddlyweb.go#L83-L123 is interesting - in PUT it reads the incoming JSON, sets js["bag"] = "bag", extracts js["text"] and then saves what's left in js as the meta column.

[simonw]

I used this research to build and release the first version: https://pypi.org/project/datasette-tiddlywiki/