Add a GitHub Pages saver

[pesho-ivanov]

There seems to be ongoing interest in a GitHub Pages saver (tutorial post).

Existing non-local savers include:

1. [depricated] Google Drive saver (#2733)
2. WebDav saver (#738)
3. [dead repo] GitHub saver

[joshuafontany]

+1

Setting up the gh-pages branch isn't hard, but it is esoteric. Being able to push a new demo wiki directly from my local copy would be awesome.

[sukima]

Is the question here how to make a saver or how to publish an already saved wiki to gh-pages. I looked at the tutorial and it is describing the later case.

Publishing to gh-pages is a fairly benign DevOps task I would suggest it is out of scope of TiddlyWiki core but deserving a documentation tiddler on the main site. Perhaps with help of tools like https://github.com/tschaub/gh-pages Especially convenient would be the following package.json for your wiki:

{
  "scripts": {
    "start": "tiddlywiki --server",
    "deploy": "gh-pages -d output"
  },
  "devDependencies": {
    "tiddlywiki": "^5.1.19",
    "gh-pages": "^2.0.1"
  }
}

[Jermolene]

Many thanks @pesho-ivanov for finding https://github.com/38911BytesFree/TiddlyWiki. I had somehow missed that the GitHub API supports pushing content. Anyhow, I've been able to make a TW5 saver based on that code, and will push it shortly.

[technillogue]

I think it would be interesting to use the same approach, but as a SyncAdapter rather than a saver. For setup, have the user paste a github token and initialize a github pages repo with a master and source branch, with the source branch containing .tid files and travis.yml, then open travis with a tutorial on how to log in and set the env vars for the automatic builds to work. (Speeding up the build process somewhat would be helpful but not strictly necessary)

This means that users can fork and PR wikis and git blame or bisect individual tiddlers.

I might try taking a stab at a prototype of this

[Jermolene]

Hi @technillogue that sounds great.

The CI script could bake the sha of latest commit into the wiki, and then the sync adaptor could use it at startup to read any commits since it was baked.

[technillogue]

every new commit immediately bakes a new wiki, I would think? if you make a change to an old wiki you get a new branch or use whatever native merging strategy

[Jermolene]

Hi @technillogue

every new commit immediately bakes a new wiki, I would think? if you make a change to an old wiki you get a new branch or use whatever native merging strategy

I think that wouldn't help users identify whether a wiki was stale, but that perhaps that only matters if the build process is lengthy.

[technillogue]

Are there any approaches to either speeding up the build process or slimming the wiki file?

On Wed, Jul 31, 2019 at 4:20 AM Jeremy Ruston notifications@github.com wrote:

Hi @technillogue https://github.com/technillogue

every new commit immediately bakes a new wiki, I would think? if you make a change to an old wiki you get a new branch or use whatever native merging strategy

I think that wouldn't help users identify whether a wiki was stale, but that perhaps that only matters if the build process is lengthy.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Jermolene/TiddlyWiki5/issues/3890?email_source=notifications&email_token=AAHG4G6TD7HJILQBHEEQQYLQCFDMPA5CNFSM4HDM2ATKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3GPJ5A#issuecomment-516748532, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHG4GZXHM64DYMAZLMQVF3QCFDMPANCNFSM4HDM2ATA .

[pmario]

Slimming the wiki file imo needs more time, since you will need extra handling for the slimming process.

[technillogue]

Could the wiki file be baked seperately from (most) of the tiddlers, load their metadata on page load and tiddler content when that tid is accessed?

On Wed, Jul 31, 2019 at 11:45 AM Mario Pietsch notifications@github.com wrote:

Slimming the wiki file imo needs more time, since you will need extra handling for the slimming process.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Jermolene/TiddlyWiki5/issues/3890?email_source=notifications&email_token=AAHG4G2OP2F4I3RJRLWBCZ3QCGXTFA5CNFSM4HDM2ATKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3HV5VY#issuecomment-516906711, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHG4G6Y6XJ67DTAMK2CPC3QCGXTFANCNFSM4HDM2ATA .

[pmario]

From your fist post, I see you want to use a "sync adaptor" instead of a "saver" + travis to build a new wiki. ... The slowest element in this process is, to start up the CI process, because it needs to create a completely new VM + the needed dependencies (node, tiddlywiki) to actually build and then deploy the new wiki. ...

Depending, how busy the CI/CD system is, this can take several minutes. Except you pay for priority. ....

So I think a different approach may be faster in the end:

• A user starts editing several tiddlers.
• Those tiddlers are temporarily saved in the local store.
• Once the user is done, all modified tiddlers are synced back to git in 1 commit.
• So the CI/CD process only needs to run once, after all the work is done.

A second possibility will follow

[pmario]

An other possibility would be to split tiddlywiki into 2 files that are saved using a "saver":

• tiddlywiki.js which contains the core and eventually side-loaded plugins
• tiddlywiki.html which contains all the content

The core already contains a recipe to build such a wiki. ... BUT ... it is meant to be used in a different context. ... So some little changes would be needed.

tiddlywiki.js could be cached client-side, if the server-side sets the right settings. .. So downloading a new .html element will be much faster. ...

[pmario]

Also saving new content will be faster, since the .html part is much smaller without the core + plugins.

[technillogue]

You only need the CI to run the wiki builder. If it's possible to commit individual tiddlers, they're immediate available via GH pages. If you can load the metadata of whatever the current tiddlers on github are on page load, you only need to rebuild the main file if you've e.g. added a plugin. This way should get faster saves (the current github saver prerelease takes a few seconds for me) and still be able to use fork/PR, git blame, and git bisect.

( I know a lot of config is also tiddlers--not every tiddler would be external, you would just be able add more tiddlers externally)

On Thu, Aug 1, 2019 at 7:40 AM Mario Pietsch notifications@github.com wrote:

From your fist post, I see you want to use a "sync adaptor" instead of a "saver" + travis to build a new wiki. ... The slowest element in this process is, to start up the CI process, because it needs to create a completely new VM + the needed dependencies (node, tiddlywiki) to actually build and then deploy the new wiki. ...

Depending, how busy the CI/CD system is, this can take several minutes. Except you pay for priority. ....

So I think a different approach may be faster in the end:

• A user starts editing several tiddlers.
• Those tiddlers are temporarily saved in the local store.
• Once the user is done, all modified tiddlers are synced back to git in 1 commit.
• So the CI/CD process only needs to run once, after all the work is done.

A second possibility will follow

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Jermolene/TiddlyWiki5/issues/3890?email_source=notifications&email_token=AAHG4G6TENWLUVQJH5KODD3QCLDRBA5CNFSM4HDM2ATKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3KJEDI#issuecomment-517247501, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHG4G4FZ7NISNFHYHV4PQDQCLDRBANCNFSM4HDM2ATA .

[technillogue]

You still wouldn't be about to use VCS tools on that, or at least not as meaningfully, as long as every tiddler is in one file

On Thu, Aug 1, 2019 at 7:46 AM Mario Pietsch notifications@github.com wrote:

Also saving new content will be faster, since the .html part is much smaller without the core + plugins.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Jermolene/TiddlyWiki5/issues/3890?email_source=notifications&email_token=AAHG4G2WJZLJTS5TQRNAJILQCLEHZA5CNFSM4HDM2ATKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3KJU5Q#issuecomment-517249654, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHG4G6WVQT7ACCRQOPJQI3QCLEHZANCNFSM4HDM2ATA .

[pmario]

...You still wouldn't be about to use VCS tools on that ...

So your main goal is using the possibilities a VCS has to offer?

[pesho-ivanov]

I am in the process of testing the GitHub saver and preparing a tutorial (feel free to copy-paste it, assume CC0).

Meanwhile several Qs to @Jermolene:

1. Can we set the Server API URL to a default value "https://api.github.com"?
2. Can the TiddlyWiki be saved as separate files (per tiddle)? What would then be the target file (currently index.html)?

[Jermolene]

Hi @pesho-ivanov

Can we set the Server API URL to a default value "https://api.github.com"?

The server API URL already defaults to "https://api.github.com". Are you having trouble with it?

Can the TiddlyWiki be saved as separate files (per tiddle)? What would then be the target file (currently index.html)?

Not at present. What we call "savers" always save the entire file; to save individual tiddlers as separate files would require a "syncer", which synchronise individual tiddlers.

[pesho-ivanov]

Thank you for the timely explanations, Jermolene. I hope to get deeper in understanding TW soon so that I can be of a greater help.

Meanwhile, I am actively using the gh-saver and here I stumbled upon one problematic use case. To undo unsaved changes, I reloading the page. The hazard is that the browser may not load the newest TW state due to caching. One quick solution I see is disabling browser caching. Do you think HTTP headers can be added to TW in order to disable it (example)?

[pesho-ivanov]

Even without caching, the problem remains that saving TW (push+serve) takes considerable time and there is still the risk to overwrite the latest commit by reloading an older one.

A safety net from data loss would be to save a version number in TW and check for new commits before pushing.

p.s. A timestamp suggested in #4224 will do the work.