Enable browser-message import from any TW

[twMat]

Without leaving the wiki, I want to fetch tiddlers from arbitrary remote TWs that are not set up in any particular way other than that they're online.

Currently, the library mechanism enables this using the browser-messaging module. However, this seems especially made to fetch from special pages (eg), not a standard TW.

Proposal: A browser-messaging module to handle any TW.

If this is implemented we actually have "federation", just like that, and a thing people can build further applications on top of.

[Jermolene]

Hi @twMat

It's true that we currently only have a "server" side implementation of browser messaging built into TiddlyWiki. In other words, TiddlyWiki knows how to use browser messaging to talk to an HTML file in an iframe, but it doesn't know how to be that HTML file itself, responding to requests from other wikis that have embedded it.

The "server" side implementation lives in a special standalone HTML app that implements the plugin library:

https://github.com/Jermolene/TiddlyWiki5/blob/master/plugins/tiddlywiki/pluginlibrary/libraryserver.js

In terms of experimentation, I think you can probably get a long way with working with the two existing components. Remember, though, that these shenanigans are only necessary because GitHub doesn't support CORS. Now that we've developed access to the GitHub API we could actually read plugins directly from GitHub, without worrying about CORS.

Proposal: A browser-messaging module to handle any TW.

I'm taking you to mean "incorporating a serverside implementation of the browser-messaging protocol into the core".

If this is implemented we actually have "federation", just like that, and a thing people can build further applications on top of.

I'm pretty sure that @inmysocks used the existing plugin library code that we're talking about within his federation implemention, so I think we've collectively gone down this path.

[twMat]

I'm taking you to mean "incorporating a serverside implementation of the browser-messaging protocol into the core".

Yes, so that a TW could fetch any tiddler from any TW.

Both the github route you mention and a plugin solution (like @inmysocks ) largely miss the main point by demanding things on the source side which is beyond the fetchers control. If the condition was only on the fetching side, it'd be OK.

Is the OP not a good idea or how come there is no focus on it? Security wise I don't see any difference from when you DnD import. There's definitely a "universal" need for this and it would enable a ton of things.

[Jermolene]

Both the github route you mention and a plugin solution (like @inmysocks ) largely miss the main point by demanding things on the source side which is beyond the fetchers control. If the condition was only on the fetching side, it'd be OK.

I'm not certain that we can have the serverside of browser messaging enabled by default. In some configurations it might represent a security or phishing risk.

Is the OP not a good idea or how come there is no focus on it?

The motivation for the plugin library architecture is the way that it works around the absence of CORS on many hosts (e.g. TiddlySpot, GitHub Pages). We realised that the browser messaging implementation we ended up with also provided the possibility for inter-wiki communication via iframes, and that triggered @inmysocks experiments.

I think that the best way forward is to pick up @inmysocks experiments, analyse the primitives that he's implemented, and try to reuse them to build something else.

[twMat]

Jed did a fantastic job with TWederation but the js is simply beyond my capabilities and Jed himself has other priorities since then. Unless perhaps @AnthonyMuscio is interested I don't think anyone else cares about the issue. So, accepting this reality I'm backing up a little to when you say;

Now that we've developed access to the GitHub API we could actually read plugins directly from GitHub, without worrying about CORS.

Does this mean gh hosted TWs could freely allow fetching of any tiddlers?

---

Let me explain a bit more about my vision (as originally co-developed with @inmysocks ) with the purpose of clarifying the end goal for my OP rather than talking about technical solutions that may rest on my faulty assumptions for what is necessary. I'd love to hear ideas on how this could safely be achieved:

A twCard is a tiddler with its most important data being a url of a source wiki and where this url serves as the twCards unique ID. The original twCard is probably(!) found on that ID url but there is also a central repository of "all" twCards, sent in by whoever (doesn't have to be by the "url owner")

A twCard could be titled e.g "GreatestMusic", "TW docs", "Palettes", "TW News", "Filters" etc (even "Great twCards") and, beyond the ID-url, each twCard contains other urls to arbitrary TWs and possibly also url-associated filters such as [tag[OneHitWonders]] or [tag[Filter operator]sort[modified]limit[10]].

The fetcher has a library-type plugin that uses his local set of twCards, as picked up from the central server or elsewhere or even made by himself, to list the "preview list" of goodies i.e like the current libraries do. The plugin can also update the twCards themselves via the ID-url.

Note: The user decides which sources (ID-urls) he trusts.

[inmysocks]

What you are describing is quite literally what the twederation plugin does.

As Jeremy said, the primitives for what you want exist in the twederation plugin, they were necessarily the first thing I implemted and are only slight variations on the system used by the plugin library. And, again as Jeremy said, having this turned on by default is not a good idea. People should be able to decide if and what they want no share and the default state should be privacy.

If you want I can pull out just the components needed for sharing and fetching tiddlers. I am certain they can be improved but they implement the functionality that is the base for any system for sharing with tiddlywiki.

[twMat]

If you want I can pull out just the components needed for sharing and fetching tiddlers.

@inmysocks very kind of you! I'm assuming it is the communications-daemon.js and the recognized-sources.js?

But so does the twederation plugin create an equivalent of that special server page but instead inside the iframed TW? Is this perhaps the bundle?

What I fail to see is why the serving side is required to have something installed considering how the full TW is pulled over. I reason it should be possible to read and extract the desired tiddlers once it is inside the fetching wiki.

People should be able to decide if and what they want no share and the default state should be privacy.

I believe Jeremys point to not have it turned on by default rather was for security reasons. As for deciding if to share or not, how does this differ from the current status where DnD importing is possible all the time? In my view, the discussed mechanism would just be a more convenient way - ??? We are talking about public TWs.

But maybe the native TW could have a simple state tiddler that is false by default to allow fetching or not. This means the serving side would make an intentional effort to share but it would at least not demand he installs a plugin to do so.

[inmysocks]

Communication between an iframe and the containing page is pretty strictly limited, this means that the wiki in the iframe needs to have some javascript components that handle sending and receiving data from the containing page. This is a necessary safety feature for sandboxing the content of an iframe and browsers are strict about not letting anything get around it.

Also, I am never going to accept the reasoning 'It isn't perfect so why not make it worse.', particularly not when it comes to privacy or safety features.

[twMat]

@inmysocks thanks.

Conclusion To meet the OP as closely as possible while fulfilling safety and privacy considerations, we'd need a version of the plugin library similar to Jeds variant (i.e that doesn't require a special server site) - and that this is necessarily included in standard distro along with a user setting to actively make tiddlers or the whole wiki fetchable.

@Jermolene - If someone implemented this, and it was well coded, would it be accepted for the standard distro? I'd appreciate a "no" more than no answer so that I can just kill this unicorn that I've fantasized about for years. I'm also willing to pay anyone capable of doing it if I knew it actually led to inclusion in standard distro because I think it would bring tremendous value to the whole project.

[Jermolene]

Hi @twMat we can't commit to adding something to the core when it is such early days. To get this stuff added to the core we need a compelling use case, and hopefully @inmysocks work gives us a platform to explore those use cases.

I'd appreciate a "no" more than no answer so that I can just kill this unicorn that I've fantasized about for years. I'm also willing to pay anyone capable of doing it if I knew it actually led to inclusion in standard distro because I think it would bring tremendous value to the whole project.

As I said above, I believe that we already have all the components needed to explore this space. So the steps might be:

1. Get the TWederation primitives exposed/documented so that end users like @twMat can experiment with them
2. Discuss the experiments
3. Form a plan for going forwards

The key thing is to empower @twMat to explore this space for themselves.

[twMat]

( @Jermolene - much appreciated that you replied so fast.)

Hi @twMat we can't commit to adding something to the core when it is such early days. To get this stuff added to the core we need a compelling use case, and hopefully @inmysocks work gives us a platform to explore those use cases.

If "compelling use cases" is all that is needed then that's very good news - but is that really it? And if "sharing tiddlers" is not specific enough in itself, then how about "collaboratively writing docs without requiring a special server"? Or what would be a hypothetical example of a "compelling use case" in your opinion?

@inmysocks has, unless I misunderstand, shifted focus and is no longer particularly interested in the twederation "wiki-to-wiki" version but only things involving servers. (I hope I'm not misrepresenting you Jed).

The key thing is to empower @twMat to explore this space for themselves.

A flattering suggestion but I of course can't expect anyone to do that.

[Jermolene]

Or what would be a hypothetical example of a "compelling use case" in your opinion?

Yes, I mean use cases that have been tried and tested through experimentation, and achieved some adoption within the community.

One of the problems we faced with the TWederation work was that users struggle to understand the principles of what is going on, and thus struggle with the user interface.

A flattering suggestion but I of course can't expect anyone to do that.

Not at all, one of my main aims with TW5 is to empower non-developers to do the stuff that is normally the domain of experienced developers.

[twMat]

achieved some adoption within the community.

A catch 22 here, kind of like when the telephone was invented; nobody to call to. But would a poll, detailing the workflow and with illustrated or mockup UI, with enough people expressing their likelihood of using it, be enough indication for you?

users struggle to understand the principles of what is going on, and thus struggle with the user interface.

UI... absolutely critical for mass adoption. Everyone is used to extremely slick UIs from commercial products so if things are not polished enough they (we!) barely bother to try things out.

[AnthonyMuscio]

To support mats effort as a superuser if someone could provide a scope and overview of prior tw federation work I would be happy to ccontribute. As far as I can see there is no overview info anywhere especialy para technical.

I would like us to make use of it.

My focus has always being single file wikis because they are the easiest to publish.

I already have a method for a loosly coupled network of wikis and I am approaching a type of federation on single file wiki php servers.

I think the discovery component, locating and identifying what one wants should be put to the side because I think it complicates the issue. We just need reliable mechanisiums to get or publish content of different kinds, including preconfigured silent get or publish and the discovery just becomes a matter of sharing and curating files, urls and filenames, with an odd filter thrown in.

For example if I choose to publish some content I will tell you how I made it easy for you. Eg here is a url and related filter. It does not need to be global.

To me the key issue remaining is "preconfigured silent get and publish" otherwise the technology is already available. On php I have the click to get or publish mechanisium worked out, which is most of the way.

To me the rest is just wikitext and design, and in time published practices and standards.

Regards Tony

[inmysocks]

@twMat I am still very much interested in the wiki-to-wiki communication, I have just been focusing on node wikis lately because the single file wikis have a rather low ceiling on the technology side and I have already made a system that lets you fetch tiddlers from other wikis. It can certainly be improved, but I don't think that there is much more that is possible on the back end with something that just requires a static file server.

@AnthonyMuscio I made a simple daemon that runs in a wiki that allows a wiki to be loaded in an iframe and communicate with the containing page and an action widget that loads a remote wiki inside an iframe and using the daemon running in the iframed wiki fetches a set of tiddlers based on a filter. The system just requires that the remote wiki is accessible on a static file server and no other components or server architecture. It is a relatively straightforward extension of what @Jermolene made for the plugin library and works on almost exactly the same principles.

I don't remember much about the state of the plugin at the moment, it is on GitHub here https://github.com/inmysocks/TW5-TWederation the important parts are the federation-core which adds the listening daemon to a wiki so that things can be fetched from it and federation-tools that allow you to fetch from remote wikis.

If you are going to make the requirement a php server than more sophisticated communication is possible, but that makes it so that you can't host connected wikis in places like GitHub or dropbox.

[AnthonyMuscio]

Thanks Jed

And mat Unfortunatly I am used to downloading released software from live wikis. Can I download all files and drag or import them? From Jeds giihub?

On this mechanisium I have used iframes of other wikis to simply drag and drop either way. This was the basis of my loosely coupled tiddlywiki networking.

If I may summerise my initial understanding;

tw-federation aims to codify this interaction between the two wikis, one in an iframe of the other?

If I may guess, you have made progress on the technical solution for federation but there is possibly a thick layer of process, practice and icing needed on top to make it accessible to a wider audience.

Please set me strait if I am wrong.

Regards Tony

[twMat]

@inmysocks that is great to hear! I sent an email just yesterday asking how you feel about what Jeremy suggests here above, i.e to "Get the TWederation primitives exposed/documented" more? The purpose is to, as Jeremy puts it, get "use cases that have been tried and tested through experimentation, and achieved some adoption within the community." ...

@AnthonyMuscio here are the main bits

• http://twederation.tiddlyspot.com/ - this uses the technology to make a discussion board. As you see, there are some 4 participants who posted but several more made wikis that were connected in the sense that they had the plugin installed and the discussion board wiki had their twCards. You can see these twCards in the Ctrlpanel
• http://twed-face.tiddlyspot.com/ - this is just a rough mockup for a more introductory page to the whole concept of TWederation. It was never used or official but I made it because I felt people mistook TWederation to be the discussion board and also to give a glimpse of many further potential applications. This should also give you some more sense of the concept (note the Tiddlypedia tab).

[twMat]

@AnthonyMuscio

tw-federation aims to codify this interaction between the two wikis, one in an iframe of the other?

Your wiki fetches the source wiki into a local (and hidden) iframe. Because the pulled in wiki has the twederation plugin it can communicate with your surrounding wiki. So your wiki requests some tiddlers from it which it bundles up and, if I understand, it "httprequestmessages" it over (...some term like that ;-) ) Your wiki then, using the same plugin, unbundles the tiddlers.

@inmysocks - is the actual bundling is necessary in this?

If I may guess, you have made progress on the technical solution for federation but there is possibly a thick layer of process, practice and icing needed on top to make it accessible to a wider audience.

As you see in my post just before this one there is a real user application. At least a prototype. There are a few problems with it:

1. It is slow. When you use it you really import all the connected wikis and perform the operations on them. The idea is/was therefore to also have servers roam wikis to batch fetch tiddlers so that the user can more smoothly pick up copies from the server. However, a problem with this is that the server should not be used for hosting spam.
2. The UI is not good enough yet and because it is a prototype it also caters to us as developers of it.
3. The limitation to only be able to fetch/import/pull is surprisingly hard to grasp, i.e that you cannot push things as one normally can in communication. E.g if I write a message to someone... I'll patiently have to hope he picks it up.
4. As noted, people probably mistook the discussion board to be TWederation.

On a more general note; the life of a plugin that is not part of the standard distro is directly correlated to discussions about it on the forum. I suspect most plugins pop up, people say "Neat! I'll use it when I need it" and then it's forgotten. This is particularly tough for plugins that totally depends on several users having it installed, or even using it, at the same time for it to be useful. This is why an eventual solution for "tiddler transfer" would, IMO, have to be in the standard distro. Kind of like a telephone infrastructure.

[AnthonyMuscio]

Thanks I am much better informed.

Now I have the big picture and insight into some of the details I would like to share my observation which is most likely based on ingnorance but may throw light on the perceptions of others note yet engaged in the concepts.

Tw federation seems in some way to be a forward thinking and expancive solution based on the insight of very expirenced people. Whats being done demonstrates this is possible and how far it could go. Yet I think it may be ahead of its time, not in the world but ahead of its time in the tiddlyverse. This means a leap to tw federation looks more like jumping a large chasm, thus I think the ability for many to start and dabble with it, is not only a big step but into an undeveloped ecosystem. That is the main motivation to adopt tw federation is the one thing it cant do yet, which is to join an ecosystem.

I for one can see the vision and love it, but there is no ecosystem for me to join. Just a demonstration of what may be possible. This vision is not what brought me into this discussion but my own current needs. I need to increase the communication between wikis.

Networks are built from a one to one links. In this case one > one and one < one links unidirectional.

Could I suggest we focus first on promoting tw federation as a wiki to wiki solution for superuses to link their own wikis. This can then mature and become a commodity for most users. Multi wiki federation will occur on lans and on peoples desktops before the trickier internet or server implementations. People will naturaly start to look further and drink the cool aid. Some will join the band wagon and the full federation will start to be demanded. Only then will we have the community support to really federate and buy into the vision.

Help me keep my plugins uptodate, collect comments from a hosted wiki, distribute content to a set of my own wikis. Thats how I think it gets started

The beauty of the federation concept is that it will guide the development of wiki to wiki links in such a way this will develope into the vision of federation.

Networks that do not grow organically either die or someone has to do all the work before even early adopters can buy in.

Regards Tony

[AnthonyMuscio]

Mat.

I see where you are comming from with the standard distro but I think it unnessasary. For two reasons

If I have to install a magical plugin to allow my wiki to join a network then thats what I do.

If you want to promote open access to the content in other wikis it is already possible and we can discuss how. Though I want some tools to support this I cant yet build.

Inmysocks In some ways my reference to php is only so far as I can publish content from my file wiki to a location on a server which is accessible. The server implementations can already do this. It a kind of selective publishing. If I can trigger this as well as trigger a load from such resources I may have all I need.

Thanks for the work in tw federation so far.

[twMat]

@AnthonyMuscio

a leap to tw federation looks more like jumping a large chasm

IMO, overcoming this is mostly a UI matter.

Could I suggest we focus first on promoting tw federation as a wiki to wiki solution for superuses to link their own wikis.

Well, first there has to be something to promote.

If I have to install a magical plugin to allow my wiki to join a network then thats what I do.

Well, the problem is that my tiddlers and plugins - available as they are if you physically visit my site - are not available to you unless I install this plugin. Instead I propose to have this natively but to very easily publish tiddlers (like in TWederation; you have a toggle button in the tiddler toolbar to publish - i.e "make fetchable" - or not).

The "feature" to be able to share/communicate is too fundamental to not be in the standard distro. IMO we don't see this only because we're so used to the current setup - but we're stuck in the past! The rest of the world are even about to leave web 2.0 and we haven't even come to it! Imagine if, say, google docs demanded a manually installed plugin on all parts to enable collaboratively editing documents! Ugh!

[inmysocks]

@AnthonyMuscio both of those things already exist. The saving part is discussed at length in different places and fetching from a wiki is the basic function of twederation.

The framework for this has been available to advanced users for over 3 years and next to no one has even looked at it. I used it for a long time to have a personal library of tiddlers and plugins using local wikis on my computer. Then I made Bob which made it unnecessary for fetching things locally.

[joshuafontany]

This is a great discussion, and the concepts discussed have a great deal of potential. I am keeping my eye on this. But, like everything else, that requires some "free time", LOL. Still, I can definitely see where this is pointing, and I like it a lot.

Thanks, Josh F

On Tue, Jul 16, 2019 at 7:02 AM jed notifications@github.com wrote:

@AnthonyMuscio https://github.com/AnthonyMuscio both of those things already exist. The saving part is discussed at length in different places and fetching from a wiki is the basic function of twederation.

The framework for this has been available to advanced users for over 3 years and next to no one has even looked at it. I used it for a long time to have a personal library of tiddlers and plugins using local wikis on my computer. Then I made Bob which made it unnecessary for fetching things locally.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Jermolene/TiddlyWiki5/issues/4105?email_source=notifications&email_token=ACFOMB4PLVAX74QC4PTHFQ3P7XIG3A5CNFSM4IDL6IH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2A6LBY#issuecomment-511829383, or mute the thread https://github.com/notifications/unsubscribe-auth/ACFOMB6JALAWTJTZU32MPMDP7XIG3ANCNFSM4IDL6IHQ .

[inmysocks]

If anyone is interested in what I have already made, I put the working core I made a few years ago into its own repo and added what may be a usable readme.

I am working on writing up some things about how it works.

https://github.com/OokTech/TW5-FederationCore

[twMat]

@inmysocks - this is terrific! I'm posting some issues there. Is it appropriate to make requests and initiate "potentially not short and actionable" discussions there also, instead of via email?

[AnthonyMuscio]

Thanks jed

My level of expertise although still shaky seems to be at a level I may be able to make use of this now. I can raise an issue if you want but I am keen eventualy to fetch content silently. Like between two wikis I trust. Could there be a way for us to share a secret or something between wikis such thay when they match they are unpacked so they are usable in the local wiki. I expect to still rely on the safety mechanisium normaly just not in trusted interwiki fetches.

The last thing I want to do is teach incidental users of a wiki I publish about fetching. Or force a manual intervention on my self if at all possible.

Thanks Tony

[inmysocks]

@AnthonyMuscio adding other behaviour on top of this is possible, but the FederationCore plugin is just going to be the very basic components to build on top of. It is set up so people can make plugins that add other behaviours like having the fetching being done silently, but things like that aren't going into this plugin.

[inmysocks]

The plugin at https://github.com/OokTech/TW5-FederationCore should be ready for general testing. It hasn't had anything remotely like in-depth testing so there are almost certainly bugs, but it has worked for the simple things I have tried so far. It also has what may be a reasonable readme and two demos for fetching and how the bundling works.

You can also drag and drop it from here https://ooktech.xyz:8443/Public

[AnthonyMuscio]

A Quick reference back to the early part of this discussion @Jermolene said

The "server" side implementation lives in a special standalone HTML app that implements the plugin library:

https://github.com/Jermolene/TiddlyWiki5/blob/master/plugins/tiddlywiki/pluginlibrary/libraryserver.js

What if we created an exporter driven by a filter that could export the special standalone HTML app to a file with a tiddler payload for people to publish libraries with?

[Jermolene]

What if we created an exporter driven by a filter that could export the special standalone HTML app to a file with a tiddler payload for people to publish libraries with?

Jed's work has given standalone TW HTML files the ability to directly act as the "server" side implementation which is I think equivalent. Any number of tiddlers can be exported to such a TW and imported by users via the plugin library.

[AnthonyMuscio]

The idea was if we are to maintain the library mechanism that we provide a method to generate them.

Where within Jed opus of work are you referring to here please Jeremy? Do you mean tw-federation? because that is catering for a more advance functionality.

TW HTML files the ability to directly act as the "server" side Is not actually the same. If I have a large tiddlywiki that documents multiple sources of content I would want to publish a link to a library containing (all and no more of) the tiddlers for a particular purpose. Forcing reference to the whole wiki may present performance issues.

Perhaps I can build my suggested solution? Does https://github.com/Jermolene/TiddlyWiki5/blob/master/plugins/tiddlywiki/pluginlibrary/libraryserver.js use the recipes/library/tiddlers.json file to access the "payload" tiddlers? I presume I need to change references to recipes/library/ to the appropriate location in my implementation.

[AnthonyMuscio]

Perhaps we even have a library edition? and just add payload tiddlers.

[Jermolene]

Hi @AnthonyMuscio I was indeed referring to @inmysocks's work on Twederation. It's not catering for more advanced functionality; it's the components making up the plugin library upgraded into general purpose things. If you want to experiment with the technologies of the plugin library then I'd be inclined to start with @inmysocks work because it's already there.

[inmysocks]

To make a plugin library using the FederationCore plugin you put the wiki somewhere accessible and put the FederationCore plugin into it along with whatever plugins you want to make available. As you add more plugins to the wiki, or modify existing plugins, the new or modified plugins will be available for fetching.

Then the fetch demo in the plugin shows a very minimal UI for how to fetch remote things, since the core is just the minimum required parts it doesn't have much else.

It is built to be modular so you can create additional plugins that build on top of it extend the functionality.

The only thing past the minimum required to make the fetching work is a bundling and unbundling widget because sending raw tiddlers that aren't packaged/sandboxed somehow is in general rather dangerous. But due to repeated requests I made a thing that does exactly that in the FederationExtended plugin. At the moment that is all that the extended plugin adds.