Core support for UUIDs

[tobibeer]

It's just essential for a wide range of projects that simply don't do well with title references only... including TiddlyMap.

So, let us provide a core plugin that, when activated...

• inits a uuid field at every new tiddler to a basic, good-enough, unique identifier, hidden from view
• allows to specify a filter for all tiddlers that are to receive such a uuid
• allows to batch-set that field for existing tiddlers matching that filter

@giffmex pointed at Workflowy and after some initial consideration it feels like remodeling it in TW would work quite poorly on tiddler titles alone, especially when wanting to rearrange things, i.e. rename items / dnd elsewhere. Working with namespaces alone would not do good here at all.

Here's some basic plugin for TW2 I once created together with @pmario that could provide a basis...

https://github.com/tobibeer/TiddlyWikiPlugins/blob/master/plugins/IdPlugin.js

[felixhayashi]

I always wanted to outsource TiddlyMap's id mechanism in form of a plugin but haven't found the time.

In addition to @tobibeer's listed plugin functionality:

1) Ids are present from the beginning

The plugin needs to make sure that tiddlers posses uuids right after startup so any module can be blindly expect that uuids are set on every tiddler (that matches the "filter").

TiddlyMap assigns id's to all Tiddlers before the rootwidget is instantiated

exports.after = [ "startup" ];
exports.before = [ "rootwidget" ];

https://github.com/felixhayashi/TW5-TiddlyMap/blob/master/src/plugins/felixhayashi/tiddlymap/startup.caretaker.js#L229

2) Id assignment for new tiddlers

The plugin needs check on every refresh whether a tiddler has an id assigned

TiddlyMap does this here https://github.com/felixhayashi/TW5-TiddlyMap/blob/master/src/plugins/felixhayashi/tiddlymap/startup.caretaker.js#L525

3) Id lookup table

The plugin needs to keep an index of all ids so fast lookups of tiddlers by id is possible

TiddlyMap maintains an id index where every time a tiddler is added, the id is registered (https://github.com/felixhayashi/TW5-TiddlyMap/blob/master/src/plugins/felixhayashi/tiddlymap/startup.caretaker.js#L206).

4) Dublicate detection

The plugin needs to make sure no id dublicates occur in a wiki

TiddlyMap checks on every modified tiddler, whether the id exists twice in the id lookup table. If it detects a dublicate it changes the id, inits a dialog that informs the user with an option to hide the dialog in the future.

https://github.com/felixhayashi/TW5-TiddlyMap/blob/master/src/plugins/felixhayashi/tiddlymap/startup.caretaker.js#L360

5) A function for id assignmend

Sometimes ids need to be assigned between refresh cycles already. In this case a function needs to exist that adds an id (if not present yet) to a tiddler

TiddlyMap has such a function here: https://github.com/felixhayashi/TW5-TiddlyMap/blob/master/src/plugins/felixhayashi/tiddlymap/lib.adapter.js#L1214

[Jermolene]

I appreciate the requirement for UUIDs, but I want to broaden the exploration to meet the slightly more general requirement of using non-human readable unique IDs (of which UUIDs would be a special case). An example of a scenario where non-UUID IDs might be useful is if one wanted to use the unique IDs established by another system.

Retrofitting the required functionality into the existing model of the store is quite difficult: the semantics of the store include the guarantee that calls to create, delete or modify tiddlers will always succeed, and the tiddler will be written as specified.

The way that I am interested to explore handling the requirement is to use the existing title field for the ID, and a new field for the human readable title for the tiddler, such as display-title. The uniqueness constraint thus gets satisfied automatically by the core.

Other changes we would need:

• The ability to generate an ID when new tiddlers are created
• Changes to many templates to use the display-title over the title, if it exists
• Changes to the link and transclude widgets to identify the target by the display-title field instead of the title (this is related to the requirement for aliases)
• I'd specifically like to avoid making the store enforce constraints such as the unique ID being a valid GUID, because it would break the semantics above

[twMat]

I assume UUID/GUID still only refer to the current TW, but might this be an opportunity to consider Tiddlyverse ID's? (TVID!) While desirable in an eventual Federation, I think already now something like @erwanm 's search engines would benefit from this?

[Jermolene]

@twMat - one of the attractive properties of GUIDs is that they are globally unique - if you and I both generate a GUID at the same time (or indeed at different times), they are guaranteed not to clash, for all practical purposes.

https://en.wikipedia.org/wiki/Globally_unique_identifier

[Jermolene]

By the way, that same property of GUIDs means, oddly, that it's not terribly important to have a mechanism to enforce uniqueness, since they are guaranteed to be unique on creation.

[erwanm]

@twMat technically the aggregator doesn't need UUIDs from the source wikis, because it can rename the collected tiddlers when they are extracted to avoid any conflict: it simply buids a new title using the source wiki id as prefix. but thank you for your concern! ;)

[tobibeer]

An example of a scenario where non-UUID IDs might be useful is if one wanted to use the unique IDs established by another system.

@Jermolene, not exactly sure there is much to do at the TW end. We simply import them as a given field, e.g. id-foo: 3j5h32k45g23j4g and then use filters or other means to access them / do lookups.

What I want to avoid in the beginning is to try and figure out all the possible ways in which such ids may or may not be beneficial to the core... or which widgets / wiki markup could work on what ids. I think the basic id functionality alone is the first, most important step.

@felixhayashi

1) Ids are present from the beginning

I don't think that is needed, they can be very much created on-demand.

2) Id assignment for new tiddlers

Also not needed, a plugin or other code can ensure that as needed.

3) Id lookup table

That would be beneficial for any explicit id[] filter or respectively id:uuid[] filter... the latter specifying an explicit field and thus distinct hash-table that would be different for id:external-id[].

4) Duplicate detection

That should definitely not be needed, otherwise we'd have really poor ids. Also, what to do with clashes? Nah, we just make sure ids are unique.

5) A function for id assignment

I think it's up to every plugin implementation how to handle, but the wiki object could have specific helper functions.

[felixhayashi]

@tobibeer

1) Ids are present from the beginning

I don't think that is needed, they can be very much created on-demand.

What do you mean by on-demand? That tiddlywiki assigns an id as soon as a tiddler is accessed? How would you e.g. build an index if not all tiddlers can be mapped to an id?

2) Id assignment for new tiddlers

Also not needed, a plugin or other code can ensure that as needed.

See above.

3) Id lookup table

That would be beneficial for any explicit |id[]| filter or respectively |id:uuid[]| filter... the latter specifying an explicit field and thus distinct hash-table that would be different for |id:external-id[]|.

This would be necessary as otherwise it would be tremendously slow to get tiddlers by ids as ids are only attributes (fields) and need to be searched for.

4) Duplicate detection

That should definitely not be needed, otherwise we'd have really poor ids. Also, what to do with clashes? Nah, we just make sure ids are unique.

This is not the point. UUID will be dublicated when cloning a tiddler.

Moreover, when a tiddler is copied to another wiki, the import mechanism needs to assign it another id, otherwise the UUID is dublicated as well.

5) A function for id assignment

If you are pro UUID support, then there should definitely be a helper functions.

@Jermolene

The way that I am interested to explore handling the requirement is to use the existing |title| field for the ID, and a new field for the human readable title for the tiddler, such as |display-title|. The uniqueness constraint thus gets satisfied automatically by the core.

This would be nice, of course. This way tiddlers would survive renaming because only the display-title would be changed. Moreover, if link and transclude widgets show the display-title when this field is present, then it would be a very complete solution. However, who is responsible for the uuid creation. Would TW create this if only a display-title field is filled and the title field is left empty?

[tobibeer]

@felixhayashi,

What do you mean by on-demand? That tiddlywiki assigns an id as soon as a tiddler is accessed?

I mean: as soon as some function explicitly requires an id. I would leave it to plugins to either only deal with tiddlers that have them or otherwise ensure they do (only) as needed. Meaning: Just because we do have a plugin that would operate on uuids, it does not mean that all tiddlers are concerned by it by default or that it would / should not know how to handle any tiddlers that don't.

How would you e.g. build an index if not all tiddlers can be mapped to an id?

As needed. Any tiddlers that have the id will be indexed, all others not... which does not mean they don't exist but only that they have not been touched by any id-based functionality / plugin.

This would be necessary as otherwise it would be tremendously slow to get tiddlers by ids as ids are only attributes (fields) and need to be searched for.

I totally agree. When we create the index is a different concern though, i.e. when first performing any such query.

UUID will be duplicated when cloning a tiddler.

That is true. Cloning should...

• not do anything if the source tiddler had no id
• create a new id if the source tiddler did => which means the duplication needs to know that there is a field of type id

To me, making a uuid mechanism available via the core is an entirely different issue from having a global setting in the control-panel that tells TiddlyWiki to...

[ ] use uuids

...and so it does in all the different places. Does that make sense? Have a look at this simple pull request so as to get what I mean: #2087. It simply provides the very basic functionality to create and use uuids. Whatever the core or some plugin makes of all that later can be decided on an as-needed / -wanted basis, rather than be a prerequisite to get started with unique ids at all.

In other words, TiddlyMap would keep most of its id handling except for the bit of code that does the generating, for now. Later we can see if and how any global settings for using tiddler-ids and the core functions / components that respect those need to be amended to ensure consistency.

Moreover, when a tiddler is copied to another wiki, the import mechanism needs to assign it another id, otherwise the UUID is dublicated as well.

Why should it not be? Why would it (need to) be given a new uuid in the target wiki? I think it should remain untouched, if only for referencing, otherwise we'd need a standard way to turn an imported uuid into some external-id... but then I would not know why we need or even want to do that by default.

If you are pro UUID support, then there should definitely be a helper functions.

See #2087.

@Jermolene

The way that I am interested to explore handling the requirement is to use the existing title field for the ID, and a new field for the human readable title for the tiddler, such as display-title. The uniqueness constraint thus gets satisfied automatically by the core.

I think this is indeed a nice functionality, and could potentially enable TiddlyWiki to implement more readable url slugs, e.g. "Some-Tiddler-Title" instead of "Some%20Tiddler%20Title". On the other hand, seems independent from unique id's for me and I wouldn't see much sense in having a load of "to do", "to do (1)", "to do (2)", etc... tiddlers whose "secondary identifiers" may be unique but we'd not quite gain anything through this form of uniqueness compared to globally unique ids which would open up a wider range of options, also in terms of inter-wiki-operability.

[Jermolene]

However, who is responsible for the uuid creation. Would TW create this if only a display-title field is filled and the title field is left empty?

I'd imagine that the "new tiddler" handling in the navigator widget would be one place that might need to be GUID aware.

On the other hand, seems independent from unique id's for me

The proposal I made was for supporting GUIDs by storing them in the title field. I'm not sure how you see it as independent.

[tobibeer]

The proposal I made was for supporting GUIDs by storing them in the title field. I'm not sure how you see it as independent.

I am aware of your proposal and it makes sense with one of the eventual goals being to allow name duplicates or pretty much to use ids in references like links or list fields rather than "pretty titles", so they survive name changes.

On the other hand, being able to simply use a persistent id in a uuid field even without a capability of duplicate names will already broaden our abilities to keep references in-tact throughout name-changes, even if the core referencing mechanisms via list fields or tiddlylinks will fail. Best example: TiddlyMap, very much builds upon that mechanism.

For me, a simple function that generates a uuid and a macro that retrieves one already broadens our abilities to create workflows or plugins that can establish the kind of reliable references they depend on, even if the core, at that point, won't. This plugin could, in fact, store those uuids to the title field and then devise its own approach to generate its output via some pretty title field.

However, without basic core support for uuid() or <<uuid>>, every plugin that wants to do so will implement its own flavor of how to generate one, which appears undesirable to me, even if we don't have full core support to make tiddlers work on uuid titles... something that appears so complex, in fact a fundamentally different modeling paradigm, so that it has not been approached yet throughout the lifetime of TiddlyWiki, at least not in a visible or comprehensive manner.

So, to take off with and encourage exploring uuids, I don't think the core needs more than basic support for creating them... only then the rest will follow.

[reidgould]

Hello all,

I see this topic is nearly two years old already but as a resolution has not been found, I thought I would share my support for adding unique ids as a core feature and how I've chosen to go about filling the need for my own plugin.

Some of the discussion in this thread proposes deeper integration than my work requires, so I'll address what I believe is the most TiddlyWiki way to use unique ids. The good stuff first:

Shortid plugin and examples of use.

shortid plugin

"shortid plugin" is the shortid package bundled using webpack and exposed in TiddlyWiki as a $tw.utils function and as a macro. The entire plugin including util function, macro, license, and attribution is 9kb.

blc plugin

BLC plugin is a Kanban / Trello style interface for manipulating "tags" and "list" fields in which I use shortids when tiddlers are generated and a custom "shorttext" field in quite similar fashion to the "display-title" field proposed above.

nn plugin

NN Plugin is only a proof of concept I made while still learning the basics while working on blc, but it shows a start down the road to implement a Workflowy / Checkvist style interface.

Shortid in title. (But encourage replacement.)

As I was looking through documentation and community plugins, I found @tobibeer's make filter. It has some great capabilities (making me wonder if it was ever proposed as a core addition) which include the ability to create a uuid and to create a random string of alphanumeric characters. But neither of those I feel address all the needs of unique ids in TiddlyWiki.

Uuids are ideal in their guaranteed uniqueness, but are a poor choice for storing in title fields as they are hard for users to interact with and are less efficient than they could be for string based listops because they use hexadecimal digits which make them very long. Alphanumeric strings are much more approachable for users which is important for ids stored in tiddler titles, but if based on Math.random() are not unique enough to be an adequate unique id.

This is when the shortid package enters. It uses the Crypto web API to generate ids with much better distribution than random, the ids are generated with variable length between 7 and 14 because using string comparison "abc" is a different id than "ab". It uses a URL friendly 64 character alphabet by default a-z, A-z, -, and _. The github ReadMe claims "Can generate any number of ids without duplicates, even millions per day."

The id could be even made more user friendly by using a package like readable-id which appends and English adjective, noun, and a shortid together.

Prefer shorttext field in display.

This solution works for both standard wiki and specialized interfaces like this if the "shorttext" / "display-title" field is handled well. The solution does double duty for heavy and light tiddlers. It allows the user to work with multiple heavy tiddlers (having proper titles) without seeing full styling and media content that may be present in the main body of a wiki entry. It also allows users to work more lightly with tiddlers. Two cases may be: to quickly create new tiddlers one after the other typing whatever stream of consciousness is running through their head without having to stop and come up with titles, and to allow duplication for cases such as a user having two "To Do" tiddlers in unrelated places in the wiki.

I chose the name "shorttext" because this field should be regarded as content, not as a substitute title. Also it is phonetically similar to "plain text" which is the only kind of content that should go into it. The field "caption" is already in use and described in the docs as "The text to be displayed on a tab or button" which isn't accurate to this use case.

In BLC Plugin's interface, only "shorttext" is shown in the view template for cards, falling back to "title" if absent. The plugin also provides a component for the normal view in the main story that displays "shorttext" above the body, both it and "title" visible at once. In BLC's edit template, both "title" and "shorttext" are present, to be available for use on preexisting tiddlers being manipulated and to encourage the user to replace a generated id with something more meaningful if they want. A component for the normal main story edit template is also provided.

It makes sense to me that when a new tiddler is created in the main wiki style interface, that focus be on giving a title first (meaning no need to generate an id), and in productivity style interfaces the focus be on immediate content creation first (necessitating an id to be generated for title). And indeed, when a new tiddler is created in BLC the cursor is positioned in "shorttext" so the user can just start typing.

What we need in the core.

We want specialized interfaces like this to integrate with existing tiddlers and make sense in context, not forgetting that we are working inside of the primary interface which is a wiki! For this reason, I do not feel that the core model needs to change from using title as the primary identifying field.

But, if various specialized interfaces are to be created by different people each solving the id problem in isolation it is very likely that much data will end up trapped, only visible in the respective interface. This would be a failure of the ecosystem. Official support for one kind of unique id and built in handling of a new field like "shorttext" or "display-title" (in main view and edit templates, and also in other prominent locations like list-links and toc-expandable) would allow a precedent to be set for new kinds of interfaces. This would be a solid first step even if the core or TW5.com edition do not actually contain any examples of such a specialized interface.

I think the solution is clear enough that the community will catch on with these minimal changes. To use myself as an example, I've only been aware of and programming with TW for several weeks and arrived at very nearly the same solution discussed here two years ago. If there is interest at this time in taking steps toward including anything discussed here in the main repository, I would feel privileged to get a chance to work on it.

Another question I have is whether there is any code borrowed from other sources already being used anywhere in TiddlyWiki and what @Jermolene's feeling is on that topic.

If you got this far, thanks for reading! and once again for making an awesome piece of software.

[reidgould]

Hello again,

I've just found the PR that references this issue. Excuse my being green with github, I'm still learning to notice the important things.

This satisfies my curiosity expressed previously about the make filter! I do understand the points made in the thread there about the intent of macros to behave as pure functions and that the "now" macro is an outlier (which I used in my plugin before bundling shortid). I do think that there is more value to be gained than simply adding another kind of generator to TiddlyWiki, but if the appropriate place for that is in the world of plugins I'll happily continue to share in the community groups whatever I create.

[pmario]

@reidgould ... Well thought, well written! ... but ... I did identify some problems.

The 3rd-party shortid library you use, uses a license, that doesn't fit the core. According to the TW Contributor License Agreement we need BSD-3-clause for the core.

Your plugin uses the right one. ... The 3rd-party lib does not. ... There's also some problems with the implementation. IMO a simple random UUID needs about 30 lines of code. See my links in a different discussion. ... So we can easily implement it ourselfs.

I did create a playground at tiddlyspace, some time ago. It uses an outdated version, ... but this can be changed.

The most important part for me is ... We should use an industry standard, as described at UUID - wikipedia ...

You are right, they are slightly longer, but they are standardiced. So there are a lot of systems out there, that can use them out of the box. ... also see: TiddlyMap.org

There's no problem to show a shorter version of the id-link. TW has built in mechanisms to achieve that.

... just 2 thoughts. -mario