Rethinking data tiddlers

[Jermolene]

We have discussed several times my view that data tiddlers were a mistake in the design of TiddlyWiki 5. Here's the best quote I could find, from 2019:

As you may recall, my view on data tiddlers has evolved over the years, and I broadly no longer think they were a good idea. The present implementation is limited, and would take a lot of code and complexity to make it universal. The implementation of data tiddlers is spread throughout the core code, and hard to work with. I think that the whole idea was misguided: it was an attempt to avoid the proliferation of tiddlers, but I now think that that is a non-goal.

Against that background, I think of data tiddlers as a candidate for deprecation in v5.2.x, and so I'm reluctant to expand support at this point. It looks like this operator could be delivered as a plugin, which I think is what I would advise at this point, although I'm happy to discuss.

In the light of experience, I have now refined my position. My earlier statements tended to conflate a number of separate features: in particular, the usage of application/json tiddlers and the {{title##index}} syntax.

It's actually the transclusion syntax that is the problem. I had originally intending to expand the syntax so that one could address arbitrary JSON properties such as {{title##0.foo.bar.3}}, and it's that work that would have been challenging. It would also have been a bit of a dead end because it only allows for accessing properties whose index is known in advance.

So, this note is intended to convey a new proposition: that we should deprecate the indexed transclusion syntax, and associated widget-level implementation (ie the index attribute of widgets that permit access to data tiddlers such as transclude, view).

I intend to shortly submit a PR for a set of filter operators that allows for complete read access to JSON data, including the use of computed indexes.

[Jermolene]

Tagging @twMat @kookma @saqimtiaz @pmario @AnthonyMuscio @sycom who I think have all been involved in earlier discussions on this topic.

[twMat]

@Jermolene thank you for the heads up and the clarification.

For any implementation related to this, it may be worth reviewing the ideas from several people expressed in this discussion.

[Jermolene]

I've posted a PR for the new filter operators at #6522

[pmario]

In the discussion @twMat mentioned I wrote: https://github.com/Jermolene/TiddlyWiki5/discussions/5634#discussioncomment-655363

I think we should define what are the properties that make datatiddlers useful and "easy to use". Then we can think about a "replacement that has the same properties" and more.

• using datatiddlers "feels" easy / good
  • If you export / import a datatiddler you can be sure "you get it all"
• can be edited with the default text editor
• Copy / Paste is possible
• 1 tiddler can transport many settings
• easy to store log entries / time-line
• View and edit workflow is fast
• can be made "self sufficient"
  • no meta data needed eg: no blocklists
• is a very basic "key : value" store
  • "Key - Value" store is a "buzz word" ...

That are my minimal properties that need to be satisfied. Or they should better be excelled!

I think none of these points are changed using data-tiddlers with JSON content. So from my point of view, it's only an extension to access structured JSON data-tiddlers.

For me it seems to be designed for a special usecase.

[Jermolene]

Thanks @pmario

For me it seems to be designed for a special usecase.

What is the "it" referred to in this sentence?

[pmario]

What is the "it" referred to in this sentence?

Uuups I had the PR in mind https://github.com/Jermolene/TiddlyWiki5/pull/6522

[Alzacon]

I agree with @pmario.

When I think in datatiddler I have in mind the palettes, where we can see how powerful it is: easy/good to write/read. We need a alternative to old (basic) data tiddler and we also need to facilitate the use of json. Maybe something like YAML could be useful, but in palettes we will have a problem because # is used for comments.

[Jermolene]

I agree with @pmario.

When I think in datatiddler I have in mind the palettes, where we can see how powerful it is: easy/good to write/read.

Right – palettes are actually dictionary tiddlers, a particular type of data tiddler (the other type being JSON tiddlers).

We need a alternative to old (basic) data tiddler and we also need to facilitate the use of json.

What alternative to the dictionary tiddler format were you thinking of? The major shortcoming of dictionary tiddlers is that they only support a flat list of name: value pairs.

Maybe something like YAML could be useful, but in palettes we will have a problem because # is used for comments.

YAML isn't practical for the core because it requires a substantial JS library to be able to process it.

[Jermolene]

I think none of these points are changed using data-tiddlers with JSON content. So from my point of view, it's only an extension to access structured JSON data-tiddlers.

What does "it" refer to?

If it is the PR #6522, that is precisely what the PR is intended to accomplish.

If it is the point raised in the OP here, then I don't understand.

For me it seems to be designed for a special usecase.

I think you're referring to #6522? Why would providing generic access to JSON data constitute a special use case?

[Alzacon]

Right – palettes are actually dictionary tiddlers, a particular type of data tiddler (the other type being JSON tiddlers).

Mmm I think now I understand it now. So the idea is maintain the same data tiddlers (dictionary and JSON) but only change the implementation to works only as filters, am I wrong? (Here is the reason I was talking about hypotetical alternative to dictionary tiddler with something like YAML)

There is a point about the filter for data tiddlers, we have a filter opertaror for dictionary tiddlers. And now, with de PR #6522 , we will have other for JSON tiddlers. Perhaps it would not be better to unify them?

Something like the following will be intuitive for me but you can found something better. (in my basic level of English) dataget[key] returns the value And if we will want to differenciate the origin of data could use a suffix for JSON tiddlers dataget:json[key]

datakeys or dataindexes as operator for return the indexes/keys, also it could have the suffix :json

[Jermolene]

Hi @Alzacon because of our backwards compatibility policies, we wouldn't remove any existing functionality. The proposal at this point is just that we add these additional operators for working with data tiddlers (the new operators will work with both JSON tiddlers and dictionary tiddlers).

The reason that the new operators are separate from the existing getindex and indexes operators is that they both have unexpected semantics that wouldn't work in the generic case. (For example, the getindex operator ignores blank string values). We may want to deprecate the old operators (which means marking them as only being maintained for old code, with a recommendation for an alternative for new code).

[Alzacon]

Hi @Jermolene . I didn't have in mind to remove the old operators. I was talked about the new funcionality for working with both type of data tiddlers. In the PR the filter has in json in the name because it is a PR for JSON tiddler. Here is the reason for my comment for the unify with a common reference to both types of data tiddler, as I commented data... will be a intuitive name for me.

[Jermolene]

Thanks @Alzacon that makes sense, I made a comment over at #6522

[pmario]

In the light of Parameterised transclusions #6666 we may review this issue again.

[AnthonyMuscio]

Some important perspectives should we "review this issue again".

• Data tiddlers are also representations of date that is on its way out (export) or on its way in (import) regardless if the preference is for content to be in tiddlers.
  • What ever the format we choose all the better if it is similar if not identical to one or more standard data formats.
• There are code patterns that demand large volumes of data for which independent tiddlers is overkill such as a word dictionary and creating tiddlers overloads the tiddler name space.
• The JSON format is both native to tiddlywiki and powerful, and recent versions have reduced the gaps but I think we need a thorough review to ensure the minimum feature set is complete.
  • and as easy to use as possible for different skill levels.
  • JSON Mangler introduces handling for application/csv we should do this and add tools for column include/exclude/filter/sort