Use container as index file

[bourgeoa]

I made a test building an index.json with only title and date modified skinnytiddlers works without any problems. I took these from the container. So no need for an index except if you want to load the tags without loading the tiddlers

[linonetwo]

The question is, is title enough to decide whether TW needs to reload a tiddler?

If you change some tiddler in another device, you may want to reload it depends on some other metadata.

But I agree that container as index file is enough since we can get "modified" from the container, look at https://groups.google.com/forum/#!topic/tiddlywikidev/mgiYWXRy0_E , seems if you only return title field, TW will use it to determining whether tiddler has changed in the backend, but if you also return a modified(Date) field, it will use it too.

[linonetwo]

But load more metadata is not so hard, we can store tiddler fields inside xxx.meta files, and:

1. only load title from the container on the startup.
2. load all metadata use globbing on the second getSkinnyTiddlers(), maybe these metadata will be used in searching?
3. only load title and modified from the container on subsequent getSkinnyTiddlers(), only for TW to determine whether to trigger loadTiddler().

FYI to use globbing:

[linonetwo]

There are two problems:

1. "modified" field from the server is the time POST reaches the server, but in the client side, it's the time user click save button. So "modified" in the client side will be earlier than that on the server side.
2. "title" contains slash "/", we can't store it directly to the server, we may need to replace "/" with some other characters.

@bourgeoa how did you dealt with that?

[bourgeoa]

I don't have any / in the title. Title is the name of the tiddler and the name of the tiddler file without extension.

I have decided to keep the index file for the time being because when I load the skinnytiddlers without the tags these are not loaded through the load tiddler function.

[bourgeoa]

I have made a RDF .tll version functional. Just need to test the migration function.

We must agree to a final commun ontology.

There is a small problem with the "Draft of 'New tiddler'" that creates 2 vocab draft.of and draft.title. A dot character being forbidden in ontologies I have converted the dot (draft.) by underscore (draft_).

[linonetwo]

Can you reuse some of the existed vocabularies for the draft? Here are some of them https://lov.linkeddata.es/dataset/lov/terms?q=draft

But I dont understand how you are going to use draft.of and draft.title, what is the subject and object?

[linonetwo]

Maybe using globbing?

[linonetwo]

I'm trying to name all metafile (each contains skinny fields for a tiddler) with pattern .meta.xxxtiddlername.ttl, so I can globbing all .meta.* so GET them in a single file.

https://github.com/solid/solid-spec/blob/master/api-rest.md#globbing-inlining-on-get

Ideally, I can globbing xxxtiddlername.meta, but seems globbing can't work in that way.

[linonetwo]

Actually, just after startup, saveTiddeler() will try to save $:/StoryList, so I'm asking how you handle / in the title.

[linonetwo]

Some tiddlers don't have the tags, will load tiddler function refused to load them? What did you mean by this?

[linonetwo]

You can see how node-tiddler handles filename, I think reading all fields from metafiles (through globbing) is a good way to avoid trouble.

[bourgeoa]

I never store $:/StoryList and I think any so called system files ($:/......).

For the filename only I use encodeUriComponent(title).

[linonetwo]

Why not? I think some plugin may store some config in the $:/ tiddlers, and maybe user want to store their configs?

[linonetwo]

BTW, I'm trying to use https://github.com/pasqLisena/rdf-translator to translate between JSON and turtle. But it's very slow.

[bourgeoa]

For draft.of and draft.title they are used as a temporarily duplicate during tiddler edit. As such I considered not using a specific ontology for them but the default one (https://example.org/vocab# or any other name).

Do you know if there is something in RDF spec on the name to use for this default non exist ontology ?

[linonetwo]

Do you know if there is something in RDF spec on the name to use for this default non exist ontology ?

I have no idea, I think it just required to resolve, so a human can read the definition. Maybe you can use https://tiddlywiki.com/static/DraftMechanism#of and https://tiddlywiki.com/static/DraftMechanism#title.

[linonetwo]

I'm using the context you suggested like this:

https://github.com/linonetwo/solid-tiddlywiki-syncadaptor/blob/01c38f661c943ba409bfc4e41891566b1a765350/src/SyncAdaptor.js#L15-L34

https://github.com/linonetwo/solid-tiddlywiki-syncadaptor/blob/01c38f661c943ba409bfc4e41891566b1a765350/src/SyncAdaptor.js#L117-L124 https://github.com/linonetwo/solid-tiddlywiki-syncadaptor/blob/01c38f661c943ba409bfc4e41891566b1a765350/src/SyncAdaptor.js#L168-L171

[linonetwo]

What do you mean by:

when I load the skinnytiddlers without the tags these are not loaded through the load tiddler function.

[bourgeoa]

It was in the specific test where I used an index file containing only the title. I do not change anything to the toddler they remained with all there fields. The skinnytiddlers loaded the list of tiddlers as having no tags. Quite normal. Then I manually select and load a tiddler that has tags. These tags did not show in the tiddlywiki, neither in the tiddler nor in the tag list.

I did not further explore and decided that skinnytiddlers needed all tiddlerfields except the text field. As I do not load all tiddlers at startup but only on demand, I keep an index file for performance.

[linonetwo]

I'm pausing my development waiting issues to resolve:

https://github.com/digitalbazaar/jsonld.js/issues/314 https://github.com/solid/node-solid-server/issues/1186

[linonetwo]

How will you name your index file? index.ttl? What if a user creates a file named index.ttl?

I'd consider using ./meta/xxx.ttl to store metadata for ./xxx.xx, just store all metadata in a sub-container, with the same name as the file we want to describe, and use globbing to GET them at once. If not using globbing, I'd use ./meta/index.ttl, I'm not sure if this is a good name.

[bourgeoa]

This index is a problem for interoperability because :

• it is not a tiddlywiki spec
• if we use different spec, : -- it will not be updated creating discrepancy -- if we rebuild it we will not know that the fileindex stays up-to-date.

Not taking in consideration that .meta and blob don't seem to be firmly in Solid spec.

I am beginning to think we need to build the skinnytiddlers directly from the tiddlers. Each app choosing his way to do it (reading all files, building simplified skinnytiddlers with LDFLEX, ...).

If the performance is too bad we could look at an other way that would be to have the index reflect the tiddlywiki internal JSON format as I have know. The filename of the index will be a param that will be stored in Solid pod. For the time being it is named __lndex__.json. (I kept it from the app I used as model)

[linonetwo]

I was on some other project since May. I'm going to try to finish solid-tiddlywiki project in June now.

to build the skinnytiddlers directly from the tiddlers

Yes, that would be the best way. But the problem is, NSS doesn't support cross-file SPARQL now, so if do so, there will be O(n) SPARQL to every tiddler. I will try doing so on an HTTP2 connection to see whether this can be a problem.

[linonetwo]

I will work on https://github.com/linonetwo/solid-tiddlywiki-syncadaptor/issues/3 PWA support first, so every tiddler can be store locally, doing sparql on them will be much more faster.

One thing I need to know is how to do fast diff with server, so it only loads changed tiddlers on the startup.

[linonetwo]

Life will be much easier if NSS supports multiple file SPARQL.

[linonetwo]

Bugs in NSS seem to be fixed, but there are several new bugs in solid-auth-client, pausing until they are fixed:

https://github.com/solid/node-solid-server/issues/1231

[linonetwo]

We can use FootPrint! https://www.w3.org/ns/footprints https://ruben.verborgh.org/blog/2019/06/17/shaping-linked-data-apps/#future-shapes

[linonetwo]

Now I'm using xxx.metadata to store tiddler metadata, and when saving each tiddler xxx, I store an xxx.metadata. When loading, I load two files at the same time.

Not using xxx.meta because NSS is buggy dealing with xxx.meta.