JSON tiddlers have TID extension

[MidnightLightning]

When using TiddlyWiki via NodeJS, Tiddlers created on the front-end get saved on the back-end as individual files. For tiddlers that are the default text/vnd.tiddlywiki format, they get saved with a .tid file extension (matching the TiddlerFile format.

When saving a tiddler with type text/x-markdown, the tiddler file that gets created has a .md extension, and a separate .md.meta file holds the tiddler metadata. That works wonderfully for editing markdown files in a separate editor.

The bug I ran into is creating a Tiddler with a type of application/json. JSON files do have a standard extension (.json), and I see that $tw.config.contentTypeInfo['application/json'].extension is set to that. However, when creating a new tiddler from the front end with content-type of application/json, the file that gets saved by Node is with a .tid extension instead (with the metadata embedded).

This should be changed, to save as a .json file, to allow easy outside editing, just like the Markdown files.

[joshuafontany]

I agree this would be very helpful.

[joshuafontany]

Oh, I should mention this here because I'm doing other things atm, and can't put together a pull request.

On load (I believe in boot.js, not sure) with both stand-alone and node.js, any plugin json tiddler with an empty or invalid-json text body will cause TW to crash. It should 'fail gracefully', load TW and throw an alert.

This would apply to .json.meta files with no corresponding .json file .

[Jermolene]

Hi @MidnightLightning @joshuafontany thanks for bringing this up. There is an explicit check for the content type application/json in the filesystemadaptor code to force use of .tid file for saving new application/json tiddlers.

The reason it was done that was was that the application/json deserializer code in boot.js will erroneously attempt to treat .json files as JSON tiddlers files.

So, it could be fixed: we'd just need to figure out a way for the deserializer architecture to distinguish between a JSON file with an accompanying .meta file (in which case the file should be treated as a single tiddler in JSON format) versus a JSON file without a metafile (in which case the file should be treated as a JSON tiddlers file).

[MidnightLightning]

What about a different content-type for a multi-tiddler JSON file? TiddlyWiki already uses text/vnd.tiddlywiki as a way to indicate "text, but with TiddlyWiki specifics". Could you use application/vnd.tiddlywiki.tiddlers+json as a content-type to indicate a multi-tiddler JSON file (and associated with a .tld extension)? That would then allow the vanilla application/json to be treated as raw data, with a .json extension?

[Jermolene]

Hi @MidnightLightning we'd have to switch to a different file extension for one or other file type (TW5 infers the content type from the file extension).

I suspect the best fix would be as suggested above: (a) remove the check on saving application/json tiddlers as json and (b) update the deserializer code in boot.js to distinguish between the two types of JSON file based on their structure. The trouble is that (b) is complicated because the deserializers don't have access to the original file (because they run in the browser etc) just the text of the file, and so they don't have any means to retrieve the .meta file.

[MidnightLightning]

Or if adding the logic in the deserializer is "too late" to get at the .meta file, is there a possibility of adding the logic earlier, to whatever process is getting the file off disk to always check and see if there's a .meta file for the file being fetched?

Or, you mentioned first that one or the other would need a different file extension. If leaving the "multi-tiddler JSON file" items as *.tid (and therefore the metadata is right in the file as a header), wouldn't that be enough to distinguish them from files with a .json extension (which must be generic JSON data blobs)?

[Jermolene]

Or if adding the logic in the deserializer is "too late" to get at the .meta file, is there a possibility of adding the logic earlier, to whatever process is getting the file off disk to always check and see if there's a .meta file for the file being fetched?

Yes, somewhere around here:

https://github.com/Jermolene/TiddlyWiki5/blob/master/boot/boot.js#L1619-L1632

Or, you mentioned first that one or the other would need a different file extension. If leaving the "multi-tiddler JSON file" items as *.tid (and therefore the metadata is right in the file as a header), wouldn't that be enough to distinguish them from files with a .json extension (which must be generic JSON data blobs)?

It's too late/expensive to change the use of JSON as the file extension for a bundle of tiddlers.

[joshuafontany]

Json tiddler files are always arrays, correct? This is because the deserializers always return arrays, correct?Even if only 1 tiddler, that is what I have seen (a one element array). The minimum definition for a tiddler is a 'title' field, so if the incoming Json has a root array and the object in index zero has a 'title' field, this is a tiddler json file. Therefore, we can ignore all root object files, and all files that have a .json.meta file, and only post-process arrays returned from the deserializer. Then it would be figuring out the priority order of what to do if an incoming file passes the above test but also has a .json.meta file. I would suggest giving the meta file priority.

/*
Load the tiddlers contained in a particular file (and optionally extract fields from the accompanying .meta file) returned as {filepath:,type:,tiddlers:[],hasMetaFile:}
*/
$tw.loadTiddlersFromFile = function(filepath,fields) {
    var ext = path.extname(filepath),
        extensionInfo = $tw.utils.getFileExtensionInfo(ext),
        type = extensionInfo ? extensionInfo.type : null,
        typeInfo = type ? $tw.config.contentTypeInfo[type] : null,
        data = fs.readFileSync(filepath,typeInfo ? typeInfo.encoding : "utf8"),
        tiddlers = $tw.wiki.deserializeTiddlers(ext,data,fields),
        metadata = $tw.loadMetadataForFile(filepath);
    if(!!metadata && tiddlers.length === 1) {       
        tiddlers = [$tw.utils.extend({},tiddlers[0],metadata)];
    }
        else if (!!metadata && tiddlers.length > 1) {       
        tiddlers = [$tw.utils.extend({},[tiddlers],metadata)];
    }
        else if(tiddlers.length > 0 && !(":"+Object.keys(tiddlers[0]).join(":")+":").match(/:title:/i) {        
        tiddlers = [$tw.utils.extend({},[tiddlers],metadata)];
    }
    return {filepath: filepath, type: type, tiddlers: tiddlers, hasMetaFile: !!metadata};
};

Trying to think if I missed a case, or if this would mess with other deserializers, but I really don't think it will. Also, just tested dropping a root object Json file onto tiddlywiki.com. If it has "title" it is imported as a titled tiddler. If not, nothing is imported. Same for any objects inside a root array. If it has a title, it gets imported, otherwise ignored....

[Jermolene]

Thanks @joshuafontany I've worked up a slightly different solution:

• First, to make it possible to load application/json tiddlers without them being interpreted as a bundle of tiddlers
• Second, to update the file system adaptor to write .json files for application/json tiddlers

Fixes coming up, I'd appreciate help with testing.