Custom Fields/Metadata

[j-maynard]

Feature Request

Custom Fields/Metadata

Use Case/Motivation

If you're journaling you may want to track metadata such as music or mood but in this case the list of things people could put in metadata is pretty limitless. Having Metadata means will allow people to search or filter based on custom metadata in the same way they can filter through tags.

Example Usage

jrnl --field="<key>:<value>, <key>:<value>" jrnl --field=<key>:<value> --field <key>:<value>

The above usages probably makes the most sense. The fields can be appended to the end of the entry as simple JSON or just an array [ ] of key value pairs at the end of each entry:

  1 [2021-11-24 00:33] A quick test journal
  2 This is a quick journal entry to see how it works. Now I've added @test tag to see how it works
  3 ["Key":"value","Key1":"value"]

Searching can be as simple as: jrnl --field key1 jrnl --field key=value

I am guess people will probably have their own ideas on how to implement this and make it work. This was my first stab at it looking over the code and file format. I may start working on a PR to implement this and see what people think if I can ever find the time.

[micahellison]

Hi there, thanks for filing this. This is something @wren and I have been talking about a lot, though we're hesitant to implement it on top of jrnl's current format. Lots of developers use jrnl, and we're particularly afraid of their copy-pasted code snippets being interpreted as metadata when they're really part of an entry.

Our hope is to introduce features like this on top of a new storage format that's a bit more rigorous than our current format when it comes to delimiters and timestamps. We're collecting issues on this in the 3.0 milestone and hope to be getting to those soon after wrapping up some issues around code architecture and configuration.

[edoardob90]

I was thinking about a few metadata fields that might be useful to add to each entry, e.g., the GPS location and the weather conditions. Do you think these could be added to the current entry format or should be deferred as well? I'm not sure where it would be more appropriate to add this info, perhaps at the very bottom of each entry?

In the location/weather example, both these metadata should be optional and could be toggled on/off with an appropriate field in the config.

[viegasfh]

I have a reading journal, and I do miss this, because I want to be able to track the pages I've read, the time I took to read, the author (I have been doing this with tags at the moment). Indeed, jrnl's format is a little limited. If it supported comments, one could embed the tags in the comments.

One thing that occurred to me, while we wait for this feature to be implemented, is to embed the metadata as JSON inside HTML comment tags. Here is an example:

<!--META {
  "title" : "My Book Test",
  "author": "Author Name",
  "date" : "2022-06-25",
  "initial-page": 1,
  "finish-page": 12,
  "start-time": "8:30",
  "finish-time": "9:30"
} -->

With this in place, you can use an external tool to extract this information through piping and produce any statistics you want.

You'd run something like this:

jrnl @reading @fiction -contains "<!--META" | cmd 

Here, cmd, would extract the JSON from the HTML comment and process it.

At the moment I am using a python script I put together to record all of this, but then I write down my findings on the reading in jrnl. I want to have everything in the same app and be able to track my reading progress.

Given that jrnl supports plugins, it could also be implemented as plugin. I am already familiar with the code, but I haven't yet had the time to look into the plugins architecture. Would this be feasible using plugins? Does anyone have any other idea on how to tackle this in some way that does not disrupt code pasted by coders? I do use jrnl to paste a lot of code examples that I comment on, so I am on the same boat @micahellison describes.

This feature will definitely enhance our experience with jrnl.

[viegasfh]

Hello again, folks!

I played around with this, and the original request by @j-maynard can already be met with the existing features the app has. As I mentioned earlier, using comments is one way of adding metadata to our documents. This will not disrupt anything, and if we are using Markdown, it will just be discarded in the output. So, here's how one can handle it:

[2022-06-26 17:01] My Test Metadata
<!--
  title=My Test Metadata
  author=john Doe
  startpage=12
  finishpage=24
  date=2022-06-26
-->

I have read this chapter with great enthusiasm. I thought it was going to be boring, but it was not.

In this case, we have the ordinary heading composed of [date] title, followed by the body. Inside the body, I have the metadata embedded in HTML comments. In this case, it is:

<!--
  title=My Test Metadata
  author=john Doe
  startpage=12
  finishpage=24
  date=2022-06-26
-->

It is important to take into account that this can be embedded anywhere in the document. So, if the user has a preference for it to be embedded at the end of the document as @j-maynard suggested, he could do so.

Following the metadata, we have the body, which can be plain text with Markdown markers if one wishes:

I have read this chapter with great enthusiasm. I thought it was going to be boring, but it was not.

So, how do you search journal entries that have this metadata? Very simple, we use the -contains parameter as follows:

jrnl -contains "author=john doe"

This will return all journal entries with an exact match of this key-value pair. Let's now say that we have a category metadata category=vacations, we would then search as follows:

jrnl -contains "category=vacations"

We can be more creative and return all entries that have a category tag as follows:

jrnl -contains "category="

And if we want to retrieve all entries with key value vacations irrespective of its key name, we would do something like:

jrnl -contains "=vacations"

Now, this is something I came up with. Anyone can create his own metadata syntax, and still use the -contains parameter to handle the searches.

Because we are outputting text, the HTML comment tags, or whatever type of comment the user decides to add to his content, will be shown in the export. However, in the case of a Markdown renderer, it is going to ignore all the comments. We could, as I have mentioned before, create an app to extract this metadata, or, as I have looked into the code, through a plugin that pre-processes them.

Despite this working with the existing features of jrnl, and given that the app is progressing and there are intentions of providing a better text format, I'd suggest that we adopt Markdown with metadata in YAML format. That is, we could have something like this

---
date: 2022-06-26 17:01 
title: My Test Metadata
author: john Doe
startpage: 12
finishpage: 24
---

I have read this chapter with great enthusiasm. I thought it was going to be boring, but it was not.

The fields title and date would be mandatory and replace the existing [date] title format.

This has a clear separation of metadata and content, and the good thing is that we don't need to create a parser for it. There is already one available that splits this for us called python-frontmatter. With this in place, we can have access to a dictionary of key-value pairs and we can have access to the content. In the Entry class we would add a metadata field, just like the ones we have for tags, etc... and the text would be the content extracted from the python-frontmatter api object.

As for the current format, would be like the version 1: be accessible in read only format. Adopting this new format will grant us better integration with other apps like Day One and Diaro. We can import the metadata fields straight into the metadata header of our entries, and the content would go below it.

As for the searching, we would then have, perhaps, a -meta parameter that would distinguish it from the -contains parameter, even though that in theory we could still use the -contains one.

Let us say that we wanted to search for all entries that have key category, we would then do something like this:

jrnl -meta "key-name:category"

If we want entries that have a key value "john doe" irrespective of key name, then we could do something like:

jrnl -meta "key-value:john doe"

If, on the other hand, we would want an exact match, we could do something like:

jrnl -meta "author=john doe"

The difference between using a = and a : is that in the former we are looking for a key-value match, whereas in the others we are looking for either a key name or a key value.

These are just a few ideas I came up with, but obviously you can evaluate and see if you agree or if you can come up with a more practical and easier approach.

As for the hack I described in the beginning of the comment, I think that it is a good candidate for the tips and tricks section of the documentation. If you agree, I can write down a quick entry in the manual for it.

Let me know your thoughts on this. It was nice to have this feature proposed, as it made me realize that sometimes it is possible to achieve something with what we already have.

[insanerwayner]

Maybe you could set a configurable symbol for a meta-tag. This special tag could include a value.

For example if I use ^ as my special meta-tag symbol I can use it follows:

^mood:chill
^music:"Pink Floyd - Time"

I'm writing a new script today with @JohnPotato. So far it's going well.