One query language to rule them all

[indianakernick]

Something that I think would be really cool is if other plugins could be built to act as views onto the Datacore index. There are a whole bunch of plugins for visualising notes such as Heatmap, Tracker, Map View, Fantasy Calendar and so on. Some of those mentioned have querying functionality built into them for filtering notes. What if this was one unified system?

One of my main frustrations with a lot of these types of plugins is that they require me to write my notes in a certain way to be visualised. I have temporal data but I have to use these particular property names for some calendar plugin to interpret it. Then I decide that I don’t like that plugin and I want to try another. A different plugin requires a different note structure so I have to go and change all that.

What if I could write a Datacore query on my notes that produces data that another plugin can understand and use to visualise the notes? I would have the full power of Datacore’s filtering and projection. Visualisation plugins require data to be in some particular shape. My notes can be in any shape I want, so long as I can write a Datacore query to view them as the desired shape.

Perhaps I want to write GPS coordinates as a pair of lat lng properties. The Map View plugin requires that a location property is used. I could write a Datacore query that projects the two lat lng properties into a location property. Perhaps I want to filter what appears on the map based on some other property. Map View has a query language for filtering but it doesn’t support properties. Datacore can do that.

Perhaps I have some event notes with a date property. I want to visualise these with Fantasy Calendar but that requires a fc-date property with a date object rather than an ISO 8601 string. I could write a Datacore query to parse the string and convert it to the object that Fantasy Calendar requires. The name of an event on Fantasy Calendar is either the value of the fc-display-name or the file name. Perhaps I want the display name to be built from concatenating several other properties together and I don’t want to duplicate the data. Datacore can do that. Perhaps the name of the calendar event is actually within a property in a linked note. Once again, a Datacore query could extract that information.

I could go on! There are countless examples where I would want to use Dataview’s filtering or projection capabilities in the context of some other plugin. Maybe Datacore can make that sort of thing possible. Maybe it’s already possible but not documented.

Combining this with #18 and we end up with something very powerful. What I’m talking about here is sort of the opposite end of that.

Also related is #16. If building a plugin that depends on a Datacore query is sufficiently simple, then Datacore’s own visualisation capabilities could be moved into a separate plugin.

This idea is probably too ambitious but I thought I’d throw it out there and see what people think.

---

I wrote all of this out and then I looked at the Heatmap Calendar plugin and saw that it used a render function. So I can calculate the data however I want and then pass it into that function. Perhaps what I’m really looking for is for more plugins to expose interfaces like that.

This midnight ramble seems to have gone nowhere but maybe there’s still something here.

[indianakernick]

I've been continuing to think about this. Code blocks could act like React components and return JSX nodes. React could do its thing and update the HTML elements that have changed rather than rebuilding the whole DOM each time. I did some digging around in the code and from what I can tell, that seems to be the plan.

[xDovos]

The problem that other plugins need their specific metadata names can't be fixed with dataview or datacore. It is a "design flaw" of the other plugins. Even right now the other plugins could load the dataviewjs api and use the dvjs search for their use. Also it is easy to imagine a fit for all tool and create a ton of possibilities but this also means a ton oc feature creep and blowing the scope out of the world. What we actually want is a plugin that gives us a stable foundation that can easily be build upon without the need to fight with it.

All the other fratures that aren't strictly required should be separate plugins that add to the core like obsidian charts is adding the charts.js library to dvjs and is handling the rendering of the charts.

[indianakernick]

@xDovos I agree. The more I think about this, the more I realise that this problem probably cannot be solved by Dataview or Datacore. Consider what I was saying about JSX. If the Heatmap Calendar or Charts plugins returned JSX instead of doing the rendering themselves, they would be worse, not better. It would mean being tied to React. As they are, those plugins don't depend on Dataview and Dataview doesn't depend on them. Yet they are interoperable. That's beautiful. That's what plugins should aspire to do. They should work with each other without being aware of each other's existence. I can't imagine a way to improve the interface of Heatmap Calendar or Datacore in a way that makes them more interoperable because a render function already seems perfect to me. Although view pages seems like it might help.

[blacksmithgu]

I think it's solvable; the datacore query language should be general enough to represent most types of useful queries, and datacore could internally convert these query types into API calls against other libraries.

I'm not sure where this implementation would best go (Datacore? other plugins? a third compat plugin?) but it seems reasonable.

[Azeirah]

I think the idea of "one unified querying language to rule them all" is really important for the Obsidian ecosystem as a whole.

Have you considered teaming up with Obsidian core team? They do have a querying language themselves too, it's really fast and reliable in what it does offer, but it lacks in flexibility and expressiveness.

I think Obsidian is working on this to an extent regardless. With your knowledge on building dataview, I think you could have many valuable things to contribute to the core. And vice-versa.