search

solid / vocab

Solid Vocabularies
https://solid.github.io/vocab/
42 stars 14 forks source link

Add CRDT vocab #69

Closed NoelDeMartin closed 1 year ago

NoelDeMartin commented 2 years ago

I'm about to release a new app I'm working on, and I've created a new vocab to track changes in resources. I'm calling it "CRDT" because I intended to implement a CRDT (Conflict-Free Replicated Datatype) data structure, using the Solid POD as storage. I'm not entirely sure if this is technically a CRDT or not, so I would be fine calling this "history tracking" or something else as well. For more background on this, we discussed it in the Solid forum in this thread: Request for Comments: CRDTish approach to Solid.

Initially, I intended to host this myself at https://vocab.noeldemartin.com/crdt or something, but talking with @timbl he mentioned that maybe this would make sense as a w3.org vocab. So here it is. I'm not entirely sure if this gets merged how the actual w3.org/ns/crdt url is configured and if I'll be able to modify the html or something. Or is that just auto-generated from this repository?

For an example of data using this vocab, you can look at the following files which represent changes made to a document over time (the parts within [[ ]] are just placeholders that will be replaced during testing, so assume those are real values):

pietercolpaert commented 2 years ago

Addition comment complementary to your proposal

If

We would be able to implement the event sourcing architectural pattern here using Linked Data Event Streams and your CRDT vocabulary. Is this of interest?

On URI dereferencing

Instead of w3.org, you could also use w3id.org, which you can make dereferenceable through a pull request here: https://github.com/perma-id/w3id.org - and this is also how we do it for the Linked Data Event Streams vocabulary: https://github.com/perma-id/w3id.org/tree/master/ldes

NoelDeMartin commented 2 years ago

Thanks everyone for the comments so far!

@pietercolpaert

If

* Every resource is immutable

* Deletions happen via lazy deletion (flagging a resource as deprecated/deleted/unscheduled/invalidated by/...)

* Updates can be full resource updates instead of partial representations

We would be able to implement the event sourcing architectural pattern here using Linked Data Event Streams and your CRDT vocabulary. Is this of interest?

Technically we could say the operations themselves are immutable, but the resource they make reference to isn't. I talked about it in the forum thread, but TLDR I want my apps to be as interoperable as possible, so the data itself still needs to have a standard format following a known vocabulary (schema.org for example).

The operations are auxiliary to resolve conflicts between edits made in different devices (the app is offline-first). It also allows me to have a history of changes, but that's something I still haven't paid much attention to.

Instead of w3.org, you could also use w3id.org, which you can make dereferenceable through a pull request here: https://github.com/perma-id/w3id.org - and this is also how we do it for the Linked Data Event Streams vocabulary: https://github.com/perma-id/w3id.org/tree/master/ldes

Thanks for pointing this out, I didn't know this existed. I guess what's cool about this is that I could host the vocab in my own website for example, right?

What I don't like is that developers trying to read the documentation would be redirected, and it'd be a bit confusing for them. If they're trying to learn about w3id.org/crdt#Operation for example, they would get redirected to vocab.noeldemartin.com/crdt#Operation and I think it's confusing that the canonical url redirects. I'd prefer it if the canonical url returns the appropiate response directly.

@csarven

What specs need this vocab?

None, currently. I'm the only one using it in a new app I'm developing. Is it necessary that vocabs are related to a spec? I don't think schema.org has a spec, I thought it'd suffice if the vocab is self-describing in how it should be used.

Is the vocab intended to be Solid-centric?

Not necessarily, I suppose. Any context using semantic data to track changes made to resources would be able to use it. When I say "resources" I mean rdfs:Resource.

Are there other tools that needs this vocab?

Not so far.

Is this vocab considered stable?

Given that what the vocab describes is so fundamental, I don't think it'll change too much. And if it does, it'll probably be to add more operations for example, but I don't think the ones I included will change. I may be mistaken though, given that I haven't even released the app yet. So I don't think I could call it stable.

The thing is that by the point I release the app, I want the canonical url of the vocab to exist. That's why I'm submitting this now.

https://github.com/solid/specification/blob/main/CONTRIBUTING.md#vocabulary-management

I've read the documents linked there, but I must confess I'm a bit lost with all the jargon. Reading the document about creating new namespaces, I didn't see anything specific I should do so I just created the vocab looking at this other PR https://github.com/solid/vocab/pull/68. Although I see they reference a specification, is that the problem?

NoelDeMartin commented 2 years ago

I just updated the vocab taking into account some of @timbl's comments.

I've also added a couple of new things to handle deleting resources: a Tombstone class that can be used to indicate that something that was there isn't anymore, and a :deletedAt property that can be used either on the Tombstone to indicate when the resource was removed, or in the Metadata to soft delete resources.

NoelDeMartin commented 1 year ago

Hey, thanks everyone for all the feedback.

I am about to release my app and this doesn't seem to be moving forward, so I decided to publish the vocab under https://vocab.noeldemartin.com/crdt/. If you want to follow along, I'm working on it in this repository: https://github.com/noeldemartin/vocab.

I'm closing this issue for now, if there's any interest in the future to add something like this to the official ontologies we can discuss it.

pietercolpaert commented 1 year ago

Aw! I do think this is quite useful! Just trying to wrap my head around possible alternatives and what should be further maintained in Solid. Really like this line of work @NoelDeMartin!