Favor reliance on existing open source implementations for our SDKs

[decentralgabe]

As a best practice, we should not write our own libraries for widely adopted industry standards when applicable. There are a number of reasons for this:

1. expertise in specialization: Creating and maintaining libraries for these specifications requires a level of expertise that is both rare and expensive. These are specialized fields where security and compliance are not just features but the foundation of the software itself. By using established open source libraries, we leverage years of collective expertise that we, frankly, do not possess and should not divert our resources to acquire, as these areas fall outside our core mission.

2. proven and trusted tools already exist: The availability of open-source tooling for these standards isn't just about what exists, but what has been battle-tested. Libraries across languages [1] [2] [3], have been scrutinized by security experts worldwide for years. This is not just about popularity but about trust – these tools have been vetted and validated across countless deployments, including environments where security cannot be compromised.

3. collective wisdom and security: By choosing to create our own solutions, we isolate ourselves from the ongoing security dialogue happening in the open-source community. The "wisdom of the crowd" in open-source projects leads to more secure, interoperable, and robust software. When vulnerabilities are discovered, the community's response is fast, something an we will have a tough time matching in speed and depth.

4. efficiency and focus: I acknowledge it's tempting to reinvent the wheel, and I have done this myself in the past a few times...usually from a desire for control and customization. That said, "not invented here" syndrome is a known pitfall, leading to wasted resources on solving problems that have already been addressed. Our efforts are better directed towards innovation in our core business areas, rather than duplicating work that doesn't offer a competitive advantage.

I will caveat this with saying there may be times where there are good reasons to create our own implementations of things, namely when using an existing open source implementation causes more harm than help and does not accelerate our improve our efforts. I believe such instances should be rare and scrutinized and follow the following practice:

1. Use open source libraries
2. If problems arise, contribute back to open source libraries
3. If contributes take time, fork open source libraries
4. As a last resort, create our own implementations

[1] Popular go JW implementations lestratt-go with 1.8k stars, 3.4k+ uses, go-jose created by Square - 2k stars, hundreds of usages [2] Popular JVM JW implementations nimbus, auth0's java-jwt with 5.6k stars, 87.4k+ usages, [3] Popular JS/TS JW* implementations jose with 8.7M+ downloads

---

* I gave the example of JW since it was most visible, but it can be true in other instances where we've created our own implementations like for VCs, DIDs, DNS packets, encoding algorithms, and cryptographic primitives.

[KendallWeihe]

I agree with this so long as it doesn’t rob us of delivering a great developer experience.

I think a lot of this hinges on what the value proposition is of our web5 implementations, and any stance would be an expression of what I think the value ought to be.

My interpretation of what's being proposed is: our web5 implementations ought to be a thin wrapper around existing tooling. In such a case the value proposition may be bundling & a consistent DX. I could get behind that proposal but would need to see it proposed concretely; for example should we not expose any JWT functionality? Because I would challenge that seeing as JWT’s are so heavily integrated into VC’s in the form of a VC-JWT, so to not do so and expect developers to juggle two different DX’s would be a poor design decision IMO.

So I suppose I would need to know concretely what we want our exposed DX design to be and then work inwards from there on what tooling is already available to reap all the benefits you laid out in the description @decentralgabe.

My only quibble is with the pitch on efficiency. I think there's a seductive sales pitch to "use something someone else already built because it'll save you time" but in reality it's not uncommon for that to end up taking more effort. How many times does that happen vs not? I have no idea but enough for me to feel compelled to say it.

[KendallWeihe]

@decentralgabe since we're defining a subset of the DIDs and VC specs in our web5 specification (here and here), we can use open source libs for these concepts, but ultimately we'll still have to define our own types and map back-and-forth to-and-from our defined type requirements and what the underlying lib defines. Am I thinking about this right? Is this consistent with what you're ideating on here?

[decentralgabe]

I see no problem in implementing our own DID + VC models — if no existing open source lib is sufficient – as that is what the spec defines, and is one of the main purposes of these libraries.

I also need to clarify that the specs set a floor but not a ceiling of functionality.

[KendallWeihe]

So I think the matter in my previous comment is a good case study for the focus of this ticket.

I also need to clarify that the specs set a floor but not a ceiling of functionality.

I think this is where the rubber meets the road, and I have to articulate with an example.

In our implementations, if we're given a DID Document in JSON representation, and we deserialize it into one of our programmatic types, and the given JSON is not compatible with one of our programmatic types, then the deserialization results in an error. So if we have two different implementations, where one implementation is closer to the ceiling and the other is at the floor, and we serialize from the former and subsequently deserialize in the latter, then the deserialization may result in an error. In such a case, both implementations are "conformant" to the spec, in accordance that the spec sets a floor but not a ceiling, but in this situation there is no interop.

We can support interop, but it takes extra work. The solution is to only serialize to the floor because we can only ever expect deserialization to support the floor. The former implementation has to take the data model from an underlying open source lib, and then map it into a web5-spec conformant type, whereafter serialization can occur. Now we have interop but we have had to write extra code for the mapping logic.

My only quibble is with the pitch on efficiency

This is the essence of what I'm getting at here. By using open source libs we introduce additional abstraction layers which ultimately have to be mapped to-and-from. I'm not taking this to the extreme, and claiming it's always extra work and extra code, but it sometimes is.

Think of our implementations as giant input-output machines. Since our web5 specification is a subset of other specifications, then the output of all of our implementations can only ever be that which is defined in our web5 specification. The input, however may be a superset of what is defined in our web5 specification. So in such a case, we can rely on open source libs, insofar as they meet the floor requirements, for the input side of the machine but we cannot rely on them for the output side of the machine. With this constraint in mind, in many such cases, using an open source lib will only serve to add required work -- we can use the open source lib for the input side of the machine but ultimately we're still on the hook for constraining what those open source libs produce to the output side of the machine.

My intuition is open source libs are pragmatic for anything algorithmic based but not pragmatic for anything data model based.

---

It's a case-by-case basis. For example, I'm working through did:jwk's in Rust right now and frankly, that spec is so simple IMO it's most pragmatic to simply roll our own. I could spend the time proving the case or I could just do it.

[decentralgabe]

In our implementations, if we're given a DID Document in JSON representation, and we deserialize it into one of our programmatic types, and the given JSON is not compatible with one of our programmatic types, then the deserialization results in an error. So if we have two different implementations, where one implementation is closer to the ceiling and the other is at the floor, and we serialize from the former and subsequently deserialize in the latter, then the deserialization may result in an error. In such a case, both implementations are "conformant" to the spec, in accordance that the spec sets a floor but not a ceiling, but in this situation there is no interop.

The scenario you outline—where deserialization of a DID Document into a non-compatible programmatic type results in an error—actually underscores a misunderstanding of spec compliance. If errors occur because our implementation can't handle certain JSON representations, we're not truly meeting the spec's baseline. The floor of the specification is our foundation; anything beyond is extra. If our implementations fully embody the spec, then slicing it into a subset is unnecessary.

The solution is to only serialize to the floor because we can only ever expect deserialization to support the floor.

If this is the case, and what I mentioned above is not feasible, we should consider amending the web5 spec to eliminating it's restrictions entirely.

With this constraint in mind, in many such cases, using an open source lib will only serve to add required work -- we can use the open source lib for the input side of the machine but ultimately we're still on the hook for constraining what those open source libs produce to the output side of the machine.

We need not reinvent the wheel or delve into the minutiae of JW* specs—there are already secure, community-vetted implementations out there. And yes, while mapping between our specs and these libraries is a task, it's a one-time effort that pales in comparison to the ongoing maintenance and security vetting required for homegrown solutions.

Central to your argument is the concern for maintaining a great DX, which I agree is crucial. A well-designed wrapper around these libraries can provide a consistent and simplified DX while still benefiting from the robustness and security of the underlying open-source implementations. This approach also allows for the encapsulation of any necessary mappings or adaptations, minimizing the impact on consumers.

My intuition is open source libs are pragmatic for anything algorithmic based but not pragmatic for anything data model based.

Maybe...but I don't think there's much value in re-implementing a JWK data model if one out there is already spec conformant and usable in our libs. Our subsetting in web5-spec does not (and must not) break spec conformance with what it subsets.

It's a case-by-case basis. For example, I'm working through did:jwk's in Rust right now and frankly, that spec is so simple IMO it's most pragmatic to simply roll our own. I could spend the time proving the case or I could just do it.

I don't agree with this assertion. You will need to be well versed in the JWK spec to support did:jwk. Of course you can have a subset implementation that only encodes some JWKs but why not just rely on an existing library that implements spec-conformant JWKs and has a community around it to keep it updated and secure?