Ways forward for go-ssb?

[decentral1se]

We're closer than ever to fixing https://github.com/ssbc/go-ssb/issues/237 but also quite far. The remaining issues are more obscure than ever and it's hard to even begin to grok what is needed to fix them. Also, they reference things that aren't even supported in the wider ecosystem like nulling feeds and private group messages.

My aims are to have a go-ssb that is stable, reliable and is keeping up with the JS ecosystem. That's currently not the case due 1) we seem to be still facing the possibility of several race conditions 2) EBT replication is broken. We're currently investing time into fixing the test suite but I'm starting to feel like the remaining fixes don't overlap with 1 & 2 and that's what I want to focus on.

Is there a case for reducing the scope of go-ssb and stripping back some of the extended functionality?

Don't we just need EBT replication and the legacy feed format to get back in the game?

It seems like to resolve 1 we need to be able to re-architect go-ssb somewhat to be able to reason about how concurrent operations happen and where in the codebase. We could learn from the scuttlego energies in this regard. And for 2, we just need to need to build some momentum because the fixes are not easy to reason about but it doesn't seem insurmountable, it just needs focus.

Some stuff that we don't need rn is metafeeds, private groups and multiple feed format support (e.g. gabbygrove)? We will need them later ofc but as an additional thing to add on and not something to try make work whilst wading through several unknowns? Strip now and come back to it?

Also, JS interoperability testing in this code base, can't we use https://github.com/ssbc/netsim instead? Then we wouldn't have to deal with running NodeJS things here, which is kinda hard. Like, I still can't run some of the tests locally. I agree interop testing is important but it seems netsim would be a much cleaner interface for this.

I'm still open to the idea of moving to scuttlego as the thing to focus on but I'd like to see this being done collectively. Do we archive go-ssb and converge on scuttle-go? As I mentioned in %jNeeIM/tOsJIYD6UuxGLQknDvnKDdOg4j7uVkpBjU2E=.sha256, it's unclear to me how we'd make scuttlego pluggable for things that don't support the Planetary use-case but everyone seems open to trying to make it work, so I feel reassured! This would demand a higher level of involvement re: coorindation from @boreq... i imagine we'd have a lot of questions coming your way from us.. and I don't know if you need that extra workload rn :upside_down_face:

I'm open to thoughts! Either way, it just seems like banging away at the drum of tryingn to fix the test suite is not The Way right now. And trying to come up with a collective plan to work together to have a working Golang SSB thing is a better idea. Regardless of level of involvement rn, I think we have a great group of Go hackers in SSB world and I think we could come up with something great @boreq @KyleMaas @cryptix @keks @cblgh and others! I won't ping all in https://github.com/ssbc/go-ssb/graphs/contributors for now...

[KyleMaas]

As far as how the program is architected, I'd have to agree that there are some major difficulties in debugging and maintaining parts of this project. It's clear that a lot of it was architecturally copied from the JS implementation where it's only run single-threaded and then had multithreading tacked onto it without thought for how this impacts Go operations. The whole Badger/Margaret/Luigi thing makes it nearly impossible to do multithreading properly, since none of them are really being used in a way that's friendly to multithreading. The goal of having indexes be able to process asynchronously is a great idea in theory, but there's so much functionality that relies on the indexes being up-to-date at all times that it's nearly a pipe dream. All in all, this project is a single-threaded project desperately trying to masquerade as a multithreaded one.

The biggest barrier I've found to actual production use of go-ssb is EBT. Not having that working is a deal breaker. But how that works and what's wrong with it is beyond my abilities to grok. I'm thankful and hopeful that the progress y'all have been making with figuring that out will be fruitful. Because if that was working, I think we're almost at a point where this project could be considered "early beta" quality instead of "alpha".

Metafeeds, private groups, and gabbygrove support seem like they were experiments that only ever got the half-baked stage. Which is not bad, but they do add a lot of complexity. They feel like they were trying to aim to copy functionality from the JS side that wasn't quite done, and then never finished. I don't know how you'd remove any of them, though, because they're rather intertwined in the codebase.

Nulling of feeds is one that I do think is a good idea to keep around. Particularly for any pubs that are open to the internet, you'd want to be able to have the functionality to eliminate a single malicious feed from the pub without wiping the entire database and starting over. It definitely adds complexity but the benefits IMHO are worth it.

Re: concentrating efforts - Personally, I'm of the opinion that more diversity in the implementation ecosystem is better. The more the merrier. And that means that in my mind a working go-ssb, scuttlego, and JS implementations are not a waste of time to work on separately.

Also, from what I understand, scuttlego uses some of the functionality from this project in more of a library context, which gives us another reason why the base "library" should at least achieve beta status at some point. Saying the official Go implementation is scuttlego but then dropping support for underlying functionality in go-ssb seems a little silly to me.

Re: JS interoperability - I don't see a whole lot of problems with using NodeJS for interoperability, since it means we can continually update the libraries it's pulling in so we can test against newer versions. netsim doesn't seem like it would be as complete of a test. And proper interoperability with the JS implementation is crucial for the success of this project. Now would be a good adjunct to the way it works now? Sure, I'd absolutely agree that more testing is better for interoperability's sake. But I think having tests run against the JS side directly is still a good idea.

This is a good discussion to have, though. Big picture planning is something that's usually lacking in open source projects in my experience, so it's good that's happening here.

[decentral1se]

Re: concentrating efforts - Personally, I'm of the opinion that more diversity in the implementation ecosystem is better. The more the merrier. And that means that in my mind a working go-ssb, scuttlego, and JS implementations are not a waste of time to work on separately.

In theory, yes. However, the reality of having few active Go SSB hackers atm needs consideration here imho. We have a wider network but not everyone is focused on working on this code base atm. If we're all solo hacking on different moving parts, we're dispersing our efforts.

We got really far by focusing on the test suite in a coordinated fashion. I imagine we'd have similar boosts if we continued in this way. I just don't know which way right now.

There is a lot of fatigure in the SSB developer ecosystem around trying to get some working reliably and new people find it very hard to get started. So, centralising efforts to get one thing working well would naturally open it up wider than ever afterwards because more people could use the tools and develop their own stuff.

That could be scuttlego, go-ssb or another thing at this point, it's not clear to me.

Also, from what I understand, scuttlego uses some of the functionality from this project in more of a library context, which gives us another reason why the base "library" should at least achieve beta status at some point. Saying the official Go implementation is scuttlego but then dropping support for underlying functionality in go-ssb seems a little silly to me.

scuttlego uses some internals but not all. More building blocks afaiu. So, go-ssb could go away and not affect it too much. Probably some code would need to be pulled out and modularised. Also, there is a plan to make scuttlego have pub magic powers.

[cryptix]

I don't want to rehash history to much but these days I'm even less convinced that trying to replicate flumedb was necessary. It might be if you need to embedded a database into your program but there are other ways to go about this, too and maintaining a custom database is more work than writing a ssb implementation should be. Some years ago I played with using Postgres as a storage backend and I'd probably do this again these days just to keep my sanity.

I also agree with @KyleMaas in that this Codebase needed rearchitecting years ago. Cutting down scope seems sensible to me, too. It would be nice if the mentioned incomplete prototypes could live ontop of that new thing and might be cool to sketch them as use-cases.

I don't care so much if that thing is scuttlego or something new but I'm all for sunseting the go-sbot as there is so much cruft and keeping the good pieces.

(My two cents and also being clear that I don't have much time right now. This might change if I manage to reduce my Dayjob hours a bit later this year but no promises. I also want to make music in my free time.. 😅)

[decentral1se]

Some great thoughts, thanks all for weighing in.

Folks are welcome to pick up a direction from here. Will close for now.

[decentral1se]

https://github.com/ssbc/minibutt-spec