Mirroring/Moving away from github - Bisqubutor's github account was banned possibly because he uses tor #1302

Open chris-belcher opened 2 years ago

bisqubutor recently came on the IRC channel and told us about how his github account got deleted/banned.

The logs are here (https://gnusha.org/joinmarket/2022-06-07.log) but I'll copypaste the relevant parts:

\<JoinMarketRelay> [hackint/bisqubutor] I was about to squash some commits on my merge requets on github, but trying to login, I got the notice my account got suspended. I don't know why but only used it over tor since. \<JoinMarketRelay> [hackint/bisqubutor] I noticed they delete all my pull requests, all of my openend "issues" and all my comments too. therefor I suggest to mirror repo to some more privacy friendly git just in case. \<JoinMarketRelay> [hackint/bisqubutor] The most recent was just some minor refactoring. Nothing that important lost in my case. but still annoying if github actively prevents contributions. \<JoinMarketRelay> [hackint/bisqubutor] Login page just redirects to=> Account suspended Access to your account has been suspended due to a violation of our Terms of Service. Please contact support for more information. \<JoinMarketRelay> [hackint/bisqubutor] Regardless, I will try to keep the joinmarket directory node alive for a while.

Regardless of whether this is an accident on github's part, or intentional exclusion of Tor users, it's still an alarming reminder that we need to do something about our vulnerability to github. We need some kind of mirror or scraper that saves our issues, PRs, comments and git tree so that we can easily recover if the worst happens with github. As a project based on privacy we really need to support Tor users.

laanwj runs a tor hidden service that mirrors many bitcoin-related git repositories: https://twitter.com/orionwl/status/1155058225299042304

jb55 also hosts his own git server https://bitcoinhackers.org/@jb55/105698471194587682

Options we could use are bitbucket, gitlab or https://gitea.com/

Mirroring pure git tree is simple, I could even host some public server for that. Issues and PRs are bigger problem, need to go through GitHub API, probably there are some ready to use tools for that.

It seems like any open source project should have the same problem. I wonder if there's a blog post somewhere describing a standard way of doing it. I think I remember reading that Bisq has mirrors its issues and PRs.

@kristapsk @chris-belcher time to build a decentralized p2p GitHub alternative over Lightning Network :)

@RiccardoMasutti Not sure Lightning Network is required. Could something like nostr be useful here (haven't looked much into details there yet)?

@RiccardoMasutti Not sure Lightning Network is required. Could something like nostr be useful here (haven't looked much into details there yet)?

It was half-joke, since nowadays everyone is building on LN :)

[hackint/bisqubutor] Regardless, I will try to keep the joinmarket directory node alive for a while.

If you are reading this bisqubutor, thanks for that :)

laanwj runs a tor hidden service that mirrors many bitcoin-related git repositories: https://twitter.com/orionwl/status/1155058225299042304

jb55 also hosts his own git server https://bitcoinhackers.org/@jb55/105698471194587682

Options we could use are bitbucket, gitlab or https://gitea.com/

Thanks for those references @chris-belcher

I have also looked briefly at gitlab and gitea before. They seem at least plausible, but: I guess we have to consider self-hosting (I'm looking at https://about.gitlab.com/handbook/marketing/strategic-marketing/dot-com-vs-self-managed/). gitlab is a company; I see no reason to believe they'd be different from github if we didn't use self-hosted (of course, temporarily they could be).

This chart is interesting (though given the source it is probably biased): https://docs.gitea.io/en-us/comparison/

Is gitea is a self-hosted only thing?

Also, we're going to struggle to get contributors signing up to and/or using something new, although to be fair, it has always been hard to have more than a very few active contributors.

I'm reluctantly somewhat reluctant to change the github thing for now, but I could definitely be persuaded. It needs someone to do the work to manage the new setup (and be reliable).

I really like Github and don't see the point to move just to some other centralized solution with high probability of same problems. But we should look into ways how to backup issue / PR stuff from Github somewhere / somehow. Having truly decentralized alternative to github would be cool. but I don't think there is one right now.

Gitea is self-hosted, but they're working on federation features so it could be a good option once they've got that working. It's very lightweight and I'd be happy to contribute some resources to running a federated instance.

Medium term, mirroring github to gitea looks reasonably easy to automate.

I suspect motivated contributors wouldn't be put off by another platform, but I'm unsure about more casual users. How much trust and discoverability does Github offer compared to an unknown third party site....

Medium term, mirroring github to gitea looks reasonably easy to automate.

That seems to be mirroring only git part, not issues and pull requests. You can do that with plain git and some simple shell script, don't even need gitea.

Some decentralised alternatives are mentioned here: https://github.com/bitcoin-core/bitcoin-devwiki/wiki/GitHub-alternatives-for-Bitcoin-Core#decentralized

@fiatjaf could build something using nostr in future

All bitcoin projects should move to alternatives IMO that works without problems particularly privacy projects

Some thoughts and trying to summarize what's been mentioned so far:

While there are interesting non-git solutions (like FossilSCM and others), it's desirable to keep the repo on git both for a smooth migration for existing stakeholders and to keep the barrier low for new entrants.
It is not GH itself per se that is the main problem, it's the counterparty risk. Moving from GH to another managed forge deployment such as gitlab.com will not be a meaningful improvement
With decentralization and this issue in mind, some options worth considering for JM:
- git-appraise: store and manage PRs and code review directly in the repo; no additional service required. there are web GUIs
- git-bug: git-based bug tracker
- sourcehut: Distributed forge doing interactions over e-mail
- There is work underway to federate git forges. Protocol built on ActivityPub for federating the things outside of git itself. In the case that JM does decide to go/stay with a canonical hosted managed forge, it would be great to pick one that supports it such as sr.ht, codeberg.org or hostea.org. Should also be an option for self-hosted gitea and sourcehut.
- radicle: Seen a lot of buzz, looks like an interesting solution without much adoption. Mostly mentioning it for completeness.
If staying on git, the two main contenders should be and sourcehut and gitea, either dedicated instance or with a provider. I don't see any major win in gitlab motivating the increased operational and resource overhead. Gitea has a more familiar web-based UI like GH; sourcehut allows contributions without need to registering an account or even a central web server so is better from a censorship and centralization perspective.
- postmarketOS are in a similar situation for other reasons and there's been a lot of discussion on their issue tracker, could be worth a read.
Most active contributors and recent commits are already using GPG signatures, which is great. I would consider requiring all future commits to be GPG-signed with a key on the current GH repo. In the event of unexpected unavailability, this would allow users to easier discover and share mirrors and archives.
This Monday was chilling and I'd advice trying to move on this sooner than later.

Something maybe worth looking at too was mentioned on Twitter:

There's a very interesting gossip protocol called Secure Scuttlebutt that has something called git-ssb

https://git.scuttlebot.io/%25n92DiQh7ietE%2BR%2BX%2FI403LQoyf2DtR3WQfCkDKlheQU%3D.sha256

This protocol could also fix centrally planned social media platforms such as Twitter.

https://scuttlebutt.nz

Although nostr README explictly mentions Secure Scuttlebutt, main advantage of nostr over Secure Scuttlebutt being simplicity.

We are in a similar situation, along with having to replicate proxies / mirroring package registries (go modules / npm).

Can vouch for sourcehut: its great service, really though you should consider the mailing list and submit fix via patch (email) which is native to git and easily done via sourcehut. This will give you a worst case failover in case total compromise occurs.

FWIW we are moving to a self hosted Gerrit instance on bare metal. If your interested can make it open source under permissive license.

Godspeed, and there are more with you than you know.

Transcript by @kanzure related to topic - Strategies for migrating Bitcoin Core off GitHub.

Here's some developments with git + nostr by @jb55. https://twitter.com/jb55/status/1595515096184532992

This looks promising, bug tracking with pure git - https://github.com/MichaelMure/git-bug.

It looks there is now 1.2 BTC bounty by Jack Dorsey to develop nostr-based GitHub replacement.

Related - Censorship-resistant open source on Nostr.

We are in a similar situation, along with having to replicate proxies / mirroring package registries (go modules / npm).

Can vouch for sourcehut: its great service, really though you should consider the mailing list and submit fix via patch (email) which is native to git and easily done via sourcehut. This will give you a worst case failover in case total compromise occurs.

FWIW we are moving to a self hosted Gerrit instance on bare metal. If your interested can make it open source under permissive license.

Godspeed, and there are more with you than you know.

Another option could be setting up a self-hosted gitlab server. But they could try to change the license to not allow TOR hosting - which then could still use the last version with the unrestricted license.

https://www.howtogeek.com/devops/how-to-set-up-a-personal-gitlab-server/

Related NIP: https://github.com/nostr-protocol/nips/pull/223

Jack Dorsey just annonced he's raising his bounty for this from 1.2 BTC to 10 BTC. https://iris.to/post/note17gfm0k0ssw4qctpge32dp3nulu975mjpdl9nqmrs78msp622d90qvdral4

https://bountsr.org/code/2023/01/19/nostr-based-github.html

It looks there is some project working in that direction - https://github.com/NostrGit/NostrGit.

It looks there is some project working in that direction - https://github.com/NostrGit/NostrGit.

I've also started working on https://github.com/akhavr/nostrya Hopefully will have something working "in two weeks" (tm)

Two projects probably worth looking at:

https://github.com/MichaelMure/git-bug (Distributed, offline-first bug tracker embedded in git, with bridges)
https://github.com/google/git-appraise (Distributed code review system for Git repos)

From this Nostr thread - https://iris.to/note1r2nahye9mekplptvc2xaptm07kyz0l4adq545ynewqgz2wdm065qwncl7d.

GitHub Metadata Backup and Mirror

https://github.com/nostr-protocol/nips/pull/997

https://gitworkshop.dev/

Probably useful tool - https://github.com/josegonzalez/python-github-backup .

JoinMarket-Org / joinmarket-clientserver

Mirroring/Moving away from github - Bisqubutor's github account was banned possibly because he uses tor #1302