Replace GitHub proxy with git over CORS proxy

[adamziel]

This issue belongs to the proxy repo, but I couldn't quickly find it so I'm dropping it here before my plane takes off.

Let's talk about github-proxy.com. It may be hosted on an automatically-scaled infrastructure, but we’ll still run into scaling issues eventually.

Short term, I’m worried about GitHub rate-limiting those servers because of all the intense git clone operations. Is there a way to plug in an API key? Or perhaps an SSH key already takes care of that?

Long term, I’m worried about the actual throughput. Once we run into speed issues, let’s explore edge-caching zip files, queuing and rate limiting requests, size limits for cloned repos and artifacts, maybe keeping hot clones of 10 most popular repos etc.

[adamziel]

Let's move all the Git-related computations (fetching, computing deltas, decompression etc.) into the browser – no server, no problem (with CPUs, memory, or storage). PHP and JavaScript can talk to Git directly. We'll still need a CORS proxy to make the repositories accessible via fetch(), but that's much easier to scale and maintain.

For v1 we need:

• Git sparse checkout support – I've prototyped it here: https://github.com/adamziel/git-sparse-checkout-in-js/tree/trunk. In the long run we could port it to PHP and allow using it in WordPress core. See also my explainer blog post.
• A CORS proxy – we'll need to host a streaming CORS proxy on a separate domain, like cors.playground.wordpress.net or even cors-playground.wordpress.net or playground-cors.com to avoid any subdomains-related risks.

[adamziel]

I think we could have the same for SVN. We’d need to implement svn+ssh over a CORS proxy which would take some time so I’m not prioritizing it, but it’s a nice future option to have.

Also porting that to PHP would unlock installing WordPress plugins from the latest trunk version and updating them on new commit. Probably not good for production sites, but I can think of a few use-cases already.