support simple HTTP protocol

[SoniEx2]

I'd like to use js-git in my project: https://ganarchy.autistic.space/project/385e734a52e13949a7a5c71827f6de920dbfea43/

to provide a web UI for repos, when there's no web UI available. unfortunately most of my repos use the simple HTTP protocol, which doesn't even need CORS or anything, but js-git doesn't support it. those repos are the main (and, arguably the only) use-case I have for this. can we get support for that?

[creationix]

Good news, your repos seem to be the most simple format. I traced a git clone to see what requests a web client would need to make:

$ GIT_CURL_VERBOSE=1 GIT_TRACE=1  git clone --verbose https://soniex2.autistic.space/git-repos/abdl.git --bare 2>&1 | grep GET
> GET /git-repos/abdl.git/info/refs?service=git-upload-pack HTTP/1.1
> GET /git-repos/abdl.git/HEAD HTTP/1.1
> GET /git-repos/abdl.git/objects/b2/59b179f5ceba60a1d04fef07559c0b01720c31 HTTP/1.1
> GET /git-repos/abdl.git/objects/a0/315c67fddc7919e6d6892e8d1eb15f328057a6 HTTP/1.1
> GET /git-repos/abdl.git/objects/c0/67e3258864d5c3bcaf03beee3ff6744cd03fae HTTP/1.1
> GET /git-repos/abdl.git/objects/be/4afe68c46ee28d84ab0082b3c4f01d6bba02db HTTP/1.1
> GET /git-repos/abdl.git/objects/e9/f667fdb1cf4176d9bd62e7fbbecedb60cde008 HTTP/1.1
> GET /git-repos/abdl.git/objects/3f/9f66712aaa071bd3bb32c46e1e4dc1fed13378 HTTP/1.1
> GET /git-repos/abdl.git/objects/90/09c4f4e3bd8519eb133d50c1bd29d8ab108bd3 HTTP/1.1
> GET /git-repos/abdl.git/objects/72/2e26908435ad18ba20c0742a8fda98a29fde51 HTTP/1.1
> GET /git-repos/abdl.git/objects/b4/abf62666666fe26a5e4d7970465b292536d9e6 HTTP/1.1
> GET /git-repos/abdl.git/objects/31/51b2adb8417f4c1d3056c244e3b86a690606a4 HTTP/1.1
> GET /git-repos/abdl.git/objects/8b/35d98f9087296098bb4b69ce663540ebbc0488 HTTP/1.1
> GET /git-repos/abdl.git/objects/16/3b9a366e80f892ec0bc3a30b1790a244d513c9 HTTP/1.1
> GET /git-repos/abdl.git/objects/d2/5baa9c0a78b0007a34a569b774d983b905f0b5 HTTP/1.1
> GET /git-repos/abdl.git/objects/be/3f7b28e564e7dd05eaf59d64adba1a4065ac0e HTTP/1.1
> GET /git-repos/abdl.git/objects/ba/f9414a35a2f48ed1b22644fd4522272fb4bc66 HTTP/1.1
> GET /git-repos/abdl.git/objects/4c/25074288dbe80f599d1d0ad7572bf321353695 HTTP/1.1
> GET /git-repos/abdl.git/objects/df/0eb072ab980a4ef9fad6b60e51ad90795d3e09 HTTP/1.1
> GET /git-repos/abdl.git/objects/8f/60342d533fae1b70d7ddbaedf0d97cbb515bca HTTP/1.1
> GET /git-repos/abdl.git/objects/34/551d96ce021d2264094a4941ef15a64224d195 HTTP/1.1
> GET /git-repos/abdl.git/objects/a3/e98b807c813435ebf5aee56e49e83a2981e426 HTTP/1.1
> GET /git-repos/abdl.git/objects/4d/c391f4d3d5baa85024cc8cc73122c5f6bbed34 HTTP/1.1
> GET /git-repos/abdl.git/objects/3c/d61f3e6902fcdb298c408c3a9d89b029967140 HTTP/1.1
> GET /git-repos/abdl.git/objects/06/f5f35148cfcb371fd8203679e0c25c021a6bce HTTP/1.1
> GET /git-repos/abdl.git/objects/bc/9bfb31bb009a7f3fa8ce79a4f1afdb74ce58c0 HTTP/1.1
> GET /git-repos/abdl.git/objects/ef/cd3dfa8926609a9f62090dd1e0c5deec2b82f8 HTTP/1.1
> GET /git-repos/abdl.git/objects/a8/b17ceb3a8f69219a706b04fdfd629cb915e350 HTTP/1.1
> GET /git-repos/abdl.git/objects/2f/76586c47a42ec68b54f515ae06d2ea338858de HTTP/1.1
> GET /git-repos/abdl.git/objects/83/ca2ca94f75671be795248183a21b1cfb8dc773 HTTP/1.1
> GET /git-repos/abdl.git/objects/98/21bd288c0c7eb0daa46fa3d54eb7e584b786ea HTTP/1.1
> GET /git-repos/abdl.git/objects/57/fb9764240ab131769717863303b9593d6baf7c HTTP/1.1
> GET /git-repos/abdl.git/objects/cb/dcd9dd71f3215520b40b0454cd21c03101bd33 HTTP/1.1
> GET /git-repos/abdl.git/objects/02/532cc7085ebe370ff482d2c56d1d19a3eb5421 HTTP/1.1
> GET /git-repos/abdl.git/objects/3f/ba7311c04e6f14d7fd8080b32ea9b80fc9a0ee HTTP/1.1
> GET /git-repos/abdl.git/objects/fd/4123ec5aba8c6b75586111d960ad59e6816095 HTTP/1.1
> GET /git-repos/abdl.git/objects/35/0ba95f9d1561486d702b9deed6391075851f78 HTTP/1.1
> GET /git-repos/abdl.git/objects/e4/f8272a637dda62b2f483244dbd2b31c588484e HTTP/1.1
> GET /git-repos/abdl.git/objects/04/29b51a212c85a0790eac764bac54e403328a17 HTTP/1.1
> GET /git-repos/abdl.git/objects/b4/971f08758a4428fe90d0a56f58257cdbd728eb HTTP/1.1
> GET /git-repos/abdl.git/objects/2c/6b26aa0b01458195f09b13e02ab98b9925f5a4 HTTP/1.1
> GET /git-repos/abdl.git/objects/3e/179a28a7c4a11eb39c90f1c855180acc5220cc HTTP/1.1
> GET /git-repos/abdl.git/objects/1d/e2e15f8997a2bd450d8aa2d5d57564c1787650 HTTP/1.1
> GET /git-repos/abdl.git/objects/96/1acff9d5c6b3c18533739f162c03b8789ef3d5 HTTP/1.1
> GET /git-repos/abdl.git/objects/4d/f9cdfd9b9e29e8abd01d0032050d9d45998204 HTTP/1.1
> GET /git-repos/abdl.git/objects/2c/ea90d76fd799834c3ad922f36a258e0de07f38 HTTP/1.1
> GET /git-repos/abdl.git/objects/73/808d4740f673f9639db321ac9b5191d9e8c1fc HTTP/1.1
> GET /git-repos/abdl.git/objects/ba/f0185c8b30287ccb90a470162bd7a4e7b3ce7c HTTP/1.1
> GET /git-repos/abdl.git/objects/3f/ed79f1de6032726db2c49b80f91e573823958a HTTP/1.1
> GET /git-repos/abdl.git/objects/d3/2a241c82cfc3f774b82d71c73d4b4e5d0aa965 HTTP/1.1
> GET /git-repos/abdl.git/objects/54/7bb68545ee09c1bd803d6e75a72db092418afb HTTP/1.1
> GET /git-repos/abdl.git/objects/5a/b7efb79e88602086f8343685355239107bf7f3 HTTP/1.1
> GET /git-repos/abdl.git/objects/65/549edbc232774810c4c30b1986f1c9e966b052 HTTP/1.1
> GET /git-repos/abdl.git/objects/d3/7053860259ad58a667e104011e680a12bbdc8a HTTP/1.1
> GET /git-repos/abdl.git/objects/fb/0b8f6d355569a8435f3427f24eb9757a76ca03 HTTP/1.1
> GET /git-repos/abdl.git/objects/a6/200fd659c806424b53bc2ec0575622f62ac3b5 HTTP/1.1
> GET /git-repos/abdl.git/objects/6f/4df5d08e147f74f91bf7ce2f882dca1390c721 HTTP/1.1
> GET /git-repos/abdl.git/objects/d7/51187bf56393a1582c9111eb2a4703d541ddde HTTP/1.1
> GET /git-repos/abdl.git/objects/c3/9a45e956441013024accdd930bc6ee46c1fd8a HTTP/1.1
> GET /git-repos/abdl.git/objects/f9/eedccc8b7f0c83fa1573b6b7473a003962f3c8 HTTP/1.1
> GET /git-repos/abdl.git/objects/06/95e3f08c034a866204a1443a5860e912a30d0f HTTP/1.1
> GET /git-repos/abdl.git/objects/af/e12d712f4364f703ed1b19f02cf058dd09fb0f HTTP/1.1
> GET /git-repos/abdl.git/objects/73/7d0df71b03234ba6fc44d8edc3467d581822bc HTTP/1.1
> GET /git-repos/abdl.git/objects/53/4bcf491e8669bcf473631a4eb3c1cae519343d HTTP/1.1
> GET /git-repos/abdl.git/objects/27/31716fccaa2820d93e7bebd50acfea5c8876ea HTTP/1.1
> GET /git-repos/abdl.git/objects/6c/87b7151c1f7dbf07a309ca6e2cc71800323089 HTTP/1.1
> GET /git-repos/abdl.git/objects/0e/c1018a0a15198e6acd518f537aa5efb70f745a HTTP/1.1
> GET /git-repos/abdl.git/objects/d9/51c7b0d752d482815491b015f75701c3df1020 HTTP/1.1
> GET /git-repos/abdl.git/objects/7f/748df5f3e341736db39977b9d3e1c3d196d054 HTTP/1.1
> GET /git-repos/abdl.git/objects/a1/b8ddf536259a04362804dbc0cf368c1fead5a2 HTTP/1.1
> GET /git-repos/abdl.git/objects/8d/de74206f433f0ab109678629d9856a48e144dd HTTP/1.1
> GET /git-repos/abdl.git/objects/fe/50369a55d087861bd288315e685c7623ca8ab6 HTTP/1.1
> GET /git-repos/abdl.git/objects/58/f49478e78c8b92ed00dc977c89e365af33e5a3 HTTP/1.1
> GET /git-repos/abdl.git/objects/dd/309c1ad5b65dd2d8f9cb98b00f4ddd87637234 HTTP/1.1
> GET /git-repos/abdl.git/objects/b8/2d59a79eaf42e6e90347acfb24f77bfc4026b9 HTTP/1.1
> GET /git-repos/abdl.git/objects/c6/9ef69afa979d7ab71c6cfdedd3b8e1d6afc116 HTTP/1.1
> GET /git-repos/abdl.git/objects/36/a629caddbb71eed332a5d55ded21923a54a5d6 HTTP/1.1
> GET /git-repos/abdl.git/objects/41/f28ebbd4fc2d03baf4136eb3345d25ac967bb3 HTTP/1.1
> GET /git-repos/abdl.git/objects/2c/d52aa1fafd8e471edab13ac401b01fbdcf79e2 HTTP/1.1
> GET /git-repos/abdl.git/objects/4e/8277e4b1bc9b5d5983b8278fbcf9a232c05a4e HTTP/1.1
> GET /git-repos/abdl.git/objects/62/1203d3239adf80fea769eda93f727cc6f4ac5a HTTP/1.1
> GET /git-repos/abdl.git/objects/f6/021c41887a70926346bce4742ccc3515e7a16f HTTP/1.1
> GET /git-repos/abdl.git/objects/c5/eef7033bdb15ef0df7f33d3064b62db0909ec4 HTTP/1.1
> GET /git-repos/abdl.git/objects/89/ed7c1d085324423f1018b55ae447a8269d717e HTTP/1.1
> GET /git-repos/abdl.git/objects/d9/0a0becf445cc4a0db74cdd0cc5e44133ced048 HTTP/1.1
> GET /git-repos/abdl.git/objects/cf/adb3cd191d1af529af6775b8b2f3491b7a1f59 HTTP/1.1
> GET /git-repos/abdl.git/objects/e1/22005c39752197c722708a363236b20d38c666 HTTP/1.1
> GET /git-repos/abdl.git/objects/ca/4d3efeb3ada7075410630e6f2820f176d13a61 HTTP/1.1
> GET /git-repos/abdl.git/objects/60/bce5a1d6fc049d40d0f0a7f946a65c35554a2e HTTP/1.1
> GET /git-repos/abdl.git/objects/de/58b9b5c1ab40f586eea107bb2ed984dec948a5 HTTP/1.1
> GET /git-repos/abdl.git/objects/09/9e8f957f3c969c148e6106bf50787d471c0d04 HTTP/1.1
> GET /git-repos/abdl.git/objects/f8/b8c80d13d6fd4350458533e61fd21fe8e5d1c1 HTTP/1.1
> GET /git-repos/abdl.git/objects/03/b9cf3e2105dfe0dda0470cf03926c89180340f HTTP/1.1
> GET /git-repos/abdl.git/objects/0d/164950a84a8e172c592eb5d37a5dc817cd3cf6 HTTP/1.1
> GET /git-repos/abdl.git/objects/f2/b77e7f73b443cbbbada16d9ce6fa28233ddd22 HTTP/1.1
> GET /git-repos/abdl.git/objects/4f/5241f6cb55a8afbe46c849b96a75e395e30ea0 HTTP/1.1
> GET /git-repos/abdl.git/objects/9c/ed95c0441c4572c3b02d88efd203ef95b8b73a HTTP/1.1
> GET /git-repos/abdl.git/objects/c0/55c6ae3d545b928fbb2b77e6f0ddc31fac84f8 HTTP/1.1
> GET /git-repos/abdl.git/objects/80/3b633a4920a80396098f57483ae9883e4e5824 HTTP/1.1
> GET /git-repos/abdl.git/objects/f6/34e813f264240dcf4cf742fb27f93fda9006c3 HTTP/1.1
> GET /git-repos/abdl.git/objects/1b/26ad799217af7e187fdae78e862a6bf46e5591 HTTP/1.1
> GET /git-repos/abdl.git/objects/d8/cb86c7639c8b5b36e5d6898acee9bce6d7ad83 HTTP/1.1
> GET /git-repos/abdl.git/objects/17/2e0a4bcf176b8ba96eb97f0410b87545d26219 HTTP/1.1
> GET /git-repos/abdl.git/objects/1a/83ed408bda6dd94275468c6c62c2afc7127fe0 HTTP/1.1
> GET /git-repos/abdl.git/objects/5d/96459ac6ab9ff89c17f1df7ccc35efb0a5cc98 HTTP/1.1
> GET /git-repos/abdl.git/objects/02/efd3875bb2c3f45b297b6409204e8b0138251f HTTP/1.1
> GET /git-repos/abdl.git/objects/c8/fb32aa865cc3db16311bd6524c6f22ea9ab93c HTTP/1.1
> GET /git-repos/abdl.git/objects/ee/305066b729a8859cbbc038425d2e4d5623fefb HTTP/1.1
> GET /git-repos/abdl.git/objects/15/7745c4a42bb2f05945beb6ba9fbe5d8aa6b12a HTTP/1.1
> GET /git-repos/abdl.git/objects/39/df4a049ce23e9fd50e6e390d84c2f63564fcf5 HTTP/1.1
> GET /git-repos/abdl.git/objects/c8/53a49df35a1b582d7e5e800fa3b42e0d16efca HTTP/1.1
> GET /git-repos/abdl.git/objects/de/5f53aa8cf6326df79cc20aade56104fbea9827 HTTP/1.1
> GET /git-repos/abdl.git/objects/73/41341afce5825b14c417a3714a26c69ab1447b HTTP/1.1
> GET /git-repos/abdl.git/objects/84/b7d4cb8d9ad36fa637ce6812b87ff94562f2fc HTTP/1.1
> GET /git-repos/abdl.git/objects/ce/213f201e6bd70965531eda4346883259d987ba HTTP/1.1
> GET /git-repos/abdl.git/objects/0f/74bd87a23b515b45da7e6f5d9cc82380443dab HTTP/1.1
> GET /git-repos/abdl.git/objects/72/cd04bc0b3375f060756f03160b1f4165537622 HTTP/1.1
> GET /git-repos/abdl.git/objects/08/0ab3feaa856a8b973de3a5bfe9e477220d3cd3 HTTP/1.1
> GET /git-repos/abdl.git/objects/83/3aad3f457461d7aa81cae843fe91d96d191547 HTTP/1.1
> GET /git-repos/abdl.git/objects/ba/c1257b0e4639fe0454ad47e80d69fe80489ff6 HTTP/1.1
> GET /git-repos/abdl.git/objects/4b/96af709001f62d40fa68c6dd508f89c968fb3e HTTP/1.1
> GET /git-repos/abdl.git/objects/93/17643b2792bff6beab8a455bf2f06604c39a18 HTTP/1.1

As you can see, it tries to start with a smart http client, but the server ignores the smart part and speaks the dumb protocol. It then proceeds to grab the loose objects one by one.

It sounds like you'll want an API that caches objects locally in indexedDB grabbing objects as needed on demand and renders some form of UI from the API that reads from the object cache.

But I also did a quick test from several domains and unless your web client is the same domain, your fetch requests will be blocked by CORS:

Access to fetch at 'https://soniex2.autistic.space/git-repos/abdl.git/info/refs' from origin 'https://exploder.creationix.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

[creationix]

Note that you currently can't even request from your https://ganarchy.autistic.space subdomain because of it's CSP settings.

> await fetch("https://soniex2.autistic.space/git-repos/abdl.git/info/refs")
Error: Refused to connect to 'https://soniex2.autistic.space/git-repos/abdl.git/info/refs' because it violates the following Content Security Policy directive: "default-src 'none'". Note that 'connect-src' was not explicitly set, so 'default-src' is used as a fallback.

[SoniEx2]

wait, does CSP block no-cors requests?

OH it blocks fetch, got it. oops. :v

(I'd consider retrying without CORS and seeing if that works btw)

[creationix]

I can't use "no-cors" mode because that makes the response opaque (the body will always be empty).

It's not possible for a website on a different domain to clone from your server without your server adding CORS headers that allow it. This is a security mechanism of the web, not just some limitation of fetch.

[creationix]

For now I can work around the CORS issue in my browser with this extension https://chrome.google.com/webstore/detail/moesif-orign-cors-changer/digfbfaphojjndkpccljibejjbppifbc

But you'll need to update your server to add the headers (or run the web client on the same domain).

[creationix]

The CSP issue is something your current ganarchy subdomain is opting into and will also prevent the requests. You'll need to make sure your web front-end doesn't disallow requests to the domains with the git repos.

[creationix]

For reference, the CSP rule in question is set via the HTTP server in this header:

Content-Security-Policy: default-src 'none'; connect-src http: https: *:*; img-src 'self'; style-src 'self'; frame-ancestors *:*; base-uri 'none'; script-src 'self' 'unsafe-inline'; form-action 'none'

[SoniEx2]

I fixed the CSP but anyway uh.

so basically all I can do is provide a web UI for my own repos, but not allowed to provide a consistent user experience to ganarchy users?

[SoniEx2]

I'll take simple HTTP protocol even if it only works for my own repos tbh. the rest... we can figure out later. this CORS thing seems to be more about IP-based DRM than anything actually useful for security, especially since it doesn't do cookies either way, and can be easily bypassed with such a simple server-side script that I'm not sure why we don't have everyone running such a thing.

(I'm definitely gonna encourage ganarchy admins to run such a thing, fwiw)

[Jookia]

Hey @SoniEx2 I think you need to add CORS headers to your web server to allow other domains to read its contents.

[SoniEx2]

I don't need to add CORS to put a git browser on soniex2.autistic.space

[Jookia]

No, and here's why: By default Javascript is only allowed to request data from the same domain. This is called the same-site policy. So on soniex2.autistic.space it can request the Git stuff and it succeeds. But it can't request things on say, git.jookia.org. However, if I added CORS header on my git.jookia.org to say 'its ok to request this in Javascript from anywhere' then it would work.

[SoniEx2]

yeah, and that's not actually useful for securing anything.

it's just DRM. https://discourse.wicg.io/t/web-based-git-clients/4404

anyway this is off-topic

[Jookia]

I'm not saying it's useful for securing anything I'm just saying to get your Git code working you will need the responses by servers to have CORS headers.

[SoniEx2]

or I can just run it on the same domain as the repos I care about.

[Jookia]

That works too! You could also proxy the git requests if you wanted I suppose. (I think you mentioned this on Mastodon)

[SoniEx2]

You could also proxy the git requests if you wanted I suppose.

project for later. it'll piss a lot of ppl off and I'm not ready for that. anyway this is still off-topic.

[creationix]

I got a working sample here. https://github.com/creationix/http-clone/blob/master/src/main.ts

I ended up coding it from scratch mostly and pulling in a couple of files from js-git.

I'm having trouble with gh-pages caching. It works pretty well if I enable an extension to add CORS headers to ganarchy.

[creationix]

Ok, fixed the gh-pages issues and the demo is now live.

You can test it by going to https://creationix.github.io/http-clone/?https://soniex2.autistic.space/git-repos/abdl.git

Since I'm not on the same domain as soniex2, I needed to use an extension to add the CORS headers.

[creationix]

Video showing interactivity. https://www.youtube.com/watch?v=W7CVVxfne_I

[SoniEx2]

is there a CORS extension that calls itself "DRMTools" yet? (turns out associating DRM with DRM can get ppl to install all sorts of anti-DRM things)

[SoniEx2]

also this doesn't work in private browsing :joy:

[themightychris]

Extensions are all disabled by default in private browsing modes, but you can select which stay enabled in your browser's prefs

[SoniEx2]

I don't mean the extension. because @creationix is using I assume local storage, the git browser is not being allowed to use local storage in private browsing.

[creationix]

Indexed db should work in private browsing in chrome, I'm not sure if Firefox ever fixed their version.

But also, it wouldn't be hard to swap out the indexeddb cache with an in memory one.