`dolt clone` should support DoltHub URLs

dolthub / dolt

Dolt – Git for Data

Apache License 2.0

17.72k stars 501 forks source link

`dolt clone` should support DoltHub URLs #3874

Open fulghum opened 2 years ago

fulghum commented 2 years ago

Git allows users to run git clone with GitHub URLs, but dolt clone doesn't support specifying a url, only the <org name>/<database name> format.

To follow Git's semantics, dolt clone should support cloning from DoltHub and DoltLab URLs (e.g. dolt clone https://www.dolthub.com/repositories/jfulghum/test).

VinaiRachakonda commented 2 years ago

I have some code ready to support this but curious about whether we should extend to remote:

Git can allow something like this

git remote add origin-two https://github.com/user/repo.git

Do we want to also support?

git remote add origin-two https://www.dolthub.com/repositories/jfulghum/test

We could open a separate issue and go from there

reltuk commented 2 years ago

This is interesting.

The repository URL on GitHub is actually a fully-fledged HTTP git remote, so it supports clone and being used as a remote without any special processing.

If we want to support URLs of the dolthub.com format being fully fledged remotes, we can do a little client work and some infra work. In particular:

Client-side, we need to account for and strip off the repositories prefix when we're handling the path. https://github.com/dolthub/dolt/blob/main/go/libraries/doltcore/remotestorage/chunk_store.go#L119 . Probably something as simple as: if len(tokens) == 3 && tokens[0] == "repositories" { tokens = tokens[1:] }, with a lingering TODO to eventually fix this.

Server-side, we can deploy ingress changes to land https://www.dolt.com/dolt.services.remotesapi.v1alpha1.ChunkStoreService/* on doltremoteapi instead of 404ing.

VinaiRachakonda commented 2 years ago

@reltuk What is that last link? I'm not able to access it.

fulghum commented 2 years ago

Very cool @reltuk. I'm assuming we do the same infra change on DoltLab to support using DoltLab URLs, too, right?

Also, just found another related issue with a customer looking for the same behavior: https://github.com/dolthub/dolt/issues/1556

VinaiRachakonda commented 2 years ago

nice catch on that existing issue @fulghum. Sounds like the immediate game plan is to get some client facing work in that maps dolthub.com to doltremoteapi and support clone operations and wait for full remote support until we have a sense of the infrastructure work.

VinaiRachakonda commented 2 years ago

Going to leave this as open until we have some sort of resolution on dolthub remotes.