[DOCS] What are the steps to clone a PRIVATE repository from GitLab? #52

Closed luxlogica closed 4 years ago

I do not seem to be able to connect to private repositories in Gitlab.

I can connect successfully with other git clients - Tower and Fork. Oddly, GitFinder does not ask me for any account details - other git clients usually ask for a 'username' and an 'access token' (issued by GitLab).

Is GitFinder able to connect only to public repositories? If not, what are the steps to clone a PRIVATE repository?

Hi @luxlogica,

Here's a bit longer explanation of what you should do and how to use GitFinder in such cases.

Oddly, GitFinder does not ask me for any account details - other git clients usually ask for a 'username' and an 'access token' (issued by GitLab).

GitFinder does't connect yet to some popular repository hosting servers, that's planned for version 1.5. However, that's really not necessary to work with private repositories. It will only bring some convenience of quickly choosing repository to clone from the list or repositories on those servers, incorporate pull requests and similar.

Is GitFinder able to connect only to public repositories?

Certainly not! This is the story. I'll keep mentioning GitLab, since that's the server you referred to in your question, but the story applies to any other repository hosting service. There are basically two secure protocols (there are more, but let's keep it simple) to authenticate and work with repositories: HTTPS and SSH. You can access every repository on GitLab, no matter public or private, using two different URLs. For HTTPS, the URL looks something like: https://gitlab.com/username/reponame.git. For SSH, the URL looks something like git@gitlab.com:username/reponame.git. You can see them both if you click the Clone button in repositories web page on GitLab: repo_urls If you decide to work with HTTPS, you can clone public repositories right away, you won't be asked for any authentication info. This is how it looks like in GitFinder. If you want to clone a private repository using HTTPS, then GitFinder needs authentication. It will look into your Keychain to try to find any existing password there for that server (gitlab.com) and if any is found it will try to authenticate using the password. In none of the passwords is a match, or no password is found for the server, GitFinder will ask for username and password, with the option to save credentials in the Keychain. This is how it looks like. If you decide to save the password in the Keychain, GitFinder will reuse it in further attempts to authenticate and you won't be bothered with it anymore.

Please note that if you have two-factor authentication enabled and use HTTPS for cloning (and working) with repository, the above combo username/password will NOT work. You need to set Personal Access Token: Screen Shot 2020-02-22 at 18 06 21 Once you've done that, you use that token INSTEAD of password! So in the case of previous video above, you enter your username in the username filed and you enter personal access token in the password field. As before, you can save those credentials in the Keychain with GitFinder.

The situation is a bit different if you decide to use SSH as authentication method. Before you can do that, you must add your public SSH key to GitLab: Screen Shot 2020-02-22 at 17 17 22 You need to create a pair of private and public SSH keys (if you don't have one already) and then add the public key to GitLab, and keep the private key secret and known only to you. How to do it is outside the scope of this explanation, but you can find all information here.

You need to add a public SSH to GitLab, regardless whether you work with public or private repositories, SSH key is mandatory! Once you added a SSH public key to GitLib, you can use its private companion to authenticate using SSH in GitFinder. When you do it for the first time, GitFinder will ask you for the location of the file storing the private SSH key (conveniently positioning itself into ~/.ssh directory, where those keys are usually stored) and if the key is password protected, it will then ask you for a key password. This is needed due to application sandbox requirements and this is how it looks like. Again, you can opt to save the password into the Keychain. If the authentication succeeded (correct key and correct password if needed, remember the key may not be password protected), GitFinder will remember the location of used key in its preferences and next time you authenticate you won't be bothered with selecting the key again.

And that's how you work with private (and public) repositories on GitLab (or any other server). I hope everything is clear and in case you have further questions, don't hesitate to shout out :-)

Thank you so much for the quick and thorough response, @milke! BTW: make sure you add this response to your docs - it's very helpful for newbies!

In my case, however, what is happening is that I can never get the dialogue that asks me for "username/password" (if using https) nor "ssh key" (if using ssh). here is what happens: I enter the clone URL, and click the 'Clone' button. I then see the 'validating' progress bar, which then gets quickly replaced by 'authenticating', and then I just get the message you saw in my original post above: "remote: The project you were looking for could not be found."

So, I never get asked for a password, or for an ssh key...

1. Which protocol do you use when trying to clone that repository: HTTPS, of SSH? 2. Does that happen with every private repository on GitLab, or only with that particular one?

I had to go and test it out, just to confirm - here is what is happening:

it happens with BOTH ssh and https - I get the exact same message (I never get asked for username/password or ssh key)
I've tried it with 3 private repositories, and it happens with all 3. One repository is in a 'group' namespace, and the other 2 use different user accounts - so I know it's not an issue with a specific namespace or account, either.

I hope this helps...

I can't reproduce it with any of my private repos on GitLab, although they all are just simple test projects. Hence, can you add me as a team member (developer) to some of those private projects/repos you have this problem with. I don't really care what files are in there and I couldn't care less to peek inside, I just want to be able to reproduce the problem.

If you're still reluctant to provide such access, can you create a new private "junk" project/repo, confirm you can reproduce the problem with it as well and then add me as a "developer" to the list of project members.

My GitLab username is dmilke.

@milke Thank you for your help - I've given you access to a couple of the private projects. Let me know how you go!

@luxlogica, unfortunately (or not?) I don't have any problem cloning those (private) repos you provided access to. See cloning of that superstudent repo using SSH and HTTPS. You can see even submodules (animejs and uikit) were properly cloned and initialised: Screen Shot 2020-02-25 at 00 50 36 Also, comingsoon repo was successfully cloned using SSH and HTTPS.

Let's first try to figure out if the problem you see is related to authentication or cloning. Clone one of those repos, e.g. superstudent using SSH and with terminal command:

git clone git@gitlab.com:i.couto/superstudent-2019.git

After the repo is cloned, open it in GitFinder (show repo browser or whatever) and try to fetch/pull/push something? Does it work? Do you get to the point you're asked for SSH key and password? Or you get any error?

Thank you for doing all this testing, @milke! Unfortunately, on my end it's reproducible every time. Here is a little bit more information - and a little bit more testing - hoping it might help:

1) My account in GitLab is setup with 2FA. I have uploaded a valid SSH key, and issued Personal Authentication Tokens for the other git clients I'm testing with (Tower and Fork).

2) AFAIK when a repository is PRIVATE, if we are trying to access it via SSH, we need to use the ssh key to authenticate, otherwise GitLab won't accept the access. If we are using HTTPS, then GitLab expects the app to use a Personal Authentication Token. The Personal Authentication Token is issued by GitLab, and supposed to be unique for each app that the user has accessing GitLab. Each app is supposed to remember it - so the git clients usually have a place in 'Account Settings' somewhere that allows us to enter the token for a GitLab account.

3) I've entered the GitLab account details in both Tower and Fork, and everything works without issues when using these clients - I can perform all git operations, and the clients don't complain or give me any errors. Using both clients, I cloned the 'comingsoon' and 'superstudent' repositories without problems - one using ssh, the other using https.

4) I then tried using GitFinder with the cloned repositories. I can indeed open the GitFinder Browser, navigate the history, and even commit changes locally. But when I try to PUSH any changes to the remote of either repo, I get the same message...:

TBH, I can't see how GitFinder will be able to work with a private repository, if it has no knowledge of my SSH key or Personal Access Token - but maybe there is something I'm missing...

OK, this is going to be rather long post in an attempt to clarify some apparent confusion about what is and isn't possible with private repos on popular hosting services and then to focus on an actual problem Hopefully will that reduce unrelated noise and discussion.

Secondly, to make things more concise, let's just concentrate on SSH authentication for the time being, once that's fixed, HTTPS will (most probably) magically get fixed too.

I then tried using GitFinder with the cloned repositories. I can indeed open the GitFinder Browser, navigate the history, and even commit changes locally. But when I try to PUSH any changes to the remote of either repo, I get the same message…

Okay, at least we know the problem is related to authentication and not cloning.

I can't see how GitFinder will be able to work with a private repository, if it has no knowledge of my SSH key or Personal Access Token - but maybe there is something I'm missing…

It DOES have a knowledge of your SSH key(s) and I've explained that above and also shown in those linked videos, but I'll try to explain in more details and also clear confusion about other clients knowing or not knowing things… And here comes that long story…

First of all, in order to work with private (or public for that matter) repos with e.g. Fork (the same applies to Tower and others, I'll concentrate on Fork for clarity), you do NOT need to ever connect them to your GitLab (or GitHub or others, again for clarity I'll concentrate on GitLab) account and enter account details. That integration with GitLab is just a convenience, enabling you to do some things easier and faster, but it's not a requirement. For what it's worth, people may have repos on some other private servers, for which Fork has no integration at all. Moreover, having that integration and connecting Fork to GitLab is NOT the thing enabling Fork to successfully authenticate with the repo using SSH. That is actually done by the process named ssh-agent, which manages your SSH private/public keys, ssh configurations etc. Fork just uses ssh-agent under the hood to do the job (actually, it uses native git command line tool, which in turn uses ssh-agent. Hence you can authenticate repos using SSH from the terminal too).

Now, another apparent confusion is about SSH key(s). SSH uses asymmetric encryption. Hence, SSH keys come in pairs: private key and public key. When you add a new SSH key to GitLab, either going to gitlab.com in your web browser or doing it in Fork with that nice service hosting integration, you are actually adding a public key. The public key is stored in the server, it's exposed (in a way). However, when Fork authenticates with a repo to do some job, it actually uses corresponding private key, stored in your machine. That's why it's private, you keep it locally only for yourself and even if your public key gets compromised, nobody can use it to authenticate without corresponding private key. And how does Fork know which private key to use (since they aren't stored on server and not available from the GitLab integration)? I've already explained that, through ssh-agent process.

You can confirm all this above yourself, although you really don't have to, as it requires some time: create a completely fresh user account on your machine, login into that new account, create new SSH private/public pair of keys, go to GitLab and add that new public key to your GitLab account. Then start Fork (for the "first" time, as it's fresh user account), NEVER enter your GitLab account details in it and go clone some private repo using just plain URL, for example git@gitlab.com:i.couto/superstudent-2019.git. You'll see it all works, even though you never entered your GitLab account details in Fork.

Now that we're (hopefully) clear about private and public SSH keys, where they're stored and what Fork (and others) (doesn't) know, let's see how it all works in GitFinder. GitFinder can't use ssh-agent directly like others do, due to sandboxing restrictions (GitFinder is securely sandboxed application). Therefore user has to explicitly tell GitFinder where the file keeping a private SSH key to use is located. You can see how it works in any of the above videos showing SSH authentication. When GitFinder is freshly installed, it doesn't know about any (I repeat again, private) SSH key. So, the first time it tries to authenticate with some repo using SSH, it will present a file selector in a sheet (see those videos), asking you to select a file, which keeps the private key for authentication. If you select a wrong file (not containing private SSH key or it does contain the key but that key cannot be used with the server, meaning you didn't store its public companion on the server), GitFinder will keep asking for a file, until you get it right, or cancel the operation altogether. If you selected a correct file/key (private SSH keys may be password protected and in such case GitFinder will ask you for a password, see those videos), after the remote operation is successfully completed GitFinder will remember the location for the private key in its preferences, so you won't be bothered selecting it again. However, you may work with more servers and hence you may be using more that one private SSH key. When you try to authenticate with another server, GitFinder will try to reuse private key files it already knows about. If it works, then fine. If not, GitFinder will ask again for the location of another private key file and when everything finishes successfully, it will remember the location of that file too. And so on, as you use new private keys, GitFinder will ask you for each ONLY once, and then remember and reuse them later.

It's a bit longer explanation, but I hope know you understand how GitFinder knows of your SSH.

Now, let's concentrate on practical things to try to isolate the problem. Please do this:

1. Make sure GitFinder is not running.

2. Open terminal and execute this: defaults delete ag.zigz.GitFinder GFPrivateKeysURLs

3. Again in terminal execute this: defaults read ag.zigz.GitFinder

Launch GitFinder and in an empty directory try to clone one of your private repos, entering SSH type URL, for example: git@gitlab.com:kirbyzone/kz-comingsoon.git

What happens? Does GitFinder present a file selector sheet asking you to "Choose private key file for SSH connection with the server at "ssh://gitlab.com""? Screen Shot 2020-02-25 at 13 31 05 If any error happens, does it happen before or after this sheet is presented?

Thank you for your thorough response, @milke - it's appreciated. I did delete the preference using the defaults terminal command, and after doing that I finally got the dialog asking me for my ssh key. Once I selected the 'right' key, GitFinder was able to clone the repository without problems.

In your explanation, you indicate that programs like Tower and Fork merely choose to keep account information for user 'convenience', rather than necessity. I guess, most users only have a single account, possibly with a single service, so in that scenario, your explanation makes sense. It also makes sense that GitFinder should try to be 'clever' and 'guess' which ssh keys to use. There are use-cases, however, when this might get GitFinder in trouble - which is what I suspect was probably happening here.

What happens if the user - for whatever reason - starts using a news ssh key with a repository/service, but don't delete the old one (because it might still be used somewhere else)? What happens when the user has multiple accounts that can connect to the same repository - eg., an account as a company 'admin', and one as a 'developer', or perhaps when transferring repositories between different companies? All those situations are better served when the user is able to configure a preference in the app, which allows it to explicitly connect different 'accounts' with different ssh keys.

Lastly, I noticed that you sidestepped the issue of Personal Authentication Tokens altogether: obviously, for these to be supported, the git client MUST have account-specific preferences (as each account is supposed to have its own token). Tokens are meant to be stored in-app, and the user is only supposed to see the token once - when copying it into the app's preferences.

So, I did another test: I tried cloning another private repository, that I have in GitLab under a different account. GitFinder gave me the same error as before. If I then delete the preference - by running the defaults command again - and then try again, I get the dialog box asking for my ssh key, as expected.

So, while it's nice to have GitFinder try to 'automagically' guess which ssh key to use in certain situations, it may be safer - and cover more use-cases - if we actually had explicit preferences that allowed us to store account details, and connect ssh keys and authentication tokens, like the other git clients do.

What happens if the user - for whatever reason - starts using a news ssh key with a repository/service, but don't delete the old one (because it might still be used somewhere else)?

… or perhaps when transferring repositories between different companies?

These aren't a problem. If, for example, you start using a new key, GitFinder will first try all keys previously saved in its preferences and after they all fail, it will offer you a chance to select a new key in the file selector, as explained above, and after successful authentication, it will save that new one in the preferences as well and use it later.

What happens when the user has multiple accounts that can connect to the same repository - eg., an account as a company 'admin', and one as a 'developer'

Well, that's a problem for GitFinder 🙁. The one you hit and now that I've figured it out, I managed to reproduce it. So, having two (or more) accounts on the same server is a problem. What happens?

Let's say, on GitLab you have two accounts; user1 with a repo repo1 and you want to use SSH key key1 for it and user2 with a repo repo2 and you want to use SSH key key2. Let's say you've already used GitFinder authenticate user1, so the key1 is already saved in the preferences. When you try to use GitFinder to authenticate user2, its _"automagically" (as you called it) procedure to guess which SSH key to use demands it first tries all keys already saved in the preferences. Among them is the key1, so eventually it will be used and it will successfully authenticate agains the server, but user1, not user2. And since user1 doesn't have the repo repo2, GitFinder (correctly) report error that "the project could not be found", Now that we know what's going one, that sentence makes perfect sense.

Even if you somehow managed to save the key2 in the preferences, that wouldn't help. GitFinder tries all keys in the preferences in the order they're saved, so the key1 would always come first (wrongly used to authenticate user2), the key2 would never get its chance.

Implementing git hosting services integration (planned for version 1.5) will fix this issue, as you've suggested, but I also need to think of some more general solution, which will work with custom servers (for example, people having git repositories on server hardware in their own companies) as well. That will probably end in having some editable list/table of servers (URLs or IP addresses) and SSH keys meant to be used for those servers.

Lastly, I noticed that you sidestepped the issue of Personal Authentication Tokens altogether

There isn't really anything special related to Personal Authentication Token (PAT). They are used when you use HTTPS for remote connection and they're used in exactly the same way as user password. Just instead of a username/password combo, you use username/PAT. If you don't have 2 Factor Authentication enabled and you use HTTPS, you can use either password or PAT, but if you enabled 2FA, then you must use PAT.

In case of HTTPS, regardless of whether a password or a PAT is used, GitFinder exhibits the similar problem when having two or more accounts on the same server. The way GitFinder works with HTTPS is to look into the Keychain and search all existing credentials for a given host and then use them one by one until one succeeds to authenticate. And if all fail, it offers you chance to enter a username and password/PAT and optionally save it in the Keychain.

So in the case similar to the one above (user1 and user2) it all depends which credentials will GitFinder find first in the Keychain. If, for example, it's for user2, but we want to work as user1 on repo1, GitFinder effectively authenticates the wrong user and will fail to find repo1

I'll try to fix this even before the full integration with hosting services integration comes out. I'll keep you posted.

Hey @luxlogica, I think I have developed a (part of) solution for this problem. I didn't include it in the 1.4.1 released two days ago, because I want it tested a bit more, but you can try it in the latest beta 1.4.2b1 which you can download here, or you can update your current copy automatically, just choose the Beta channel in the update preferences.

I dived into code of both libssh2 and libgit2 libraries and modified them slightly to distinguish the cases of interest here. Then I also changed GitFinder code to accommodate these changes. And now it all should (hopefully) work seamlessly, no matter now many different accounts you have on a single server, whether you use different SSH keys or not and whether you use SSH or HTTPS authentication. I'd appreciate if you give it a try and use it in the next few weeks and if you don't find any problems, I'll release it officially. If you find a problem, please report it here.

There's only one case now that GitFinder can't quite properly handle; trying to work with non-existing repository. If you, for example, try to clone a non-existing repo, GitFinder will try authenticating with all known SSH keys in preferences (if you use SSH), or all credentials for that server remembered in the Keychain (if you use HTTPS) and finally will offer you to select a new SSH key file (if you use SSH), or to enter new username/password combo (if you use HTTPS). And it will keep asking you that until you finally cancel the operation. Before, GitFinder would tell you whatever the server returned (if you used SSH, GitHub returned "ERROR: Repository not found.", whereas GitLab returned that message with a lot of equation marks and "The project you were looking for could not be found.") or "unexpected HTTP status code: 404" (if you used HTTPS). Now, it will just keep asking for SSH key / userpass until you cancel.

Here is why that happens; I'll use HTTPS for explanation, but the similar applies to SSH. Let's say (again), on GitLab you have two accounts; user1 and user2, both with username/password remembered in the Keychain. You try to clone non-existing repo with account user2. GitFinder finds all credentials in the Keychain for server GitLab. Let's say it first finds and tries credentials for user1. It manages to authenticate with those and then goes to clone the repo. Since the repo doesn't exist for user1, GitFinder actually internally recognised that (HTTPS 404), but it can not know if you wanted perhaps to clone that repo with user2 and maybe (again, GitFinder cannot know) there is such a repo in user2's account. Hence, it continues authenticating with the next credentials found in the Keychain for server GitLab, and that is for user2. Again, It manages to authenticate the user with those new credentials and again goes to clone the repo. But again, the repo doesn't exist, GitFinder actually recognised that (HTTPS 404) but again, it cannot know if you perhaps have yet another account with the same server (GitLab), which credentials aren't saved in the Keychain, and maybe the repo exists for that eventual user. Hence, GitFinder offers you to enter username/password for that eventual new account. I know this is a bit tricky to explain (and being nonnative English speaker certainly doesn't help), but I hope you understand what I'm trying to say.

I tried to clarify this possible situation by now stating the full repository URL/path in dialogs asking for SSH key and username/password, like in the pictures below: Screen Shot 2020-03-02 at 12 19 03 So hopefully, if a user keeps getting asked about key/password he/she would eventually notice the URL/path is wrong and cancel the operation. I believe this is a small price to pay in unlikely cases to get everything "automagically" working in general. There is perhaps a way to solve this more elegantly, but I couldn't think of one at the moment.

Please note, when version 1.5 introduces connectivity with popular repo hosting services, GitFinder will offer possibility to specify SSH key to use for particular accounts. That configuration will have a preference than all "automagical" guessing explained above, but I still think what I did in this beta is a good thing because:

1. People may have different accounts on some custom server (in a company, for example) not offering all that integration like GitHub/GitLab/other does. And I want GitFinder to seamlessly work in those situations too.

2. Even with hosting providers integration, one can still not/forget to configure SSH key correctly, like seen here in Fork: Screen Shot 2020-03-02 at 12 25 26 In case of such misconfiguration, I want GitFinder to fall back to this "automagical" guessing, not to simply fail, like e.g. Fork (I just wanted to fetch remote of the existing repo here, but with misconfigured SSH key for connected GitHub account, and this happened):

Screen Shot 2020-03-02 at 12 33 47

Sorry about the long story and please try this beta and see how it behaves with multiple accounts, keys and whatever configuration(s) you may have.

Your explanation is pretty clear, @milke! Thank you for your continuing work, and for trying to make things 'automagical' in GitFinder.

@luxlogica, no problem. Just please, let me know after 2 - 3 weeks if everything works as expected, I'd really appreciate that. If so, I'll make it official version.

This is now officially in version 1.4.2 and it seems to be working just fine. If you experience any further troubles regarding authentication when working with more accounts on the same server, feel free to reopen.

milke / GitFinder-Issues

[DOCS] What are the steps to clone a PRIVATE repository from GitLab? #52