SSH support in Pkg

[davidanthoff]

Julia 0.5 has removed all support for cloning private repos on Windows. In previous versions one could always clone private repos via SSH remote URLs, but support for the git protocol seems entirely removed. Cloning private repos via https also doesn't work on Windows, I get the following error:

ERROR: ccall: could not find function getpass
 [inlined code] from .\c.jl:89
 in #prompt#1(::ASCIIString, ::Bool, ::Any, ::ASCIIString) at .\libgit2\utils.jl:17
 in credentials_callback(::Ptr{Ptr{Void}}, ::Cstring, ::Cstring, ::UInt32, ::Ptr{Void}) at .\libgit2\callbacks.jl:56
 [inlined code] from .\refpointer.jl:32
 in clone(::ASCIIString, ::SubString{UTF8String}, ::Base.LibGit2.CloneOptions) at .\libgit2\repository.jl:189
 in #clone#98(::ASCIIString, ::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractPayload}, ::Any, ::ASCIIString, ::SubString{UTF8String}) at .\libgit2.jl:310
 [inlined code] from .\base.jl:111
 in clone(::ASCIIString, ::SubString{UTF8String}) at .\pkg\entry.jl:195
 in clone(::ASCIIString) at .\pkg\entry.jl:221
 [inlined code] from .\promotion.jl:229
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{ASCIIString}})() at .\pkg\dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{ASCIIString}}, ::ASCIIString) at .\file.jl:58
 in #cd#1(::Array{Any,1}, ::Any, ::Any, ::ASCIIString, ::Vararg{ASCIIString}) at .\pkg\dir.jl:31
 in clone(::ASCIIString) at .\pkg.jl:128
 in eval(::Module, ::Any) at .\boot.jl:236

I think there are potentially two solutions: 1) Re-enable support for the git transport protocol 2) Fix the error with https connections on Windows (might be enough to fix #8228)

I believe that only having 2) would be very painful, because my understanding is that every Pkg.update would for example query for the username/password combo for private repos, i.e. as far as I know there is no support for credential storage in libgit2, right?

So I think if the git transport is not re-enabled, one would really also need integration with the git credential manager interface, so that https credentials that are stored in a registered git credential manager would be picked up by Pkg.clone and Pkg.update. My understanding is that libgit2 has no support for this, so it would probably have to be implemented manually...

I think this issue here should get the 0.5.0 milestone attached. Not being able to work with private repos is a major regression relative to 0.4 that would affect essentially anyone who has package work in private repos.

[tkelman]

Could you open that branch as a WIP PR so it's easier to review?

[tkelman]

the only thing needed to merge https://github.com/JuliaLang/julia/pull/17391 and close this issue is for someone to independently test that branch (or binaries linked within the PR) and confirm that it works with a package over an ssh remote.

[tkelman]

reopening until agent etc works well

[wildart]

We need to disable ssh-agent request on windows. This will solve problem on windows.

[tkelman]

what should be used instead?

[EricForgy]

Whatever solution should allow us to Pkg.update() without entering our passwords 50 times (for 50 private repos), right? :pray:

[wildart]

Credentials are cached after first entry till the end of the update.

[EricForgy]

Ok. So it means we still need to enter our password once? That is not the end of the world, but is still a regression compared to v0.4. After setting up my SSH key one time, I never had to enter the password again on any update. Hmm...

[EricForgy]

Would this be the same for linux and OSX? They also need to enter password once every time they need to update?

[Keno]

Linux and OS X should use the agent, no password entry required. Why do we need to disable the agent on Windows? As far as I know libssh supports pageant, which some people may want to use.

[tkelman]

pageant is probably a better option to target at the moment since the win32 native version of the openssh command line utilities is still a WIP as I understand

[blakejohnson]

I just tested the new SSH support on macOS and I am getting errors, e.g.:

=> Package QSimulator cannot be updated.
GitError(Code:ERROR, Class:SSH, Failed to authenticate SSH session: Callback returned error)
 in macro expansion at ./libgit2/error.jl:98 [inlined]
 in #fetch#52(::Base.LibGit2.FetchOptions, ::String, ::Function, ::Base.LibGit2.GitRemote, ::Array{AbstractString,1}) at ./libgit2/remote.jl:70
 in (::Base.LibGit2.#kw##fetch)(::Array{Any,1}, ::Base.LibGit2.#fetch, ::Base.LibGit2.GitRemote, ::Array{AbstractString,1}) at ./<missing>:0
 in #fetch#91(::String, ::String, ::Array{AbstractString,1}, ::Nullable{Base.LibGit2.CachedCredentials}, ::Function, ::Base.LibGit2.GitRepo) at ./libgit2/libgit2.jl:159
 in (::Base.LibGit2.#kw##fetch)(::Array{Any,1}, ::Base.LibGit2.#fetch, ::Base.LibGit2.GitRepo) at ./<missing>:0
 in (::Base.Pkg.Entry.##39#45)(::Base.LibGit2.GitRepo) at ./pkg/entry.jl:425
 in with(::Base.Pkg.Entry.##39#45, ::Base.LibGit2.GitRepo) at ./libgit2/types.jl:660
 in update(::String, ::Set{String}) at ./pkg/entry.jl:415
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#update,Tuple{String,Set{String}}})() at ./pkg/dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#update,Tuple{String,Set{String}}}, ::String) at ./file.jl:59
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:31
 in update() at ./pkg/pkg.jl:210
 in eval(::Module, ::Any) at ./boot.jl:234
 in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:62
 in macro expansion at ./REPL.jl:92 [inlined]
 in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46

Is there some way to get more verbose information about what failed in SSH authentication?

[wildart]

@blakejohnson Do you use ssh-agent? What is the size of your ssh key? Corresponding libssh2 error for "Failed to authenticate SSH session: Callback returned error" is http://libssh2.sourceforge.net/doc/#libssh2errorpublickeyunverified

[blakejohnson]

I do use ssh-agent (it is launched by default in recent versions of macOS). My SSH key is 1.7 KB.

[wildart]

Are you sure that the correct key is used to access github?

[blakejohnson]

It's working for me now. I think the issue was caused by a redirect. I've moved one of my repositories and GitHub provides a redirect from the old URI. The new Pkg SSH support doesn't handle the redirect, but after updating the remote address, it is working fine.

[tkelman]

that seems like a bug that we should try to address (and should be opened separately), but probably not a release blocking one

[Keno]

Agreed on both points.

[aviks]

On a Mac, with today's master.

So I generated a package with PkgDev, set it's remote to a private git@.. url, pushed it with command line git. After that, Pkg.update() went through successfully. No password prompt at all (I use ssh-agent). I verified that it pulls in the changes from the repo.

I then Pkg.cloned two private packages with https://... urls. There was a password prompt for each, and the clone went thru successfully. A subsequent Pkg.update() threw up only one password prompt, even though there were two packages. No echoing in the password either.

All good so far.

However, Pkg.clone fails with a git@... url. It prompts me for a key file, and a passphrase. So its clearly not using the ssh-agent. So that's one issue. After I type in a passphrase, it then fails with:

julia> Pkg.clone("git@bitbucket.org:aviksengupta/testssh.jl.git")
INFO: Cloning testssh from git@bitbucket.org:aviksengupta/testssh.jl.git
Private key location for 'git@bitbucket.org' [/Users/aviks/.ssh/id_rsa]:
Passphrase for /Users/aviks/.ssh/id_rsa:
ERROR: GitError(Code:ERROR, Class:SSH, Failed to authenticate SSH session: Waiting for USERAUTH response)
 in macro expansion at ./libgit2/error.jl:98 [inlined]
 in clone(::String, ::SubString{String}, ::Base.LibGit2.CloneOptions) at ./libgit2/repository.jl:189
 in #clone#105(::String, ::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractPayload}, ::Function, ::String, ::SubString{String}) at ./libgit2/libgit2.jl:308
 in clone(::String, ::SubString{String}) at ./pkg/entry.jl:195
 in clone(::String) at ./pkg/entry.jl:221
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}})() at ./pkg/dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}}, ::String) at ./file.jl:59
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:31
 in clone(::String) at ./pkg/pkg.jl:151
 in eval(::Module, ::Any) at ./boot.jl:234
 in macro expansion at ./REPL.jl:92 [inlined]
 in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46

[Keno]

From my perspective, this is done.

[tkelman]

the changes have introduced some bugs though https://github.com/JuliaLang/julia/issues/17553

[blakejohnson]

This is back to not working at all for me. It seems like the hooks to ssh-agent aren't working, because I always get a prompt for my SSH key location. And even providing that I still get an SSH authentication failure error. I've tried both a 1024-bit DSA key and a 4096-bit RSA key. Same error.

[tkelman]

can you try https://github.com/JuliaLang/julia/pull/17586? @Keno what's your status there, how much are you intending to finish before rc1?

[Keno]

I suppose we can merge the first three commits and put off the tests for later. I've made some progress on the test case, but it's still flakey.

[tkelman]

It's not like we have any tests for it now. While we do really badly need some and we shouldn't let it remain untested for any longer than we absolutely have to, working and untested is a bit better than not working and untested.

[wildart]

Mbedtls does not support DSA keys. We probably can tweak mbedtls configuration to support for 4096-bit RSA keys.

[blakejohnson]

So, even dropping back to 2048-bit RSA doesn't work for me.

[wildart]

@blakejohnson are you on linux?

[blakejohnson]

No, Mac OS X 10.11.5.

[blakejohnson]

After #17586, Pkg.update() does work. However, I am still getting prompted for a passphrase for a key which ssh-agent knows about. Does that mean the interface with ssh-agent is still not functioning?

[tkelman]

how often do you get prompted? I don't know how long we cache the credentials for, or if we have it working in a way where we can let ssh-agent entirely handle the passphrase

[Keno]

passphrase should be entered when adding the key to the agent, so we shouldn't need to prompt for it.

[Keno]

Indeed ssh-agent authentication appears to be broken again. Will take a look.

[tkelman]

@blakejohnson have you had a chance to re-test that redirection issue again recently? Did you ever open a separate issue on it?

[blakejohnson]

No, I haven't tried that again. Though, given the number of other issues Keno has fixed in recent days, I wonder if I wasn't premature in identifying the temporary cause of success. I'll put that on my queue to test, and if it is still a problem, I'll file an issue.

[giordano]

I do experience the problem of being prompted for the passphrase of the wrong key (GitHub key is not the default one), while I don't have this issue with other applications using SSH (nor I have with Julia 0.4.6). I'm running Julia Version 0.6.0-dev.629, commit 155147e (2016-09-14 17:49 UTC), should I file a new issue?

[iamed2]

@giordano Do you have that key set in your SSH config file? Unfortunately, libssh2 (and consequently libgit2) cannot use the SSH config file information.

[giordano]

Yes, I have. Am I hopeless? At least I have to type the password just once ;-)

[iamed2]

@giordano You and I are both in the same crappy boat, I'm afraid. I expect it won't be fixed unless libssh2 starts parsing SSH config files.

[EricForgy]

Although inputting a password once is not the end of the world, it is not completely symmetric if linux/OSX do not need to input passwords. Until this is fixed for Windows, I almost say we should break it for linux/OSX too, just to be fair and so it doesn't drop off the radar (given this issue is closed - presumably in a rush so they could release 0.5 - although it doesn't seem 100% satisfactory).

[EricForgy]

Windows should not be a second class Julia citizen.

[tkelman]

What leads you to think this is Windows-specific?

[EricForgy]

Ignorant assumptions? :)

[tkelman]

windows is kind of a second class citizen with ssh usability in general apart from julia. we try not to intentionally break things on any platform, but if there are remaining improvements to make on any platform we should have issues to track them (see https://github.com/JuliaLang/julia/issues/17541, and there are a few already open and being tracked)

[iamed2]

@EricForgy I'm on OS X. I actually don't have to input a passphrase as my key doesn't have one, but I have to type the full path to my key every time.

[IssamT]

On OSX el capitan 10.11.6 I am unable to get Pkg.clone() working on julia 0.5

http://stackoverflow.com/questions/39849236/pkg-clone-changed-on-julia-0-5

[BoundaryValueProblems]

With julia 0.5.1, Pkg.clone for private repos works on Mac OS X (El Capitan 10.11.6) and Ubuntu/Linux (16.04 LTS), but on Windows 10, the situation is quite peculiar: it works with https while it fails using ssh with the following errors:

ERROR: ArgumentError: drive mismatch: METADATA git@gitlab.com:...
 in joinpath(::String, ::String) at .\path.jl:82
 in clone(::String) at .\pkg\entry.jl:210
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}})() at .\pkg\dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}}, ::String) at .\file.jl:48
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at .\pkg\dir.jl:31
 in clone(::String) at .\pkg\pkg.jl:151

Hence, this issue should be reopened!

[yuyichao]

No, that's the windows path issue.

[StefanKarpinski]

Unrelated problem, addressed by this PR: https://github.com/JuliaLang/julia/pull/20914.