iterative / dvc

🦉 ML Experiments and Data Management with Git
https://dvc.org
Apache License 2.0
13.6k stars 1.18k forks source link

import: failing with SSH key-based authentication from a GitHub repo #9737

Closed wjn0 closed 1 year ago

wjn0 commented 1 year ago

Bug Report

Description

I have a private github repo username/repo.git created from following the data registry tutorial on my own data. I am trying to import a file from it like dvc import git@github.com:username/repo.git filename and it is failing on an ssh error.

Reproduce

  1. Create directory tmprepo and cd tmprepo
  2. git init
  3. dvc init
  4. dvc import git@github.com:username/repo.git filename I have also tried adding ssh://... protocol indicator and a few other modifications, no dice)
Output with `-v`: ``` 2023-07-14 10:37:43,026 DEBUG: v3.5.1 (pip), CPython 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27 2023-07-14 10:37:43,027 DEBUG: command: /home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/bin/dvc import -v ssh://git@github.com/githubusername/projectname_data.git raw -o data/raw 2023-07-14 10:37:43,529 DEBUG: Removing output 'data/raw' of stage: 'data/raw.dvc'. 2023-07-14 10:37:43,530 DEBUG: Removing '/storage/dhi_work/username/projectname_preprocessing/data/raw' Importing 'raw (ssh://git@github.com/githubusername/projectname_data.git)' -> 'data/raw' 2023-07-14 10:37:43,547 DEBUG: Computed stage: 'data/raw.dvc' md5: 'a5b888cce3b0c91f50f9272fecd11db3' 2023-07-14 10:37:43,547 DEBUG: 'md5' of stage: 'data/raw.dvc' changed. 2023-07-14 10:37:43,548 DEBUG: Creating external repo ssh://git@github.com/githubusername/projectname_data.git@None 2023-07-14 10:37:43,548 DEBUG: erepo: git clone 'ssh://git@github.com/githubusername/projectname_data.git' to a temporary dir 2023-07-14 10:37:43,877 ERROR: failed to import 'raw' - SCM error: Failed to clone repo 'ssh://git@github.com/githubusername/projectname_data.git' to '/tmp/tmplhjcufk9dvc-clone': Authentication failed for: 'git@github.com:22': Permission denied Traceback (most recent call last): File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 204, in _run_command conn = await asyncssh.connect( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/asyncssh/connection.py", line 8093, in connect return await asyncio.wait_for( ^^^^^^^^^^^^^^^^^^^^^^^ File "/lmod/packages/python/3.11.3/lib/python3.11/asyncio/tasks.py", line 442, in wait_for return await fut ^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncssh.misc.PermissionDenied: Permission denied The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 220, in clone repo = clone_from() ^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/porcelain.py", line 514, in clone return client.clone( ^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 703, in clone result = self.fetch(path, target, progress=progress, depth=depth) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 781, in fetch result = self.fetch_pack( ^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 1097, in fetch_pack proto, can_read, stderr = self._connect(b"upload-pack", path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 1746, in _connect con = self.ssh_vendor.run_command( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 121, in wrapper return sync(self.loop, func, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 106, in sync raise return_result File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 61, in _runner result[0] = await coro ^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 217, in _run_command raise AuthError(f"{username}@{host}:{port or 22}") from exc scmrepo.exceptions.AuthError: Authentication failed for: 'git@github.com:22' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/scm.py", line 160, in clone git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/__init__.py", line 142, in clone backend.clone(url, to_path, **kwargs) File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 225, in clone raise CloneError(url, to_path) from exc scmrepo.exceptions.CloneError: Failed to clone repo 'ssh://git@github.com/githubusername/projectname_data.git' to '/tmp/tmplhjcufk9dvc-clone' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/commands/imp.py", line 17, in run self.repo.imp( File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/imp.py", line 6, in imp return self.imp_url(path, out=out, erepo=erepo, frozen=True, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/__init__.py", line 64, in wrapper return f(repo, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/scm_context.py", line 151, in run return method(repo, *args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/imp_url.py", line 85, in imp_url stage.run(jobs=jobs, no_download=no_download) File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() ^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 600, in run self._sync_import(dry, force, kwargs.get("jobs", None), no_download) File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() ^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 637, in _sync_import sync_import(self, dry, force, jobs, no_download) File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/imports.py", line 57, in sync_import stage.save_deps() File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 491, in save_deps dep.save() File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/dependency/repo.py", line 58, in save rev = self.fs.repo.get_rev() ^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 427, in repo return self.fs.repo ^^^^^^^ File "/lmod/packages/python/3.11.3/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 416, in fs return _DVCFileSystem(**self.fs_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__ obj = super().__call__(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 128, in __init__ repo = self._make_repo(url=url, rev=rev, subrepos=subrepos, **repo_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 188, in _make_repo with Repo.open(uninitialized=True, **kwargs) as repo: File "/lmod/packages/python/3.11.3/lib/python3.11/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 30, in _external_repo path = _cached_clone(url, rev, for_write=for_write) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 154, in _cached_clone clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/flow.py", line 246, in wrap_with return call() ^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 222, in _clone_default_branch git = clone(url, clone_path) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/scm.py", line 165, in clone raise CloneError("SCM error") from exc dvc.scm.CloneError: SCM error 2023-07-14 10:37:43,889 DEBUG: Analytics is enabled. 2023-07-14 10:37:43,950 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpmy3np8c0']' 2023-07-14 10:37:43,952 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpmy3np8c0']' ```

Expected

I expect to be prompted for my SSH key password and then I expect filename to be imported to the repo.

Environment information

Initially encountered under Poetry but confirmed it occurs outside of Poetry as well. Tried downgrading to dvc 2.9.5 based on another github bug report but failed on an import so I think that version of the package is just too old for me to try. I tried adding dvc[ssh] and this did not work (same error).

Output of dvc doctor:

$ poetry run dvc doctor
DVC version: 3.5.1 (pip)
------------------------
Platform: Python 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27
Subprojects:
        dvc_data = 2.5.0
        dvc_objects = 0.23.0
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.0.4
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.4.1)
Config:
        Global: /home/username/.config/dvc
        System: /etc/xdg/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: nfs on nfs:/nfs_path
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/435bb81e569bac9d0867f3b2c02bd890

Additional Information (if any):

The remote is local for now but given that it fails before prompting for my SSH key password I don't think that's relevant.

dberenbaum commented 1 year ago

I can confirm this behavior happens when a passphrase is required and the AsyncSSHVendor is used. @pmrowla could you take a look?

pmrowla commented 1 year ago

@wjn0 can you try the linked PR and verify that it works for you (you should be able to import using either the git@github.com:... URL after installing the patch)

From your DVC environment:

pip install 'scmrepo @ git+https://github.com/iterative/scmrepo.git@refs/pull/255/head'
wjn0 commented 1 year ago

Thanks so much for the prompt repro and the update toward a fix!

@pmrowla I get the same traceback as before. Updated dvc doctor output:

DVC version: 3.5.1 (pip)
------------------------
Platform: Python 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27
Subprojects:
        dvc_data = 2.5.0
        dvc_objects = 0.23.0
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.0.5.dev5+g4980cba
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.7.0)
Config:
        Global: /home/username/.config/dvc
        System: /etc/xdg/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/75c1f1aa060b9e2589889fc41a39e68e

I ran the same dvc import git@github.com:... as before. To confirm the repo is cloneable with my current set of keys, I did git clone git@github.com:... and it correctly identifies my keypair in ~/.ssh before cloning the repo. Let me know if I can try anything else, and thanks again!

Edit: Adding the traceback below as it is not in fact identical: ``` 2023-07-20 16:24:07,713 DEBUG: v3.5.0 (pip), CPython 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27 2023-07-20 16:24:07,713 DEBUG: command: /home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/bin/dvc import -v git@github.com:username/projectname_data.git raw -o data/raw 2023-07-20 16:24:08,229 DEBUG: Removing output 'data/raw' of stage: 'data/raw.dvc'. 2023-07-20 16:24:08,229 DEBUG: Removing '/storage/dhi_work/walter/projectname_preprocessing/data/raw' Importing 'raw (git@github.com:username/projectname_data.git)' -> 'data/raw' 2023-07-20 16:24:08,235 DEBUG: Computed stage: 'data/raw.dvc' md5: '5656116a17910fc725358de2d15f5bc9' 2023-07-20 16:24:08,236 DEBUG: 'md5' of stage: 'data/raw.dvc' changed. 2023-07-20 16:24:08,236 DEBUG: Creating external repo git@github.com:username/projectname_data.git@None 2023-07-20 16:24:08,237 DEBUG: erepo: git clone 'git@github.com:username/projectname_data.git' to a temporary dir 2023-07-20 16:24:08,523 ERROR: failed to import 'raw' - SCM error: Failed to clone repo 'git@github.com:username/projectname_data.git' to '/tmp/tmp55hi15vsdvc-clone': Authentication failed for: 'git@github.com:22': Permission denied Traceback (most recent call last): File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 286, in _run_command conn = await asyncssh.connect( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/asyncssh/connection.py", line 8093, in connect return await asyncio.wait_for( ^^^^^^^^^^^^^^^^^^^^^^^ File "/lmod/packages/python/3.11.3/lib/python3.11/asyncio/tasks.py", line 442, in wait_for return await fut ^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncssh.misc.PermissionDenied: Permission denied The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 226, in clone repo = clone_from() ^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/porcelain.py", line 514, in clone return client.clone( ^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 703, in clone result = self.fetch(path, target, progress=progress, depth=depth) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 781, in fetch result = self.fetch_pack( ^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 1097, in fetch_pack proto, can_read, stderr = self._connect(b"upload-pack", path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dulwich/client.py", line 1746, in _connect con = self.ssh_vendor.run_command( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 121, in wrapper return sync(self.loop, func, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 106, in sync raise return_result File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/asyn.py", line 61, in _runner result[0] = await coro ^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 299, in _run_command raise AuthError(f"{username}@{host}:{port or 22}") from exc scmrepo.exceptions.AuthError: Authentication failed for: 'git@github.com:22' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/scm.py", line 160, in clone git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/__init__.py", line 145, in clone backend.clone(url, to_path, bare=bare, mirror=mirror, **kwargs) File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 234, in clone raise CloneError(url, to_path) from exc scmrepo.exceptions.CloneError: Failed to clone repo 'git@github.com:username/projectname_data.git' to '/tmp/tmp55hi15vsdvc-clone' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/commands/imp.py", line 17, in run self.repo.imp( File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/imp.py", line 6, in imp return self.imp_url(path, out=out, erepo=erepo, frozen=True, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/__init__.py", line 64, in wrapper return f(repo, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/scm_context.py", line 151, in run return method(repo, *args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/imp_url.py", line 85, in imp_url stage.run(jobs=jobs, no_download=no_download) File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() ^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 600, in run self._sync_import(dry, force, kwargs.get("jobs", None), no_download) File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() ^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 637, in _sync_import sync_import(self, dry, force, jobs, no_download) File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/imports.py", line 57, in sync_import stage.save_deps() File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/stage/__init__.py", line 491, in save_deps dep.save() File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/dependency/repo.py", line 58, in save rev = self.fs.repo.get_rev() ^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 427, in repo return self.fs.repo ^^^^^^^ File "/lmod/packages/python/3.11.3/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 416, in fs return _DVCFileSystem(**self.fs_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__ obj = super().__call__(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 128, in __init__ repo = self._make_repo(url=url, rev=rev, subrepos=subrepos, **repo_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/fs/dvc.py", line 188, in _make_repo with Repo.open(uninitialized=True, **kwargs) as repo: File "/lmod/packages/python/3.11.3/lib/python3.11/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 30, in _external_repo path = _cached_clone(url, rev, for_write=for_write) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 154, in _cached_clone clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/flow.py", line 246, in wrap_with return call() ^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 222, in _clone_default_branch git = clone(url, clone_path) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.cache/pypoetry/virtualenvs/projectname-preprocessing-v421l50I-py3.11/lib/python3.11/site-packages/dvc/scm.py", line 165, in clone raise CloneError("SCM error") from exc dvc.scm.CloneError: SCM error 2023-07-20 16:24:08,533 DEBUG: Analytics is enabled. 2023-07-20 16:24:08,595 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpo5xtbm_y']' 2023-07-20 16:24:08,597 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpo5xtbm_y']' ```
wjn0 commented 1 year ago

I was able to work around this by falling back to dulwich and setting the GIT_SSH=ssh environment variable on my system.

I wonder if manually specifying the keyfile somehow for import would make it work out of the box?

pmrowla commented 1 year ago

@wjn0 you are running two different versions of DVC. dvc doctor is reporting 3.5.1 but the traceback for your import command is 3.5.0. It looks like you may have multiple DVC environments installed and the patch is only installed in the 3.5.1 case?

edit: I'm not sure if you can even test a dependency patch this way with poetry, you may need to just manually create a venv for DVC and test without using poetry.

$ python -m venv <venv directory>
$ source <venv directory>/bin/activate
$ pip install -U pip dvc
$ pip install 'scmrepo @ git+https://github.com/iterative/scmrepo.git@refs/pull/255/head'
$ dvc import ...

You can run deactivate and then remove the venv dir when you are finished testing

wjn0 commented 1 year ago

Very sorry - that's my fault for getting my wires crossed between the two environments, but I did in fact try the patch in both environments (I think this is evidenced by the traceback - starting on line 286 await asyncssh.connect(... which is line-number-consistent with your PR on scmrepo as best I can tell).

For clarity, here is the output of dvc doctor and the import command in a clean repo (i.e. no poetry, a single commit with just git init and dvc init having been run and the result committed):

DVC version: 3.5.1 (pip)
------------------------
Platform: Python 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27
Subprojects:
        dvc_data = 2.5.0
        dvc_objects = 0.23.0
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.0.5.dev5+g4980cba
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.7.0)
Config:
        Global: /home/nelsonw/.config/dvc
        System: /etc/xdg/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/1c7fe7f1a46cb3955b64410884ebdc76
And the traceback in that environment: ```2023-07-21 11:23:51,953 DEBUG: v3.5.1 (pip), CPython 3.11.3 on Linux-4.15.0-202-generic-x86_64-with-glibc2.27 2023-07-21 11:23:51,953 DEBUG: command: /home/username/.local/bin/dvc import -v git@github.com:username/projectname_data.git raw 2023-07-21 11:23:52,468 DEBUG: Removing output 'raw' of stage: 'raw.dvc'. 2023-07-21 11:23:52,468 DEBUG: Removing '/tmp/tmprepo/raw' Importing 'raw (git@github.com:username/projectname_data.git)' -> 'raw' 2023-07-21 11:23:52,469 DEBUG: Computed stage: 'raw.dvc' md5: '5656116a17910fc725358de2d15f5bc9' 2023-07-21 11:23:52,469 DEBUG: 'md5' of stage: 'raw.dvc' changed. 2023-07-21 11:23:52,470 DEBUG: Creating external repo git@github.com:username/projectname_data.git@None 2023-07-21 11:23:52,470 DEBUG: erepo: git clone 'git@github.com:username/projectname_data.git' to a temporary dir 2023-07-21 11:23:52,797 ERROR: failed to import 'raw' - SCM error: Failed to clone repo 'git@github.com:username/projectname_data.git' to '/tmp/tmpfen5eixidvc-clone': Authentication failed for: 'git@github.com:22': Permission denied Traceback (most recent call last): File "/home/username/.local/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 286, in _run_command conn = await asyncssh.connect( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/asyncssh/connection.py", line 8093, in connect return await asyncio.wait_for( ^^^^^^^^^^^^^^^^^^^^^^^ File "/lmod/packages/python/3.11.3/lib/python3.11/asyncio/tasks.py", line 442, in wait_for return await fut ^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncssh.misc.PermissionDenied: Permission denied The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/username/.local/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 226, in clone repo = clone_from() ^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dulwich/porcelain.py", line 514, in clone return client.clone( ^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dulwich/client.py", line 703, in clone result = self.fetch(path, target, progress=progress, depth=depth) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dulwich/client.py", line 781, in fetch result = self.fetch_pack( ^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dulwich/client.py", line 1097, in fetch_pack proto, can_read, stderr = self._connect(b"upload-pack", path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dulwich/client.py", line 1746, in _connect con = self.ssh_vendor.run_command( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/fsspec/asyn.py", line 121, in wrapper return sync(self.loop, func, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/fsspec/asyn.py", line 106, in sync raise return_result File "/home/username/.local/lib/python3.11/site-packages/fsspec/asyn.py", line 61, in _runner result[0] = await coro ^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 299, in _run_command raise AuthError(f"{username}@{host}:{port or 22}") from exc scmrepo.exceptions.AuthError: Authentication failed for: 'git@github.com:22' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/username/.local/lib/python3.11/site-packages/dvc/scm.py", line 160, in clone git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/scmrepo/git/__init__.py", line 145, in clone backend.clone(url, to_path, bare=bare, mirror=mirror, **kwargs) File "/home/username/.local/lib/python3.11/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 234, in clone raise CloneError(url, to_path) from exc scmrepo.exceptions.CloneError: Failed to clone repo 'git@github.com:username/projectname_data.git' to '/tmp/tmpfen5eixidvc-clone' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/username/.local/lib/python3.11/site-packages/dvc/commands/imp.py", line 17, in run self.repo.imp( File "/home/username/.local/lib/python3.11/site-packages/dvc/repo/imp.py", line 6, in imp return self.imp_url(path, out=out, erepo=erepo, frozen=True, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/repo/__init__.py", line 64, in wrapper return f(repo, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/repo/scm_context.py", line 151, in run return method(repo, *args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/repo/imp_url.py", line 85, in imp_url stage.run(jobs=jobs, no_download=no_download) File "/home/username/.local/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() ^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/stage/__init__.py", line 600, in run self._sync_import(dry, force, kwargs.get("jobs", None), no_download) File "/home/username/.local/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() ^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/stage/__init__.py", line 637, in _sync_import sync_import(self, dry, force, jobs, no_download) File "/home/username/.local/lib/python3.11/site-packages/dvc/stage/imports.py", line 57, in sync_import stage.save_deps() File "/home/username/.local/lib/python3.11/site-packages/dvc/stage/__init__.py", line 491, in save_deps dep.save() File "/home/username/.local/lib/python3.11/site-packages/dvc/dependency/repo.py", line 58, in save rev = self.fs.repo.get_rev() ^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/fs/dvc.py", line 427, in repo return self.fs.repo ^^^^^^^ File "/lmod/packages/python/3.11.3/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/fs/dvc.py", line 416, in fs return _DVCFileSystem(**self.fs_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__ obj = super().__call__(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/fs/dvc.py", line 128, in __init__ repo = self._make_repo(url=url, rev=rev, subrepos=subrepos, **repo_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/fs/dvc.py", line 188, in _make_repo with Repo.open(uninitialized=True, **kwargs) as repo: File "/lmod/packages/python/3.11.3/lib/python3.11/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 30, in _external_repo path = _cached_clone(url, rev, for_write=for_write) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 154, in _cached_clone clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/funcy/flow.py", line 246, in wrap_with return call() ^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/repo/open_repo.py", line 222, in _clone_default_branch git = clone(url, clone_path) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/username/.local/lib/python3.11/site-packages/dvc/scm.py", line 165, in clone raise CloneError("SCM error") from exc dvc.scm.CloneError: SCM error 2023-07-21 11:23:52,807 DEBUG: Analytics is enabled. 2023-07-21 11:23:52,849 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpgk28u63j']' 2023-07-21 11:23:52,850 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpgk28u63j']' ```

In case it's helpful for other Poetry users reading this, it is in fact possible to do poetry add "scmrepo @ ..." the way one would do with Pip - or at least, it seems to work.

Again, sorry for the mixup - and hopefully I'm not missing something obvious here, I'm actually fairly new user of poetry as well and have not used dvc since version 2ish. Thanks for all your help!

pmrowla commented 1 year ago

@wjn0 Can you share your ~/.ssh/config? Be sure to sanitize any private values, I mainly need to know what options you are using

wjn0 commented 1 year ago

@pmrowla Hm, I don't seem to have a ~/.ssh/config on this set of hosts. I wonder if that's a problem?

To check what config's being used, I did GIT_SSH_COMMAND="ssh -vvv" git clone git@github.com:username/prviate_repo.git and it pointed me to /etc/ssh/ssh_config (line excerpt: debug1: Reading configuration data /etc/ssh/ssh_config). That config has:

$ cat /etc/ssh/ssh_config  | grep -ve "^#"
Host *
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes

which seems to be pretty standard. Please feel free to let me know if I can provide anything further.

pmrowla commented 1 year ago

@wjn0 can you try installing asyncssh[gssapi] in your environment and then retry the DVC import

wjn0 commented 1 year ago

That Pip install seems to error out trying to find a Kerberos-related binary:

  Downloading gssapi-1.8.2.tar.gz (94 kB)
     ________________________________________ 94.3/94.3 kB 8.3 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  _ Getting requirements to build wheel did not run successfully.
  _ exit code: 1
  __> [25 lines of output]
      /bin/sh: 1: krb5-config: not found
      Traceback (most recent call last):
        File "/lmod/packages/python/3.11.3/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/lmod/packages/python/3.11.3/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/lmod/packages/python/3.11.3/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-1ff__d9j/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-1ff__d9j/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-1ff__d9j/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 109, in <module>
        File "<string>", line 22, in get_output
        File "/lmod/packages/python/3.11.3/lib/python3.11/subprocess.py", line 466, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/lmod/packages/python/3.11.3/lib/python3.11/subprocess.py", line 571, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command 'krb5-config --libs gssapi' returned non-zero exit status 127.
      [end of output]

I tried adding a HOST github.com section to a new ~/.ssh/config file setting GSSAPIAuthentication no (and confirmed that the setting was recognized when running the usual git clone ...) but the dvc import still failed on the same error.

pmrowla commented 1 year ago

@pmrowla Hm, I don't seem to have a ~/.ssh/config on this set of hosts.

I was able to reproduce the issue thanks to this. I'm assuming that your key is one of the openssh defaults (i.e one of ~/.ssh/id_rsa/~/.ssh/id_ed25519/etc)?

The issue was a combination of using a default key path plus the interactive passphrase issue which was fixed in the initial PR.

It should be resolved now in

'scmrepo @ git+https://github.com/iterative/scmrepo.git@refs/pull/259/head'

and will be available in the next scmrepo release.