search

iterative / dvc-ssh

SSH/SFTP plugin for dvc
Apache License 2.0
1 stars 3 forks source link

new dvc[ssh] install fails with link type hardlink is not available #26

Closed Tel44 closed 1 year ago

Tel44 commented 1 year ago

Bug Report

push/pull fails with

2023-02-06 17:26:47,874 DEBUG: Preparing to transfer data from '/home/user/dvc' to '/home/user/mydir/repo/.dvc/cache'
2023-02-06 17:26:47,874 DEBUG: Preparing to collect status from '/home/user/mydir/repo/.dvc/cache'
2023-02-06 17:26:47,876 DEBUG: Collecting status from '/home/user/mydir/repo/.dvc/cache'
2023-02-06 17:26:47,912 DEBUG: Preparing to collect status from '/home/user/dvc'
2023-02-06 17:26:47,913 DEBUG: Collecting status from '/home/user/dvc'
2023-02-06 17:26:47,925 DEBUG: Querying 12 oids via object_exists
2023-02-06 17:26:50,072 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-02-06 17:26:50,072 DEBUG: Removing '/home/user/mydir/.TmhEhw7u8VjWmGd8Q2CVwf.tmp'
2023-02-06 17:26:50,073 DEBUG: link type hardlink is not available ([Errno 95] no more link types left to try out)
2023-02-06 17:26:50,073 DEBUG: Removing '/home/user/mydir/.TmhEhw7u8VjWmGd8Q2CVwf.tmp'
2023-02-06 17:26:50,074 DEBUG: Removing '/home/user/mydir/.TmhEhw7u8VjWmGd8Q2CVwf.tmp'
2023-02-06 17:26:50,074 DEBUG: Removing '/home/user/mydir/repo/.dvc/cache/.mvsfiMnfFpZ9otaNXLahk8.tmp'
2023-02-06 17:26:50,572 DEBUG: Version info for developers:
DVC version: 2.43.1 (pip)
---------------------------------
Platform: Python 3.10.6 on Linux-5.14.21-150400.24.41-default-x86_64-with-glibc2.35
Subprojects:
    dvc_data = 0.35.1
    dvc_objects = 0.19.0
    dvc_render = 0.0.17
    dvc_task = 0.1.11
    dvclive = 1.3.3
    scmrepo = 0.1.6
Supports:
    http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
    https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
    ssh (sshfs = 2023.1.0)
Cache types: symlink
Cache directory: ext4 on /dev/mapper/encdata
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/mapper/encdata
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-02-06 17:26:50,577 DEBUG: Analytics is enabled.
2023-02-06 17:26:50,663 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpb46lnae8']'
2023-02-06 17:26:50,666 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpb46lnae8']'

If I use the old install from an old docker it goes through without any issues with the very same remote/key/user filesystem.

I am at a loss how to debug this further.

efiop commented 1 year ago

@Tel44 Seems like the log is incomplete. Could you double check you've shared the whole verbose log, please?

Tel44 commented 1 year ago

Sorry, I only gave you STDOUT and not STDERR. Here is both:

2023-02-06 18:00:55,923 DEBUG: Preparing to transfer data from '/home/user/dvc' to '/home/user/mydir/repo/.dvc/cache'
2023-02-06 18:00:55,924 DEBUG: Preparing to collect status from '/home/user/mydir/repo/.dvc/cache'
2023-02-06 18:00:55,925 DEBUG: Collecting status from '/home/user/mydir/repo/.dvc/cache'
2023-02-06 18:00:55,955 DEBUG: Preparing to collect status from '/home/user/dvc'
2023-02-06 18:00:55,956 DEBUG: Collecting status from '/home/user/dvc'
  0% Querying remote cache|                                                                                                 |0/12 [00:00<?,    ?files/s]
2023-02-06 18:00:55,979 DEBUG: Querying 12 oids via object_exists
2023-02-06 18:00:57,617 ERROR: unexpected error - Can't create any SFTP connections!
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/dvc/cli/__init__.py", line 184, in main
    ret = cmd.do_run()
  File "/usr/local/lib/python3.10/dist-packages/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/usr/local/lib/python3.10/dist-packages/dvc/commands/data_sync.py", line 31, in run
    stats = self.repo.pull(
  File "/usr/local/lib/python3.10/dist-packages/dvc/repo/__init__.py", line 66, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/dvc/repo/pull.py", line 34, in pull
    processed_files_count = self.fetch(
  File "/usr/local/lib/python3.10/dist-packages/dvc/repo/__init__.py", line 66, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/dvc/repo/fetch.py", line 86, in fetch
    d, f = _fetch(
  File "/usr/local/lib/python3.10/dist-packages/dvc/repo/fetch.py", line 166, in _fetch
    d, f = repo.cloud.pull(
  File "/usr/local/lib/python3.10/dist-packages/dvc/data_cloud.py", line 169, in pull
    return self.transfer(
  File "/usr/local/lib/python3.10/dist-packages/dvc/data_cloud.py", line 123, in transfer
    return transfer(src_odb, dest_odb, objs, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/dvc_data/hashfile/transfer.py", line 203, in transfer
    status = compare_status(
  File "/usr/local/lib/python3.10/dist-packages/dvc_data/hashfile/status.py", line 189, in compare_status
    src_exists, src_missing = status(
  File "/usr/local/lib/python3.10/dist-packages/dvc_data/hashfile/status.py", line 134, in status
    exists = hashes.intersection(
  File "/usr/local/lib/python3.10/dist-packages/dvc_data/hashfile/status.py", line 42, in _indexed_dir_hashes
    indexed_dir_exists.update(hashes)
  File "/usr/local/lib/python3.10/dist-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.10/dist-packages/dvc_objects/db.py", line 355, in list_oids_exists
    in_remote = self.fs.exists(paths, batch_size=jobs)
  File "/usr/local/lib/python3.10/dist-packages/dvc_objects/fs/base.py", line 351, in exists
    return fut.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/dvc_objects/executors.py", line 134, in batch_coros
    result = fut.result()
  File "/usr/local/lib/python3.10/dist-packages/fsspec/asyn.py", line 568, in _exists
    await self._info(path)
  File "/usr/local/lib/python3.10/dist-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/sshfs/spec.py", line 125, in _info
    async with self._pool.get() as channel:
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/sshfs/pools/soft.py", line 38, in get
    raise ValueError("Can't create any SFTP connections!")
ValueError: Can't create any SFTP connections!
------------------------------------------------------------
2023-02-06 18:00:58,095 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-02-06 18:00:58,096 DEBUG: Removing '/home/user/mydir/.65KNmgarsuAVDzBKdTzgyX.tmp'
2023-02-06 18:00:58,096 DEBUG: link type hardlink is not available ([Errno 95] no more link types left to try out)
2023-02-06 18:00:58,096 DEBUG: Removing '/home/user/mydir/.65KNmgarsuAVDzBKdTzgyX.tmp'
2023-02-06 18:00:58,097 DEBUG: Removing '/home/user/mydir/.65KNmgarsuAVDzBKdTzgyX.tmp'
2023-02-06 18:00:58,097 DEBUG: Removing '/home/user/mydir/repo/.dvc/cache/.gRy58GfY8Hzr9BtydWbo6D.tmp'
2023-02-06 18:00:58,402 DEBUG: Version info for developers:
DVC version: 2.43.1 (pip)
---------------------------------
Platform: Python 3.10.6 on Linux-5.14.21-150400.24.41-default-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 0.35.1
        dvc_objects = 0.19.0
        dvc_render = 0.0.17
        dvc_task = 0.1.11
        dvclive = 1.3.3
        scmrepo = 0.1.6
Supports:
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.1.0)
Cache types: symlink
Cache directory: ext4 on /dev/mapper/encdata
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/mapper/encdata
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-02-06 18:00:58,405 DEBUG: Analytics is enabled.
2023-02-06 18:00:58,467 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp715_x690']'
2023-02-06 18:00:58,469 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp715_x690']'
efiop commented 1 year ago

Closing in favor of https://github.com/iterative/dvc-ssh/issues/16 The core issue is the connections.

@Tel44 Maybe tweaking MaxSessions would help, similar to https://github.com/iterative/dvc-ssh/issues/16#issuecomment-1417042631 , would you be able to try that out?