search

iterative / dvc-ssh

SSH/SFTP plugin for dvc
Apache License 2.0
1 stars 2 forks source link

pull: Permission Denied from SSH #33

Closed haimat closed 1 year ago

haimat commented 1 year ago

Bug Report

Description

Using an existing SSH remote for DVC, I am not able to pull any data from it - permission denied. I get the following:

$ dvc pull --jobs 1 --remote aimero -v
2023-03-06 16:05:58,629 DEBUG: v2.45.1 (pip), CPython 3.10.6 on Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
2023-03-06 16:05:58,629 DEBUG: command: /home/mfb/.local/bin/dvc pull --jobs 1 --remote aimero -v
2023-03-06 16:05:59,875 DEBUG: Preparing to transfer data from '/data/dvc' to '/mnt/wsl/PhysicalDrive6/clients/head/dev/.dvc/cache'
2023-03-06 16:05:59,875 DEBUG: Preparing to collect status from '/mnt/wsl/PhysicalDrive6/clients/head/dev/.dvc/cache'
2023-03-06 16:05:59,876 DEBUG: Collecting status from '/mnt/wsl/PhysicalDrive6/clients/head/dev/.dvc/cache'
2023-03-06 16:05:59,907 DEBUG: Preparing to collect status from '/data/dvc'
2023-03-06 16:05:59,909 DEBUG: Collecting status from '/data/dvc'
2023-03-06 16:05:59,965 DEBUG: Querying 3 oids via object_exists
2023-03-06 16:06:00,310 ERROR: unexpected error - Permission denied: Permission denied
Traceback (most recent call last):
  File "/home/mfb/.local/lib/python3.10/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/home/mfb/.local/lib/python3.10/site-packages/sshfs/spec.py", line 82, in _connect
    client = await self._stack.enter_async_context(_raw_client)
  File "/usr/lib/python3.10/contextlib.py", line 619, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/home/mfb/.local/lib/python3.10/site-packages/asyncssh/misc.py", line 274, in __aenter__
    self._coro_result = await self._coro
  File "/home/mfb/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 8047, in connect
    return await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/home/mfb/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 447, in _connect
    await options.waiter
asyncssh.misc.PermissionDenied: Permission denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/cli/__init__.py", line 210, in main
    ret = cmd.do_run()
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/commands/data_sync.py", line 31, in run
    stats = self.repo.pull(
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/repo/pull.py", line 34, in pull
    processed_files_count = self.fetch(
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/repo/fetch.py", line 86, in fetch
    d, f = _fetch(
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/repo/fetch.py", line 166, in _fetch
    d, f = repo.cloud.pull(
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/data_cloud.py", line 181, in pull
    return self.transfer(
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc/data_cloud.py", line 135, in transfer
    return transfer(src_odb, dest_odb, objs, **kwargs)
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 203, in transfer
    status = compare_status(
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 189, in compare_status
    src_exists, src_missing = status(
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 134, in status
    exists = hashes.intersection(
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 42, in _indexed_dir_hashes
    indexed_dir_exists.update(hashes)
  File "/home/mfb/.local/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc_objects/db.py", line 357, in list_oids_exists
    in_remote = self.fs.exists(paths, batch_size=jobs)
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 332, in exists
    if self.fs.async_impl:
  File "/home/mfb/.local/lib/python3.10/site-packages/funcy/objects.py", line 50, in __get__
    return prop.__get__(instance, type)
  File "/home/mfb/.local/lib/python3.10/site-packages/funcy/objects.py", line 28, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/home/mfb/.local/lib/python3.10/site-packages/dvc_ssh/__init__.py", line 116, in fs
    return _SSHFileSystem(**self.fs_args)
  File "/home/mfb/.local/lib/python3.10/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/home/mfb/.local/lib/python3.10/site-packages/sshfs/spec.py", line 65, in __init__
    self._client, self._pool = self.connect(
  File "/home/mfb/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 114, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/mfb/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 99, in sync
    raise return_result
  File "/home/mfb/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 54, in _runner
    result[0] = await coro
  File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/home/mfb/.local/lib/python3.10/site-packages/sshfs/utils.py", line 29, in wrapper
    raise PermissionError(exc.reason) from exc
PermissionError: Permission denied

2023-03-06 16:06:00,729 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-03-06 16:06:00,729 DEBUG: Removing '/mnt/wsl/PhysicalDrive6/clients/head/.XNtbKRnf2CryuWzSHMVuwi.tmp'
2023-03-06 16:06:00,729 DEBUG: Removing '/mnt/wsl/PhysicalDrive6/clients/head/.XNtbKRnf2CryuWzSHMVuwi.tmp'
2023-03-06 16:06:00,730 DEBUG: Removing '/mnt/wsl/PhysicalDrive6/clients/head/.XNtbKRnf2CryuWzSHMVuwi.tmp'
2023-03-06 16:06:00,730 DEBUG: Removing '/mnt/wsl/PhysicalDrive6/clients/head/dev/.dvc/cache/.RmDzkxGmuUSP7x8XfG5uRy.tmp'
2023-03-06 16:06:00,747 DEBUG: Version info for developers:
DVC version: 2.45.1 (pip)
-------------------------
Platform: Python 3.10.6 on Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 0.40.3
        dvc_objects = 0.19.3
        dvc_render = 0.2.0
        dvc_task = 0.1.11
        dvclive = 2.0.2
        scmrepo = 0.1.11
Supports:
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.1.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdd
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/sdd
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-03-06 16:06:00,748 DEBUG: Analytics is enabled.
2023-03-06 16:06:00,776 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpkuvx4hq4']'
2023-03-06 16:06:00,778 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpkuvx4hq4']'

Note that I can connect and access the DVC remote aimero via that SSH user without problem:

$ dvc remote list
aimero  ssh://dvcro@<server>/data/dvc
$ $ ssh -l dvcro <server> ls /data/dvc/ff
1006f3a315c5d2e87d4ddeaa603442
36c52eb770c57c607f8633dab69b74
3be7b255a84e232ae1bec5747fced1
426db5c036e37206ca7af249929658
46a66eb951291e0932323e90523cf7.dir
5235fdfdddbe89ed185b20fe3bca06
5a32d34630cd889b3d0699b4be65a9
60d5b0798431ed6b284e7cdb9fb443
611b4079d26b35d16881d3423616f6
76937f839e40770dff8a52852a3687
85f68062839edf919deb6088a3c6bb
b37ddad510e6bd15d56eb78dce524c
b74df064b69577713009060e573df4
c743e761313ffce24b3364cbbe3fe6
d4c8af792731e651a7b7ae8fa1c2d0
e63a054e87cc6b80ff2e06a25d9189
f1395d4545d36850a1ec761871bb6f
f180159b7e84d3ff837c809297810f
f42f4077898f1d9d8f4d2e60fa8cc1
fd7112dc945af63e10250ba3822bee

Environment information

Environment info see above.

shcheklein commented 1 year ago

@haimat just to confirm - are you running on WSL2? any details on the SSH config - anything special there? Do you use a key agent? Is your key encrypted (do you need to type in your passphrase every time)? Do you have VPN, etc? Do you have some firewall enabled? All the details re SSH setup matter and can help us try it faster ...

haimat commented 1 year ago

@shcheklein It's on WSL2, yes, but nothing fancy here. No key agent, no public key, just a normal password. Using that same SSH user dvcro on the server it also does not work via Studio - "No permission to fetch remote data." But as said, I can login via command line using that user and access the DVC cache on that machine without problem (see above).

Also there is no firewall or so.. I can connect fine via SSH on the command line, both with password and with a key file. When I log into the remote server that way, I can also access all the files in the DVC cache. So the problem seems to be related to the way the DVC client tries to access the SSH server. I also tried it with the -vv option to see even more output, but that didn't help.

Is there a way to debug this on my end, so that I could see where exactly this "permission denied" error message comes from?

shcheklein commented 1 year ago

with a key file

so, it is passphrase protected? Let me even rephrase, do you need to enter any password / phrase when you do the regular SSH via command line?

I'll try to reproduce it on WSL today and test the studio SSH one more time to see if there is an issue there.

haimat commented 1 year ago

No the key file does not have a passphrase. I tried it first without providing the key file, only the normal user password, and then with providing the key file (without any password). In both cases I could normally login via SSH and access that files in the DVC cache. So login itself works fine with SSH. I am willing to help and debug for you, I just don't know what else I could try. Happy for any suggestions!

haimat commented 1 year ago

Ohh one more thing: It has worked from the same WSL box with another SSH user on the remote server to dvc pull. So the problem is not related to WSL I guess, but to that SSH user. As said, I would like to debug that on my end and see where the error comes from, I just don't know how I can get more info from the dvc tool?

haimat commented 1 year ago

@shcheklein OK I might have a clue. On the remote server I have two users related to DVC - one is the main user with read+write permissions in the DVC data folder, the other user is in the same Linux group as the first, but that group has only read permissions in the DVC data folder. Now when I dvc pull locally with the main user (r+w permissions), then everything works fine and I get all DVC data. But when I use the read-only user instead, then I get this "Permission denied" error.

So does this mean that DVC needs write-permissions in the DVC data folder, even for dvc pull commands? If that is the case, but I don't want to provide access to the main user with write permissions to Studio, how else can I then limit access to the DVC data folder to read-only?

shcheklein commented 1 year ago

@haimat to double check, could you try to scp the whole data cache with the read-only user?

@efiop any thoughts about this So does this mean that DVC needs write-permissions in the DVC data folder, even for dvc pull commands? - I think it should not be the case, but may be I'm missing something?

I would like to debug that on my end and see where the error comes from, I just don't know how I can get more info from the dvc tool?

-v enabled logs for DVC, but I'm not sure it can be compared with ssh logs. Let me see if there is way to enable more information from the SSH lib that we use.

efiop commented 1 year ago

No, we shouldn't need the write permissions for regular remotes. The error posted above is coming from just creating a connection, we don't even try to list/read/execute anything yet.

I would try playing with asyncssh directly https://github.com/ronf/asyncssh since it looks like we've narrowed it down to connect call at least.

haimat commented 1 year ago

@haimat to double check, could you try to scp the whole data cache with the read-only user?

Yep, I was able to fully rsync the whole DVC data cache folder to my local machine using the read-only user 👍

shcheklein commented 1 year ago

thanks Ruslan, @haimat will you be able to try the asyncssh directly to connect with the second user? btw, how do you provide different SSH keys for different users in DVC?

haimat commented 1 year ago

@shcheklein Yes I will try to reproduce this problem directly with the asyncssh library. As for your other question: I don't need to provide the SSH key to DVC, I simply have set these keys to the respective users on OS level, so SSH automatically uses the the right key for the right user. Is this what you wanted to know?

shcheklein commented 1 year ago

Yes, how does it look like practically? some SSH config? could you share it please, so that I can try to do the same on my end.

haimat commented 1 year ago

I think there is a misunderstanding. I simply created two different keys and stored them to the two different users. When I connect via SSH on the command line I provide the ssh -i switch to specify the key file to use. But I guess this is not possible via DVC, right? So maybe that is the issue here? But I am not sure about that - why would Studio then end up with the same erorr? I provided a key for the credentials there.

haimat commented 1 year ago

Ahh now we are getting somewhere! I have just copied over my SSH key to the remote machine, and now it seems I can use that read-only user to pull from the DVC remote. But that does not explain the issue with Studio, right? Is it another thing there?

shcheklein commented 1 year ago

Hmm, have you tried to force import one more time?

But I guess this is not possible via DVC, right?

It's possible https://dvc.org/doc/command-reference/remote/modify#ssh - you should use something like this:

$ dvc remote modify --local myremote keyfile /path/to/keyfile

I have just copied over my SSH key to the remote machine

yo you mean you added it to the authorized_keys or something else?

haimat commented 1 year ago

Ok, so I was able to track down the problem - it was on my end. I had not provided the public key file for the read-only user when I tried to dvc pull, thus obviously DVC was not able to properly connect to the SSH remote. (note: I thought DVC would ask me for a password when it cannot connect via key file, which it didn't, so maybe you might want to add a proper message to the dvc command line tool for this case to avoid this confusion.) After adding that public key file to the remote user (via the ssh-copy-id tool), I am able to run dvc pull with the read-only user without problems 👍

Sorry for the confusion.

However, the main reason why I came to this issue in the first place is that Studio seems to have problems accessing the DVC remote via SSH. I thought that is the same problem as discussed here, but I don't think so any more. To keep things in order I have created a new Github issue for Studio, so we can close this one now.

efiop commented 1 year ago

@haimat Could you elaborate about the public key? Not sure I understand where you didn't provide it. You mean on the server side?

haimat commented 1 year ago

@efiop Exactly, I forgot to copy over the public key of my local user to read-only user on the SSH server. I didn't think about that, because as said I thought dvc would just ask me for the SSH password if it cannot authenticate via a SSH key file (as the SSH command line tool would do).

daavoo commented 1 year ago

I thought dvc would just ask me for the SSH password if it cannot authenticate via a SSH key file (as the SSH command line tool would do

It is a little hidden in the docs (https://dvc.org/doc/command-reference/remote/modify#ssh) but there is a config option that you need to enable if you want DVC to ask for a password:

dvc remote modify myremote ask_password true
haimat commented 1 year ago

@daavoo Thanks for the link 👍 Anyway, this issue is solved now, as said, using the SSH key file for the ro-user. I hope the Studio issue is also as easy to fix!