search

iterative / dvc-webdav

Webdav plugin for dvc
Apache License 2.0
1 stars 2 forks source link

Unexpected error occurred when providing invalid credentials #24

Open hqdncw opened 11 months ago

hqdncw commented 11 months ago

Bug Report

Description

When using the webdavs remote with DVC, if you don't provide both a username and password for your WebDAV server, DVC will will fail with an internal error message that may not be clear to users.

Reproduce

  1. Set up an SFTPGo server that supports WebDAV. This server will listen on a specific port number (10080) on your local machine. 
    
#!/usr/bin/env bash

CONTAINER_NAME="webdav" SECRETS_DIR="/usr/local/src/secrets"

graceful_shutdown() { docker stop $CONTAINER_NAME && docker rm $CONTAINER_NAME exit } trap graceful_shutdown INT TERM

docker create --name $CONTAINER_NAME \ -a STDOUT \ -p 8080:8080 \ -p 2022:2022 \ -p 10080:10080 \ -e SFTPGO_WEBDAVDBINDINGS0__PORT=10080 \ -t "drakkan/sftpgo:v2.5.5" || exit

docker cp -q ./secrets/ $CONTAINER_NAME:$SECRETS_DIR || exit docker start --attach $CONTAINER_NAME || exit


2. Initialize DVC.
```bash
dvc init --no-scm

  1. Track some changes

    touch example.xml && dvc add example.xml

  2. Add the remote

    dvc remote add test webdavs://localhost:10080

  3. Specify a username for the remote

    dvc remote modify test user myuser

  4. Try to push changes to the remote.

    
$ dvc push --remote test
ERROR: unexpected error - sequence item 1: expected a bytes-like object, NoneType found

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!


<details>

  <summary>Verbose output</summary>

```bash
$ dvc push
ERROR: unexpected error - sequence item 1: expected a bytes-like object, NoneType found                                                                                   

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
(.venv) ➜  .local git:(3.30.1) dvc push --verbose
2023-11-22 12:21:44,644 DEBUG: v3.30.1, CPython 3.11.2 on <REDACTED>
2023-11-22 12:21:44,644 DEBUG: command: push --verbose
2023-11-22 12:21:44,862 DEBUG: Preparing to transfer data from '/home/sid/workspace/dvc/.local/.dvc/cache/files/md5' to 'https://localhost:10080/files/md5'         
2023-11-22 12:21:44,863 DEBUG: Preparing to collect status from 'files/md5'                                                                                               
2023-11-22 12:21:44,863 DEBUG: Collecting status from 'files/md5'                                                                                                         
2023-11-22 12:21:44,864 DEBUG: Querying 1 oids via object_exists                                                                                                          
2023-11-22 12:21:44,951 ERROR: unexpected error - sequence item 1: expected a bytes-like object, NoneType found                                                           
Traceback (most recent call last):                                                                                                                                        
  File "/home/sid/workspace/dvc/dvc/repo/push.py", line 117, in push
    push_transferred, push_failed = ipush(
                                    ^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_data/index/push.py", line 68, in push
    result = transfer(
             ^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_data/hashfile/transfer.py", line 204, in transfer
    status = compare_status(
             ^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_data/hashfile/status.py", line 176, in compare_status
    dest_exists, dest_missing = status(
                                ^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_data/hashfile/status.py", line 148, in status
    exists.update(odb.oids_exist(hashes, jobs=jobs, progress=pbar.callback))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/db.py", line 411, in oids_exist
    return list(wrap_iter(remote_oids, callback))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/db.py", line 36, in wrap_iter
    for index, item in enumerate(iterable, start=1):
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/db.py", line 359, in list_oids_exists
    in_remote = self.fs.exists(paths, batch_size=jobs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 358, in exists
    if self.fs.async_impl:
       ^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 47, in __get__
    return prop.__get__(instance, type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
                                                  ^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_webdav/__init__.py", line 86, in fs
    return WebdavFileSystem(**self.fs_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__
    obj = super().__call__(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/webdav4/fsspec.py", line 103, in __init__
    self.client = client or Client(base_url, auth=auth, **client_opts)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/webdav4/client.py", line 265, in __init__
    self.http: HTTPClient = http_client or HTTPClient(**client_opts)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 646, in __init__
    super().__init__(
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 180, in __init__
    self._auth = self._build_auth(auth)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 433, in _build_auth
    return BasicAuth(username=auth[0], password=auth[1])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_auth.py", line 130, in __init__
    self._auth_header = self._build_auth_header(username, password)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_auth.py", line 139, in _build_auth_header
    userpass = b":".join((to_bytes(username), to_bytes(password)))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: sequence item 1: expected a bytes-like object, NoneType found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sid/workspace/dvc/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
          ^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/dvc/cli/command.py", line 27, in do_run
    return self.run()
           ^^^^^^^^^^
  File "/home/sid/workspace/dvc/dvc/commands/data_sync.py", line 64, in run
    processed_files_count = self.repo.push(
                            ^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/dvc/repo/__init__.py", line 60, in wrapper
    return f(repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/dvc/repo/push.py", line 125, in push
    _update_meta(
  File "/home/sid/workspace/dvc/dvc/repo/push.py", line 18, in _update_meta
    new = build_data_index(
          ^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/dvc/repo/index.py", line 783, in build_data_index
    if not fs.exists(out_path):
           ^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 355, in exists
    return self.fs.exists(path)
           ^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 47, in __get__
    return prop.__get__(instance, type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
                                                  ^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_webdav/__init__.py", line 86, in fs
    return WebdavFileSystem(**self.fs_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__
    obj = super().__call__(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/webdav4/fsspec.py", line 103, in __init__
    self.client = client or Client(base_url, auth=auth, **client_opts)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/webdav4/client.py", line 265, in __init__
    self.http: HTTPClient = http_client or HTTPClient(**client_opts)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 646, in __init__
    super().__init__(
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 180, in __init__
    self._auth = self._build_auth(auth)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 433, in _build_auth
    return BasicAuth(username=auth[0], password=auth[1])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_auth.py", line 130, in __init__
    self._auth_header = self._build_auth_header(username, password)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_auth.py", line 139, in _build_auth_header
    userpass = b":".join((to_bytes(username), to_bytes(password)))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: sequence item 1: expected a bytes-like object, NoneType found

2023-11-22 12:21:44,999 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-11-22 12:21:44,999 DEBUG: Removing '/home/sid/workspace/dvc/.bQmqxAzbbLKem5ydCE5cQL.tmp'
2023-11-22 12:21:44,999 DEBUG: Removing '/home/sid/workspace/dvc/.bQmqxAzbbLKem5ydCE5cQL.tmp'
2023-11-22 12:21:44,999 DEBUG: Removing '/home/sid/workspace/dvc/.bQmqxAzbbLKem5ydCE5cQL.tmp'
2023-11-22 12:21:45,000 DEBUG: Removing '/home/sid/workspace/dvc/.local/.dvc/cache/files/md5/.87YBr5G5csNYhnrtPGBXpo.tmp'
2023-11-22 12:21:45,001 DEBUG: Version info for developers:
DVC version: 3.30.1
-------------------
Platform: Python 3.11.2 on <REDACTED>
Subprojects:
        dvc_data = 2.22.0
        dvc_objects = 1.1.0
        dvc_render = 0.6.0
        dvc_task = 0.3.0
        scmrepo = 1.4.1
Supports:
        azure (adlfs = 2023.10.0, knack = 0.11.0, azure-identity = 1.15.0),
        gdrive (pydrive2 = 1.17.0),
        gs (gcsfs = 2023.9.2),
        hdfs (fsspec = 2023.9.2, pyarrow = 14.0.1),
        http (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
        oss (ossfs = 2021.8.0),
        s3 (s3fs = 2023.9.2, boto3 = 1.28.17),
        ssh (sshfs = 2023.10.0),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2023.9.2)
Config:
        Global: /home/sid/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda9
Caches: local
Remotes: webdavs
Workspace directory: ext4 on /dev/sda9
Repo: dvc (no_scm)
Repo.site_cache_dir: /var/tmp/dvc/repo/f3aaa8fd4f4e85f507ddd8996bd9016b

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-11-22 12:21:45,001 DEBUG: Analytics is disabled.

Expected

DVC to fail with an error message stating that the password field is not specified in .dvc/config.

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 3.30.1
-------------------
Platform: Python 3.11.2 on <REDACTED>
Subprojects:
        dvc_data = 2.22.0
        dvc_objects = 1.1.0
        dvc_render = 0.6.0
        dvc_task = 0.3.0
        scmrepo = 1.4.1
Supports:
        azure (adlfs = 2023.10.0, knack = 0.11.0, azure-identity = 1.15.0),
        gdrive (pydrive2 = 1.17.0),
        gs (gcsfs = 2023.9.2),
        hdfs (fsspec = 2023.9.2, pyarrow = 14.0.1),
        http (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
        oss (ossfs = 2021.8.0),
        s3 (s3fs = 2023.9.2, boto3 = 1.28.17),
        ssh (sshfs = 2023.10.0),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2023.9.2)
Config:
        Global: /home/sid/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda9
Caches: local
Remotes: webdavs
Workspace directory: ext4 on /dev/sda9
Repo: dvc (no_scm)
Repo.site_cache_dir: /var/tmp/dvc/repo/f3aaa8fd4f4e85f507ddd8996bd9016b