iterative / dvc

🦉 ML Experiments and Data Management with Git
https://dvc.org
Apache License 2.0
13.63k stars 1.18k forks source link

DVC doesn't consider url parameter in .dvc/config file when using import #8215

Open hemker opened 2 years ago

hemker commented 2 years ago

Bug Report

import: doesn't consider url parameter if config.local is available

Description

If you have set up two remotes in your .dvc/config file and handle their authorization in .dvc/config.local, using the dvc import command produces the output ERROR: failed to import 'path_from_repo' from 'https://github.com/myrepo.git'. - config file error: expected 'url' for dictionary value @ data['remote']['my-inbox']

Reproduce

.dvc/config contains:

[core]
    remote = my-storage
['remote "my-inbox"']
    url = s3://labelstudio-inbox
    endpointurl = https://my-url
['remote "my-storage"']
    url = s3://my-storage
    endpointurl = https://my-url

.dvc/config.local contains:

['remote "my-inbox"']
    access_key_id = ACCES__KEY
    secret_access_key = SECRET_KEY
['remote "my-storage"']
    access_key_id = ACCES__KEY
    secret_access_key = SECRET_KEY

Then run

dvc import https://github.com/NVIDIA/DeepLearningExamples.git PyTorch/Detection/SSD -o SSD

Expected

I expect that the specified path from the given repository is checked out in the given folder name SSD.

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.13.0 (pip)
---------------------------------
Platform: Python 3.10.4 on Linux-5.15.0-46-generic-x86_64-with-glibc2.35
Supports:
    webhdfs (fsspec = 2022.5.0),
    http (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
    https (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
    s3 (s3fs = 2022.5.0, boto3 = 1.21.21)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p3
Caches: local
Remotes: s3, s3
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc, git

Additional Information (if any): See import-url error thread in your Discord channel.

rlamy commented 2 years ago

Simpler repro:

git init
dvc init
dvc remote add storage ../storage
dvc remote modify storage --local jobs 1
dvc import https://github.com/iterative/dvc-data README.md