[Feature request] Allow to download private components (PACMAN-249)

DCSBL commented 2 years ago

Is your feature request related to a problem? Please describe.

We host reusable components privately on GitHub. The component manager can't download these. My machine has access to the repo if I just call git clone <url> (over SSH).

Describe the solution you'd like

It would be awesome if the component manager somehow can have access to private repos.

Describe alternatives you've considered

We could make the components public, but this is not possible for all of them because of IP protection.

Additional context

If it is possible to do this, maybe we can add an explainer somewhere.

idf_component.yml

## IDF Component Manager Manifest File
dependencies:
  rgb_driver:
    path: rgb_driver
    git: ssh://git@github.com:dcsbl/rgb-driver.git

  idf:
    version: ">=4.1.0"

Log output

$ idf.py reconfigure

CMake Error at ~/esp/esp-idf-v4.3/tools/cmake/project.cmake:209 (message):
  Traceback (most recent call last):

    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_tools/git_client.py", line 112, in run
      return subprocess.check_output(  # nosec
    File "~/.pyenv/versions/3.9.4/lib/python3.9/subprocess.py", line 424, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "~/.pyenv/versions/3.9.4/lib/python3.9/subprocess.py", line 528, in run
      raise CalledProcessError(retcode, process.args,

  subprocess.CalledProcessError: Command '['git', 'fetch', 'origin']'
  returned non-zero exit status 128.

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):

    File "~/.pyenv/versions/3.9.4/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "~/.pyenv/versions/3.9.4/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/prepare_components/__main__.py", line 3, in <module>
      main()
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/prepare_components/prepare.py", line 110, in main
      args.func(args)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/prepare_components/prepare.py", line 37, in prepare_dep_dirs
      ComponentManager(args.project_dir).prepare_dep_dirs(
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/core.py", line 316, in prepare_dep_dirs
      downloaded_component_paths = download_project_dependencies(
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/dependencies.py", line 36, in download_project_dependencies
      solution = solver.solve()
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/version_solver/version_solver.py", line 29, in solve
      self.solve_manifest(manifest)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/version_solver/version_solver.py", line 46, in solve_manifest
      self.solve_component(requirement)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/version_solver/version_solver.py", line 49, in solve_component
      cmp_with_versions = requirement.source.versions(
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_tools/sources/git.py", line 94, in versions
      commit_id = self._checkout_git_source(version)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_tools/sources/git.py", line 40, in _checkout_git_source
      return self._client.prepare_ref(repo=self.git_repo, path=self.cache_path(), ref=version, with_submodules=True)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_tools/git_client.py", line 36, in wrapper
      return func(self, *args, **kwargs)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_tools/git_client.py", line 76, in prepare_ref
      self.run(['fetch', 'origin'], cwd=path)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_tools/git_client.py", line 118, in run
      raise GitCommandError(

  idf_component_tools.git_client.GitCommandError: 'git fetch origin' failed with exit code 128
  ssh: Could not resolve hostname github.com:dcsbl: nodename nor servname provided, or not known
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights and the repository exists.

Call Stack (most recent call first):
  ~/esp/esp-idf-v4.3/tools/cmake/project.cmake:296 (__project_init)
  CMakeLists.txt:9 (project)

DCSBL commented 2 years ago

Small update:

If I replace git: ssh://git@github.com:dcsbl/rgb-driver.git with git git@github.com:dcsbl/rgb-driver.git a different error is generated:

CMake Error at ~/esp/esp-idf-v4.3/tools/cmake/project.cmake:209 (message):
  Traceback (most recent call last):

    File "~/.pyenv/versions/3.9.4/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "~/.pyenv/versions/3.9.4/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/prepare_components/__main__.py", line 3, in <module>
      main()
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/prepare_components/prepare.py", line 110, in main
      args.func(args)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/prepare_components/prepare.py", line 37, in prepare_dep_dirs
      ComponentManager(args.project_dir).prepare_dep_dirs(
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/core.py", line 316, in prepare_dep_dirs
      downloaded_component_paths = download_project_dependencies(
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_manager/dependencies.py", line 49, in download_project_dependencies
      download_paths = ComponentFetcher(component, managed_components_path).download()
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_tools/sources/fetcher.py", line 32, in download
      return self.source.download(self.component, managed_path)
    File "~/.espressif/python_env/idf4.3_py3.9_env/lib/python3.9/site-packages/idf_component_tools/sources/git.py", line 88, in download
      shutil.copytree(source_path, download_path)
    File "~/.pyenv/versions/3.9.4/lib/python3.9/shutil.py", line 555, in copytree
      with os.scandir(src) as itr:

  FileNotFoundError: [Errno 2] No such file or directory:
  '~/Library/Caches/Espressif/ComponentManager/git_37fb9bb3/rgb_driver'

Call Stack (most recent call first):
  ~/esp/esp-idf-v4.3-beta3/tools/cmake/project.cmake:296 (__project_init)
  CMakeLists.txt:9 (project)

kumekay commented 2 years ago

Hi Duco, thank you for the issue, I've just tried and there is no issue with private repos, if the URL is set like, with a slash

dependencies:
  rgb_driver:
    path: rgb_driver
    git: ssh://git@github.com/dcsbl/rgb-driver.git

This should work for the first problem.

As for the second problem, is the component stored in the root of the repo or in a subdirectory named rgb_driver? If it's in the root you do not need to provide the path.

kumekay commented 2 years ago

But in both cases there is a room to improve error reporting, so please don't close the issue for now

DCSBL commented 2 years ago

Hi! Thanks for your quick response 👍 I've removed path and that didn't work. But when I removed ssh:// and path it all worked out, weirdly.

dependencies:
  rgb_driver:
    git: git@github.com:dcsbl/rgb-driver.git

  idf:
    version: ">=4.1.0"

I agree that some improvements can be made with error reporting.

kumekay commented 2 years ago

@DCSBL Did you try with ssh, but replacing ":" with "/" after .com: ssh://git@github.com/dcsbl/rgb-driver.git

DCSBL commented 2 years ago

Yes that works:

dependencies:
  rgb_driver:
    git: ssh://git@github.com/homewizard/rgb-driver.git

  idf:
    version: ">=4.1.0"

But why would you defer from the default URL structure given by GitHub (and GitLab)?

kumekay commented 2 years ago

But why would you defer from the default URL structure given by GitHub (and GitLab)?

It wasn't intentional, just a default behavior of python's urlparse function, we will support these default URLs too

kumekay commented 2 years ago

@DCSBL a small update after a further check of the issue:

ssh://git@github.com/homewizard/rgb-driver.git - works with component manager (Default GitLab url format)
git@github.com:dcsbl/rgb-driver.git - also works with component manager (Default GitHub format)
ssh://git@github.com:dcsbl/rgb-driver.git - won't work, but git clone ssh://git@github.com:dcsbl/rgb-driver.git also fails. It's not a correct URI. So, my bad, it wasn't a problem with urlparse

A second issue that you had with a manifest like this:

## IDF Component Manager Manifest File
dependencies:
  rgb_driver:
    path: rgb_driver
    git: git@github.com:dcsbl/rgb-driver.git

  idf:
    version: ">=4.1.0"

It failed with error:

 FileNotFoundError: [Errno 2] No such file or directory:
  '~/Library/Caches/Espressif/ComponentManager/git_37fb9bb3/rgb_driver'

Which actually means Cannot find directory "rgb_driver" in the repository "git@github.com:dcsbl/rgb-driver.git". We will update the error message.

kumekay commented 2 years ago

@DCSBL We updated an error message for the case when a directory doesn't exist in the git repo, so hopefully it will be clear now. Also, it works correctly with default gitlab/github SSH URLs, So I'm closing this issue now. Feel free to reopen if you think that further actions are required.

DCSBL commented 2 years ago

Thank you 👍

espressif / idf-component-manager