Load component from git repository using ssh + key verification (PACMAN-572)

[JulianBeran]

The Component Manager version

v1.2.1

ESP-IDF Version

v5.0

python Version

3.8.7

Operating System

Windows 10

Browser (for https://components.espressif.com Issues)

Chrome

Description

If you try to add a repo by ssh using ssh key authentication results in an error like this:

idf_component_tools.git_client.GitCommandError: 'git fetch origin' failed
  with exit code 128

  git@github.com: Permission denied (publickey).
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.

if i use git via command line: git clone git@github.com:user/test-component.git it will promt for the passphrase and then go on and clone the repository.

The configuration of ssh seams to work, but not with component manager.

It looks like the component manger is not using the right authentication method.

In the future I only want to use ssh authentication on my own github, but for that this has to work.

To Reproduce

1. Setup a repository with an example component "test-component" and push it on github. 
2. Setup ssh key authentication on github (keygen, upload, ssh config like below)
3. Write `idf_component.yml` like below
4. build idf project

~/.ssh/config

Host github.com
  User user
  IdentityFile ~/.ssh/ida_rsa_github

idf_component.yml

dependencies:  
  idf: ">=5.0"

  test-component:
    git: git@github.com:user/test-component.git

Expected behaviour

Use of the saved ssh key, like using git with git@... syntax via command line and prompting for the ssh key passphrase.

Pulling the component from git, configure as external component in my idf-project and compile.

Additional info

CMake Error at C:/Users/user/esp_idf_5.0/esp-idf/tools/cmake/build.cmake:519 (message):
  Traceback (most recent call last):

    File "runpy.py", line 194, in _run_module_as_main
    File "runpy.py", line 87, in _run_code
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_manager\prepare_components\__main__.py", line 6, in <module>        
      main()
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_manager\prepare_components\prepare.py", line 124, in main
      args.func(args)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_manager\prepare_components\prepare.py", line 27, in prepare_dep_dirs
      ComponentManager(
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_manager\core.py", line 62, in wrapper
      return func(self, *args, **kwargs)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_manager\core.py", line 438, in prepare_dep_dirs
      downloaded_component_paths, downloaded_component_version_dict = download_project_dependencies(
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_manager\dependencies.py", line 106, in download_project_dependencies
      if is_solve_required(project_requirements, solution):
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_manager\dependencies.py", line 76, in is_solve_required
      if project_requirements.manifest_hash != solution.manifest_hash\
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\manifest\manifest.py", line 310, in manifest_hash
      manifest_hashes = [manifest.manifest_hash for manifest in self.manifests]
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\manifest\manifest.py", line 310, in <listcomp>
      manifest_hashes = [manifest.manifest_hash for manifest in self.manifests]
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\manifest\manifest.py", line 145, in manifest_hash
      serialized = self.serialize(serialize_default=False)  # type: ignore
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\serialization.py", line 63, in _serialize
      serialization_properties[prop] = serialize(getattr(self, prop), serialize_default)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\serialization.py", line 31, in serialize
      return [serialize(v, serialize_default) for v in value]
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\serialization.py", line 31, in <listcomp>
      return [serialize(v, serialize_default) for v in value]
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\serialization.py", line 34, in serialize
      return value.serialize(serialize_default)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\serialization.py", line 63, in _serialize
      serialization_properties[prop] = serialize(getattr(self, prop), serialize_default)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\manifest\manifest.py", line 195, in version_spec
      return self.source.normalize_spec(self._version_spec)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\sources\git.py", line 196, in normalize_spec
      commit_id = self._client.get_commit_id_by_ref(self.git_repo, self.cache_path(), ref)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\git_client.py", line 41, in wrapper
      return func(self, *args, **kwargs)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\git_client.py", line 69, in wrapper
      self._update_bare_repo(*args, **kwargs)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\git_client.py", line 63, in _update_bare_repo
      self.run(['fetch', 'origin'], cwd=bare_path)
    File "C:\Users\user\.espressif_tools_5.0\python_env\idf5.0_py3.8_env\lib\site-packages\idf_component_tools\git_client.py", line 183, in run
      raise GitCommandError(

  idf_component_tools.git_client.GitCommandError: 'git fetch origin' failed
  with exit code 128

  git@github.com: Permission denied (publickey).

  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights

  and the repository exists.

Call Stack (most recent call first):
  C:/Users/user/esp_idf_5.0/esp-idf/tools/cmake/project.cmake:440 (idf_build_process)
  CMakeLists.txt:6 (project)

I have checked existing issues and online Documentation

• [X] I confirm I have checked existing issues and online Documentation.

[kumekay]

@JulianBeran Thank you for bringing this issue to our attention. We will review it as soon as possible.

[kumekay]

Hello @JulianBeran

The problem you're facing has two parts. Firstly, the Python traceback during an ssh error will be fixed in the upcoming release.

Regarding the ssh keyphrase issue, it requires a different solution. The idf.py build command is non-interactive and cannot prompt for a password. To resolve this limitation for non-interactive programs, it's recommended to use ssh-agent - a separate program that manages your keys and passphrases.

You can find some instructions for setting up ssh-agent on GitHub docs, but they are focused on git bash for Windows.

I also found a couple of tutorials with a clearer explanation for Windows, but I don't have a Windows machine to test them:

• https://snowdrift.tech/cli/ssh/git/tutorials/2019/01/31/using-ssh-agent-git-windows.html
• https://interworks.com/blog/2021/09/15/setting-up-ssh-agent-in-windows-for-passwordless-git-authentication/ (Last 2 paragraphs)

[kumekay]

@JulianBeran

The stack trace part of this issue has been resolved in the commit at https://github.com/espressif/idf-component-manager/commit/798908a680205a0d00297ac4fb24bb0b485ebfe8.

I am now closing this issue. If you need any more help, please feel free to reopen the issue.