agentos-project / agentos

The Python Component System (PCS) is an API and CLI for building, running, and sharing Python code. AgentOS is a set of libraries built on top of PCS that make it easy to build, run, and share agents that use Reinforcement Learning.
https://agentos.org
Apache License 2.0
13 stars 4 forks source link

`GitManager._checkout()` fails in windows if repo has a symlink in it #417

Open andyk opened 2 years ago

andyk commented 2 years ago

Our windows tests started failing when the SB3 repo (the agentos freeze test uses) added a symlink (on June 22 2022) to their repo: https://github.com/mgbellemare/Arcade-Learning-Environment/blob/master/src/python/roms/md5.txt

The file is getting created in my windows 11 filesystem but it is not a valid symlink invalid_file_screenshot

The error is in Dulwich's porcelain.py file: https://github.com/jelmer/dulwich/blob/bbf100b63fb00ebb4a1d0ca698b1abe337611bae/dulwich/porcelain.py#L230 in path_to_tree_path() in a call to Path.resolve() on the symlinked file.

The full failed test stack trace is:


    def test_cli_freeze(cli_runner, tmpdir):
        run_args = ["sb3_agent", "-f"]
        run_kwargs = {"--registry-file": str(SB3_AGENT_DIR / "components.yaml")}
>       run_test_command(
            cli_runner, freeze, cli_args=run_args, cli_kwargs=run_kwargs
        )

C:\Users\andyk\Development\agentos\tests\test_core.py:57: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Users\andyk\Development\agentos\tests\utils.py:32: in run_test_command
    result = cli_runner.invoke(
C:\Users\andyk\Development\agentos\tests\conftest.py:26: in wrapper
    result = f(*args, **kwargs)
C:\Users\andyk\Development\agentos\windows_dev\lib\site-packages\click\testing.py:408: in invoke
    return_value = cli.main(args=args or (), prog_name=prog_name, **extra)
C:\Users\andyk\Development\agentos\windows_dev\lib\site-packages\click\core.py:1062: in main
    rv = self.invoke(ctx)
C:\Users\andyk\Development\agentos\windows_dev\lib\site-packages\click\core.py:1404: in invoke
    return ctx.invoke(self.callback, **ctx.params)
C:\Users\andyk\Development\agentos\windows_dev\lib\site-packages\click\core.py:763: in invoke
    return __callback(*args, **kwargs)
C:\Users\andyk\Development\agentos\agentos\cli.py:269: in freeze
    frozen_reg = module.freeze(force=force).to_registry()
C:\Users\andyk\Development\agentos\pcs\instance_manager.py:46: in freeze
    find_and_replace_leaves(
C:\Users\andyk\Development\agentos\pcs\utils.py:220: in find_and_replace_leaves
    leaf_replace(data_struct, leaf_list, replace_fn)
C:\Users\andyk\Development\agentos\pcs\utils.py:156: in leaf_replace
    data_struct[leaf_list[0]] = replacement_fn(next_inner)
C:\Users\andyk\Development\agentos\pcs\instance_manager.py:49: in <lambda>
    lambda x: x.freeze(force),
C:\Users\andyk\Development\agentos\pcs\class_manager.py:86: in freeze
    self_copy.module = self.module.freeze(force)
C:\Users\andyk\Development\agentos\pcs\module_manager.py:184: in freeze
    return self.to_versioned_module(force)
C:\Users\andyk\Development\agentos\pcs\module_manager.py:156: in to_versioned_module
    repo_url, version = self.repo.get_version_from_git(
C:\Users\andyk\Development\agentos\pcs\repo.py:73: in get_version_from_git
    full_path = self.get_local_file_path(file_path, version)
C:\Users\andyk\Development\agentos\pcs\repo.py:123: in get_local_file_path
    local_repo_path = self.get_local_repo_dir(version)
C:\Users\andyk\Development\agentos\pcs\repo.py:115: in get_local_repo_dir
    local_repo_path = self.GIT.clone_repo(
C:\Users\andyk\Development\agentos\pcs\git_manager.py:295: in clone_repo
    self._clone_repo(
C:\Users\andyk\Development\agentos\pcs\git_manager.py:317: in _clone_repo
    self._checkout_version(clone_destination, version)
C:\Users\andyk\Development\agentos\pcs\git_manager.py:331: in _checkout_version
    self._checkout(repo=repo, target=treeish, force=True)
C:\Users\andyk\Development\agentos\pcs\git_manager.py:473: in _checkout
    for file in porcelain.get_untracked_paths(
C:\Users\andyk\Development\agentos\windows_dev\lib\site-packages\dulwich\porcelain.py:1283: in get_untracked_paths
    ip = path_to_tree_path(basepath, ap)
C:\Users\andyk\Development\agentos\windows_dev\lib\site-packages\dulwich\porcelain.py:233: in path_to_tree_path
    resolved_path = path.resolve()
C:\Python39\lib\pathlib.py:1204: in resolve
    s = self._flavour.resolve(self, strict=strict)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pathlib._WindowsFlavour object at 0x0000018C2CDBFA30>
path = WindowsPath('C:/Users/andyk/.agentos/cache/repos_cache/mgbellemare/Arcade-Learning-Environment/db3728264f382402120913d76c4fa0dc320ef59f/src/python/roms/md5.txt')
strict = False

    def resolve(self, path, strict=False):
        s = str(path)
        if not s:
            return os.getcwd()
        previous_s = None
        if _getfinalpathname is not None:
            if strict:
                return self._ext_to_normal(_getfinalpathname(s))
            else:
                tail_parts = []  # End of the path after the first one not found
                while True:
                    try:
>                       s = self._ext_to_normal(_getfinalpathname(s))
E                       OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'C:\\Users\\andyk\\.agentos\\cache\\repos_cache\\mgbellemare\\Arcade-Learning-Environment\\db3728264f382402120913d76c4fa0dc320ef59f\\src\\python\\roms\\md5.txt'

C:\Python39\lib\pathlib.py:205: OSError