FileExistsError: [Errno 17] File exists: '__sources__/ruamel/' -> '/root/miniforge3/envs/<REDACTED>/lib/python3.10/site-packages/ez_yaml/__dependencies__/ruamel'

[bjfar]

I am getting this error importing ez_yaml in one my CI jobs and I'm trying to track down the cause (its a bit difficult because it only seems to happen in the CI...)

/root/git_repos/<REDACTED>.py:6: in <module>
    import ez_yaml
/root/miniforge3/envs/<REDACTED>/lib/python3.10/site-packages/ez_yaml/__init__.py:1: in <module>
    from .__dependencies__ import ruamel
/root/miniforge3/envs/<REDACTED>/lib/python3.10/site-packages/ez_yaml/__dependencies__/__init__.py:171: in <module>
    target_path_obj.symlink_to(dependency_info["path"])
/root/miniforge3/envs/<REDACTED>/lib/python3.10/pathlib.py:1255: in symlink_to
    self._accessor.symlink(target, self, target_is_directory)
E   FileExistsError: [Errno 17] File exists: '__sources__/ruamel/' -> '/root/miniforge3/envs/<REDACTED>/lib/python3.10/site-packages/ez_yaml/__dependencies__/ruamel'

Do you know what the problem could be? I am using a mainly conda environment with a few extra pip things installed on top, including ez_yaml. I assume it is some sort of environment configuration issue, so I tried some things like installing ruamel with conda before installing ez_yaml with pip. And it seemed to work at first, but then the error started coming back. I'm still confused at what is different between runs where it works and runs where it doesn't.

But then I looked in the code more and it looks like ez_yaml comes with its own version of ruamel? So I should probably not install a different version of it with conda first? But then I am back to square one with regarding to figuring out what the problem is...

[jeff-hykin]

Thanks for reporting this

But then I looked in the code more and it looks like ez_yaml comes with its own version of ruamel?

It does but it actually is done in a way to NOT conflict with an already installed version of ruamel yaml. I "bundled" ruamel yaml specifically so that you could install any version of ruamel yaml and it not affect the behavior of ez_yaml.

It looks like adding a check to ez_yaml should fix the problem. Which is odd because it should already be checking if that symlink exists.

I'll take a look at it

[jeff-hykin]

@bjfar whats strange is, the error says file exists, but on the line above the python removes/deletes that file (to ensure this error doesn't happen). Which means its really weird that it says the file exists.

Is it possible to edit the venv file in your CI? or do you know if it ignores file-delete requests

[bjfar]

@bjfar whats strange is, the error says file exists, but on the line above the python removes/deletes that file (to ensure this error doesn't happen). Which means its really weird that it says the file exists.

Is it possible to edit the venv file in your CI? or do you know if it ignores file-delete requests

Yeah I have full control of the CI so can edit anything. I am running these tests using pytest in parallel, so maybe it is some weird race condition? I'll try a few things to see if I can influence the behaviour, i.e. turn off parallel testing, turn off the tests that test the command line interface (which will be launching a bunch of subprocesses in parallel etc). I've never been able to trigger it locally though.

I don't think the OS should be anything weird, this is all running in a docker container built on the ubuntu:jammy image, so it should be pretty normal.

[jeff-hykin]

If you edit the file, add a try catch for file exists error, and that works, then I can add the try catch to the codebase and publish a new version

[bjfar]

I'll see what I can do. In the meantime, I turned off parallelisation of the tests in the CI and the problem went away. So I guess it is some sort of race condition setting up this symlink.