Builds for Helix Dockerfiles are all failing setting up virtual environment

[mthalman]

This command is failing: RUN python3 -m virtualenv /home/helixbot/.vsts-env

#9 0.494 RuntimeError: failed to build image pip, setuptools, wheel because:
#9 0.494 Traceback (most recent call last):
#9 0.494   File "/usr/local/lib/python3.10/dist-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 54, in _install
#9 0.494     with parent.non_reentrant_lock_for_key(wheel_img.name):
#9 0.494   File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
#9 0.494     return next(self.gen)
#9 0.494   File "/usr/local/lib/python3.10/dist-packages/virtualenv/util/lock.py", line 147, in non_reentrant_lock_for_key
#9 0.494     with _CountedFileLock(str(self.path / "{}.lock".format(name))):
#9 0.494   File "/usr/local/lib/python3.10/dist-packages/virtualenv/util/lock.py", line 24, in __init__
#9 0.494     super(_CountedFileLock, self).__init__(lock_file)
#9 0.494 TypeError: object.__init__() takes exactly one argument (the instance to initialize)
#9 0.494 
#9 0.494 Traceback (most recent call last):
#9 0.494   File "/usr/local/lib/python3.10/dist-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 54, in _install
#9 0.494     with parent.non_reentrant_lock_for_key(wheel_img.name):
#9 0.494   File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
#9 0.494     return next(self.gen)
#9 0.494   File "/usr/local/lib/python3.10/dist-packages/virtualenv/util/lock.py", line 147, in non_reentrant_lock_for_key
#9 0.494     with _CountedFileLock(str(self.path / "{}.lock".format(name))):
#9 0.494   File "/usr/local/lib/python3.10/dist-packages/virtualenv/util/lock.py", line 24, in __init__
#9 0.494     super(_CountedFileLock, self).__init__(lock_file)
#9 0.494 TypeError: object.__init__() takes exactly one argument (the instance to initialize)
#9 0.494 
#9 0.494 Traceback (most recent call last):
#9 0.494   File "/usr/local/lib/python3.10/dist-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 54, in _install
#9 0.494     with parent.non_reentrant_lock_for_key(wheel_img.name):
#9 0.494   File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
#9 0.494     return next(self.gen)
#9 0.494   File "/usr/local/lib/python3.10/dist-packages/virtualenv/util/lock.py", line 147, in non_reentrant_lock_for_key
#9 0.494     with _CountedFileLock(str(self.path / "{}.lock".format(name))):
#9 0.494   File "/usr/local/lib/python3.10/dist-packages/virtualenv/util/lock.py", line 24, in __init__
#9 0.494     super(_CountedFileLock, self).__init__(lock_file)
#9 0.494 TypeError: object.__init__() takes exactly one argument (the instance to initialize)
#9 0.494 
#9 0.519 Exception ignored in: <function BaseFileLock.__del__ at 0xf097375f3ac0>
#9 0.521 Traceback (most recent call last):
#9 0.521   File "/usr/local/lib/python3.10/dist-packages/filelock/_api.py", line 365, in __del__
#9 0.521     self.release(force=True)
#9 0.521   File "/usr/local/lib/python3.10/dist-packages/virtualenv/util/lock.py", line 35, in release
#9 0.522     with self.thread_safe:
#9 0.522 AttributeError: '_CountedFileLock' object has no attribute 'thread_safe'
#9 0.522 Exception ignored in: <function BaseFileLock.__del__ at 0xf097375f3ac0>
#9 0.522 Traceback (most recent call last):
#9 0.522   File "/usr/local/lib/python3.10/dist-packages/filelock/_api.py", line 365, in __del__
#9 0.522     self.release(force=True)
#9 0.522   File "/usr/local/lib/python3.10/dist-packages/virtualenv/util/lock.py", line 35, in release
#9 0.522     with self.thread_safe:
#9 0.522 AttributeError: '_CountedFileLock' object has no attribute 'thread_safe'
#9 0.522 Exception ignored in: <function BaseFileLock.__del__ at 0xf097375f3ac0>
#9 0.522 Traceback (most recent call last):
#9 0.522   File "/usr/local/lib/python3.10/dist-packages/filelock/_api.py", line 365, in __del__
#9 0.522     self.release(force=True)
#9 0.523   File "/usr/local/lib/python3.10/dist-packages/virtualenv/util/lock.py", line 35, in release
#9 0.523     with self.thread_safe:
#9 0.523 AttributeError: '_CountedFileLock' object has no attribute 'thread_safe'
#9 ERROR: process "/bin/sh -c python -m virtualenv /home/helixbot/.vsts-env" did not complete successfully: exit code: 1

Example build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=705519&view=results

[lbussell]

[Triage] @garath Helix python setup is failing in Helix Docker images.

Is this related to yesterday's Helix rollout?

[garath]

Is this related to yesterday's Helix rollout?

Possibly... I'm not sure how to debug this. Do you have any pointers? What is this pipeline doing?

[mthalman]

It's just building Dockerfiles. You can repro by building this Dockerfile, for example.

[garath]

So you suspect there was a change dnceng made that was deployed in the last rollout that broke this? (Sorry for being pedantic, I'm not seeing anything familiar or anything obviously connecting it to the rollout so I'm trying to find some touchpoints.)

[mthalman]

Nobody on our team is very knowledgeable on Python so we're just guessing here. But this is what might be the cause: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/a6ccb1abd3323eaa97bdb2a73c0a62c52c2225d6/src/alpine/3.18/helix/amd64/Dockerfile#L67

So if something recently rolled out there it could impact the behavior in the build here.

[mthalman]

It also might be purely in the realm of a Python change, unrelated to Helix. We're not sure yet.

[garath]

Looking at the change history... I see no changes related to either the helix client or the version of Python we install on machines. Hmm...

[lbussell]

Nobody on our team is very knowledgeable on Python

... or Helix

[richlander]

Related: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/issues/1093

[richlander]

When helix images are launched, what does that look like? Does anyone have a link to the script that does that? That would be super useful.