Failure in Windows and MacOS CI

Redfire75369 commented 2 years ago

Currently, MacOS CI is failing. It seems to be due to an issue in configure.py but I have no idea what it could be. Windows CI is also failing, which is very weird given it compiles just fine for me locally. https://github.com/Redfire75369/rust-mozjs/runs/8271056707

jdm commented 2 years ago

Ah, this is the same issue as https://github.com/servo/servo/issues/28911 and needs to be addressed like https://github.com/servo/servo/pull/28960.

sagudev commented 2 years ago

I believe SM update might solve this issues.

jdm commented 2 years ago

The windows errors look a bit like we may need an LLVM upgrade.

  C:\Program Files\LLVM\lib\clang\14.0.6\include\emmintrin.h:2378:19: error: use of undeclared identifier '__builtin_elementwise_max', err: true

Redfire75369 commented 2 years ago

I see upgrading LLVM fixed the windows builds, but I'm just wondering how it requires an llvm update even though the code didn't change. How did the currently used llvm version start to not work, if the code is just the same?

jdm commented 2 years ago

Because the MSVC installation in the CI environment is updated regularly, and those upgrades can include a new version of clang. However using bindgen in the build requires a full LLVM installation, which MSVC doesn't include, and so the version of LLVM that we install needs to remain in sync with the clang version that is present in MSVC.

jdm commented 2 years ago

I believe SM update might solve this issues.

I think you're right. It may be a similar problem to https://bugzilla.mozilla.org/show_bug.cgi?id=1657717

Redfire75369 commented 2 years ago

On another note, thanks for the information about MSVC in the CI. It helped me realise why spiderfire's CI was failing, I didn't peg the version so it just went to LLVM 15. 😅

It seems the Windows CI environment has LLVM already, would it be possible to use that instead of installing it with scoop? (https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md)

yvt commented 2 years ago

I believe SM update might solve this issues.

I think you're right. It may be a similar problem to https://bugzilla.mozilla.org/show_bug.cgi?id=1657717

The symptom is similar, but I think it's a separate issue. I was able to reproduce the issue locally and found that the whole _virtualenvs/init_py3/bin directory (not just the bin/python file as indicated by https://bugzilla.mozilla.org/show_bug.cgi?id=1657717#c1) was missing. The Python binaries were instead placed in a different directory, _virtualenvs/init_py3/opt/homebrew/bin:

$ ls -R target/release/build/mozjs_sys-30b55714e66609b8/out/build/_virtualenvs/init_py3
 ...
target/release/build/mozjs_sys-30b55714e66609b8/out/build/_virtualenvs/init_py3/opt/homebrew/bin:
activate      activate.fish  activate.xsh      easy_install       easy_install3     pip       pip3     python   python3.10  wheel-3.10  wheel3.10
activate.csh  activate.ps1   activate_this.py  easy_install-3.10  easy_install3.10  pip-3.10  pip3.10  python3  wheel       wheel3

It appears to be the combination of the following factors that caused this issue:

The current CI environment installs Python 3.10 by Homebrew. (It was 3.9 until a few months ago.)
The Homebrew Python 3.10 formula patches sysconfig to return Homebrew-specific system paths, e.g., /opt/homebrew/bin. (The Python 3.9 formula doesn't do this.)
virtualenv 20.2.2 (bundled with mozjs_sys) locates installation paths by calling sysconfig.get_path(i, expand=False). (I'm speculating that virtualenv contents are laid out according to this.)
The Homebrew Python 3.10 formula also (sort-of) back-ported the venv scheme from https://github.com/python/cpython/pull/31034 (officially added in Python 3.11). Tools like virtualenv must pass this scheme to sysconfig.get_path to get the correct paths, e.g., {base}/bin, for virtual environments.
However, the support for the venv scheme was absent until virtualenv 20.10.0 (where https://github.com/pypa/virtualenv/pull/2209 was merged), so virtualenv 20.2.2 happily lays out virtualenv contents according to the Homebrew-specific installation paths, breaking our tools.

Edit: Found the upstream issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1766497

Redfire75369 commented 2 years ago

From a first read through, I think this would require a patch to update virtualenv to 20.10.0? At least until an update to SM97 or further.

I'm not sure if mozjs/python/mozbuild/mozbuild/virtualenv.py needs to be patched, but mozjs/third_party/python/virtualenv/ probably does.

yvt commented 2 years ago

Looks like they attempted to bump virtualenv to 20.15.0 in https://hg.mozilla.org/mozilla-central/rev/450e42238ace99a6e8ff5fc167b44c6a734e8e6d, although it was reverted later due to CI regressions. I think we can just incorporate this patch until a reliable solution is implemented in upstream.

Meanwhile, I think I've found the right upstream issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1766497

Alternatively, we could temporarily downgrade Python to 3.9 in the CI environment.

servo / rust-mozjs

Failure in Windows and MacOS CI #559