search

pyodide / pyodide

Pyodide is a Python distribution for the browser and Node.js based on WebAssembly
https://pyodide.org/en/stable/
Mozilla Public License 2.0
11.75k stars 795 forks source link

Mac builds: Pyodide Venv: `pytest` breaks when `python -m pytest` does not #4802

Open hoodmane opened 1 month ago

hoodmane commented 1 month ago

🐛 Bug

See comment here: https://github.com/pypa/cibuildwheel/pull/1456/files#r1615308216

mayeut commented 1 month ago

It seems that the issue only happens on macOS (either arm64 or x86_64). I tested on linux (aarch64/x86_64) and there were no issues using pytest directly.

hoodmane commented 1 month ago

@henryiii @mayeut can either of you reproduce this locally?

mayeut commented 1 month ago

I think I narrowed down the issue a bit. All entry points are broken (not just pytest) on macOS. This boils down to the fact that the shebang in the entry points scripts refer to .pyodide-xbuildenv-0.26.0/0.26.0/xbuildenv/pyodide-root/dist/python which itself is a script. macOS only accepts binaries for the shebang. So the shebang is simply ignored with bash/zsh simply trying to execute remaining lines which gives the error message:

/private/var/folders/hb/2n_7f3yn20v06lzz3129dtw80000gn/T/cibw-run-zmblysfg/cp312-pyodide_wasm32/venv-test/bin/pytest: line 3: import: command not found
/private/var/folders/hb/2n_7f3yn20v06lzz3129dtw80000gn/T/cibw-run-zmblysfg/cp312-pyodide_wasm32/venv-test/bin/pytest: line 4: import: command not found
/private/var/folders/hb/2n_7f3yn20v06lzz3129dtw80000gn/T/cibw-run-zmblysfg/cp312-pyodide_wasm32/venv-test/bin/pytest: line 5: from: command not found
/private/var/folders/hb/2n_7f3yn20v06lzz3129dtw80000gn/T/cibw-run-zmblysfg/cp312-pyodide_wasm32/venv-test/bin/pytest: line 7: syntax error near unexpected token `('
/private/var/folders/hb/2n_7f3yn20v06lzz3129dtw80000gn/T/cibw-run-zmblysfg/cp312-pyodide_wasm32/venv-test/bin/pytest: line 7: `    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'
hoodmane commented 1 month ago

Ah interesting...

hoodmane commented 1 month ago

Maybe we can make a tiny binary that just shells out to this script?

hoodmane commented 1 month ago

Oh but it might have to be the same file. So we'd be looking to make a polyglot which is simultaneously a valid js file and a valid darwin binary. That sounds hard...

hoodmane commented 1 month ago

There's some discussion here but not much details:

Some script language (JavaScript, HTML, PHP, Ruby...) tolerate binary contents and can be combined with binary formats.

https://github.com/corkami/mitra#script-polyglots

hoodmane commented 1 month ago

I think this may be in the category of too complicated to be worth our limited development time, unless someone wants to work on it. We can tell people to build on linux.

ryanking13 commented 1 month ago

I think this may be in the category of too complicated to be worth our limited development time, unless someone wants to work on it. We can tell people to build on linux.

Yes, I think it is okay to tell people to use linux for now. As you said, it seems like too much effort compared to what we can get out of it.

mayeut commented 1 month ago

@hoodmane, can you edit the title, it might be misleading for linux users.