search

pantsbuild / scie-pants

Protects your Pants from the elements.
https://www.pantsbuild.org/docs/installation
Apache License 2.0
19 stars 19 forks source link

Tests fail on Mac M1 #92

Closed kaos closed 1 year ago

kaos commented 1 year ago

I have some issue running the tests as below.

This on current main 4f06c9de4e99e667bacde43c6367accd930b7013

$ cargo run -p package -- test --tools-pex-mismatch-warn
   Compiling package v0.2.0 (/Users/andreas.stenius/src/github/kaos/scie-pants/package)
    Finished dev [unoptimized + debuginfo] target(s) in 2.18s
     Running `target/debug/package test --tools-pex-mismatch-warn`
 1.) Loading the `ptex` v0.6.0 binary from the cache...
 2.) Loading the `jump` v0.10.0 binary from the cache...
 3.) Executing scie-jump boot-pack of the `pbt` helper binary...
/Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/pbt/lift.json: /Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/pbt/pbt
 4.) Building the scie_pants `tools.pex`...
 5.) Building the scie-pants Rust binary....
  Installing scie-pants v0.3.2 (/Users/andreas.stenius/src/github/kaos/scie-pants)
    Updating crates.io index
   Compiling scie-pants v0.3.2 (/Users/andreas.stenius/src/github/kaos/scie-pants)
    Finished release [optimized] target(s) in 6.67s
   Replacing /Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/bin/scie-pants
    Replaced package `scie-pants v0.3.2 (/Users/andreas.stenius/src/github/kaos/scie-pants)` with `scie-pants v0.3.2 (/Users/andreas.stenius/src/github/kaos/scie-pants)` (executable `scie-pants`)
 6.) Building the `scie-pants` scie...
/Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/scie-pants/lift.json: /Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/scie-pants/scie-pants
 7.) Running smoke tests...
>> Linting, testing and packaging the tools codebase
21:22:27.27 [INFO] Initializing scheduler...
21:22:27.63 [INFO] Scheduler initialized.
21:22:30.20 [INFO] Completed: Format with Black - black made no changes.
21:22:31.19 [INFO] Completed: Format with isort - isort made no changes.

✓ black made no changes.
✓ isort made no changes.

✓ black succeeded.
✓ isort succeeded.
21:22:31.47 [INFO] Completed: Typecheck using MyPy - mypy - mypy succeeded.
Success: no issues found in 13 source files

✓ mypy succeeded.
21:22:31.50 [INFO] Completed: Run Pytest - tools/tests/test_bootstrap_tools.py - succeeded.
21:22:31.50 [INFO] Completed: Run Pytest - tools/tests/test_pants_bootstrap.py - succeeded.

✓ tools/tests/test_bootstrap_tools.py succeeded in 1.08s (cached locally).
✓ tools/tests/test_pants_bootstrap.py succeeded in 0.48s (cached locally).
21:22:31.51 [INFO] Wrote dist/tools/tools.pex
>> Checking .pants.bootstrap handling ignores bash functions
2.15.0rc0
>> Verifying the tools.pex built by the package crate matches the tools.pex built by Pants
The tools.pex generated by Pants does not match ours:
Ours:  /Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/tools.pex
->     50dc7700a3e0372a096b2a425c0cad857b2026224940a18d48ca14cd0edb4369
Pants: /Users/andreas.stenius/src/github/kaos/scie-pants/dist/tools/tools.pex
->     03a341070cfe94b408b90074330ca54fe825f9b2966b0a0320f4e69606fa0717

>> Verifying PANTS_BOOTSTRAP_TOOLS works
python_distribution_hash=64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab pants_version=2.15.0rc0
>> Verifying PANTS_SHA is respected
Bootstrapping Pants 2.14.0a0+git8e381dbf using cpython 3.9.15
Installing pantsbuild.pants==2.14.0a0+git8e381dbf into a virtual environment at /Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.14.0a0+git8e381dbf

[notice] A new release of pip available: 22.3.1 -> 23.0
[notice] To update, run: python -m pip install --upgrade pip
New virtual environment successfully created at /Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.14.0a0+git8e381dbf.
21:22:58.25 [INFO] Initializing scheduler...
21:22:58.40 [INFO] Scheduler initialized.
2.14.0a0+git8e381dbf
>> Verifying --python-repos-repos is used prior to Pants 2.13 (no warnings should be issued by Pants)
Bootstrapping Pants 2.12.1 using cpython 3.9.15
Installing pantsbuild.pants==2.12.1 into a virtual environment at /Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1

[notice] A new release of pip available: 22.3.1 -> 23.0
[notice] To update, run: python -m pip install --upgrade pip
New virtual environment successfully created at /Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1.
Traceback (most recent call last):
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 115, in main
    PantsLoader.main()
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 111, in main
    cls.run_default_entrypoint()
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 64, in run
    options_bootstrapper = OptionsBootstrapper.create(
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/option/options_bootstrapper.py", line 175, in create
    alias = CliAlias.from_dict(alias_dict)
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/option/alias.py", line 79, in from_dict
    definitions = {key: tuple(shlex.split(value)) for key, value in aliases.items()}
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/option/alias.py", line 79, in <dictcomp>
    definitions = {key: tuple(shlex.split(value)) for key, value in aliases.items()}
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 315, in split
    return list(lex)
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 300, in __next__
    token = self.get_token()
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 109, in get_token
    raw = self.read_token()
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 140, in read_token
    nextchar = self.instream.read(1)
AttributeError: 'dict' object has no attribute 'read'
Error: Command "/Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/scie-pants/scie-pants-macos-aarch64" "--no-verify-config" "-V" failed with exit code: Some(1)
STDOUT not captured.
STDERR not captured.
kaos commented 1 year ago

Is there something to clean up to start fresh.. in case I have gotten into a broken state?

jsirois commented 1 year ago

Well, so, you in particular are an interesting user here. Forgetting the immediate context and focusing on just Pants:

    alias = CliAlias.from_dict(alias_dict)
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/option/alias.py", line 79, in from_dict
    definitions = {key: tuple(shlex.split(value)) for key, value in aliases.items()}
...
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 140, in read_token
    nextchar = self.instream.read(1)
AttributeError: 'dict' object has no attribute 'read'

Do you have .pants.rc or some other similar config that might have CLI alias definitions not compatible with Pants 2.12 / exposing an old bug?

jsirois commented 1 year ago

To answer your question though, no. I've never seen anything like this at all and neither has the Mac ARM CI job.

jsirois commented 1 year ago

This is Python; so the easiest debug step is to modify the py file in the nce cache directly to print the CLI alias dict just prior to stacktrace. That should help move things forward. Just don't forget to blow away (that portion of) the nce when all debugging is done.

jsirois commented 1 year ago

IOW: scie-pants is just executing pants from a venv it built earlier under the nce. You can experiment and debug in that venv directly.

kaos commented 1 year ago

Thanks for the pointers @jsirois

Indeed I have custom aliases in a ~/.pants.rc file. I did notice this was in the vicinity of cli aliases, but didn't connect (too late at night again?). Also the issue reported by the unexpected tools.pex checksum had me thinking my setup could be borked.

I'll dig into this 👍🏽

jsirois commented 1 year ago

Ok, great. FWIW the checksum issue is well known and fully understood: https://github.com/pantsbuild/scie-pants/issues/2

kaos commented 1 year ago

This was caused by my ~/.pants.rc having configuration that was incompatible with pants 2.12.

Adding PANTS_PANTSRC=False globally for all smoke tests in #75