search

mrk-its / rust-mos

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
55 stars 7 forks source link

Error running `create_most_targets.py` (last `rust-mos` installation step) #2

Open U007D opened 2 years ago

U007D commented 2 years ago

Transferred from email thread:

Hi, @mrk-its,

I was delighted to see your rust-mos​ crate.

I followed the installation instructions and am getting a failure on the very last line. (So close!)

I am wondering if you recognize the issue and can give me a clue on how to resolve it.

First, I did the following:

llvm-mos

(Note the change to -DLIBXML2_LIBRARY​, as I am on Mac OS X.) Build succeeded​:

git clone https://github.com/llvm-mos/llvm-mos

cd llvm-mos
cmake -C clang/cmake/caches/MOS.cmake -G "Ninja" -S llvm -B build \
   -DLLVM_INSTALL_TOOLCHAIN_ONLY=OFF \
   -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON \
   -DLLVM_INSTALL_UTILS=ON -DLLVM_BUILD_UTILS=ON -DLLVM_TOOLCHAIN_UTILITIES=FileCheck \
   -DLLVM_TOOLCHAIN_TOOLS="llvm-addr2line;llvm-ar;llvm-cxxfilt;llvm-dwarfdump;llvm-mc;llvm-nm;llvm-objcopy;llvm-objdump;llvm-ranlib;llvm-readelf;llvm-readobj;llvm-size;llvm-strings;llvm-strip;llvm-symbolizer;llvm-config;llc" \
   -DLIBXML2_LIBRARY=/usr/local/opt/libxml2/lib/libxml2.dylib \
   -DLLVM_TARGETS_TO_BUILD="MOS;X86" \
   -DLLVM_ENABLE_PROJECTS="clang;lld;lldb"
cmake --build build -t install

llvm-mos-sdk​

(No changes to your instructions.) Build succeeded:

git clone https://github.com/llvm-mos/llvm-mos-sdk
cd llvm-mos-sdk
cmake -G "Ninja" -B build
cmake --build build -t install

rust-mos​

(Build succeeded, but create_mos_targets.py​ step is failing, with both default and custom $RUST_TARGET_PATH​):

export RUST_TARGET_PATH=/Users/sergibli/Development/3rd_party/llvm-mos/rust-mos/bin
cp config.toml.example config.toml
# in config.toml adjust path to llvm-config
# if llvm-mos is installed to other than /usr/local prefix
  # I uncommented and adjusted `prefix = "/Users/sergibli/Development/3rd_party/llvm-mos/rust-mos/"`,
  # as I was unsure if `sudo` permissions were a problem for installation (probably not?)
./x.py build -i --stage 0 src/tools/cargo
./x.py build -i && (
    ln -s ../../stage0-tools-bin/cargo build/x86_64-apple-darwin/stage1/bin/cargo
    rustup toolchain link mos build/x86_64-apple-darwin/stage1
    rustup default mos
    mkdir -p $RUST_TARGET_PATH
    python3 create_mos_targets.py $RUST_TARGET_PATH
)

gives:

   ...
   Compiling rustdoc v0.0.0 (/Users/sergibli/Development/3rd_party/llvm-mos/rust-mos/src/librustdoc)
   Compiling rustdoc-tool v0.0.0 (/Users/sergibli/Development/3rd_party/llvm-mos/rust-mos/src/tools/rustdoc)
    Finished release [optimized] target(s) in 1m 35s
Build completed successfully in 0:03:13
info: default toolchain set to 'mos'
Traceback (most recent call last):
  File "/Users/sergibli/Development/3rd_party/llvm-mos/rust-mos/create_mos_targets.py", line 15, in 
    target_spec = json.loads(
  File "/Users/sergibli/.pyenv/versions/3.9.6/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/Users/sergibli/.pyenv/versions/3.9.6/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/sergibli/.pyenv/versions/3.9.6/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Examining create_mos_targets.py​ shows that l.15 shells out to rustup​. Running that line directly gives:

▶ rustup run mos rustc --target mos-unknown-none -Z unstable-options --print target-spec-json
rustc -Cllvm-args="..." with: Unknown command line argument '--align-large-globals=false'.  Try: 'rustc -Cllvm-args="..." with --help'
rustc -Cllvm-args="..." with: Did you mean '--asan-opt-globals=false'?

--target mos-unknown-none​ seems to be the culprit, even though the target does exist:

▶ rustup run mos rustc -Z unstable-options --print target-spec-json
{
  "abi-return-struct-as-int": true,
  "arch": "x86_64",
  "archive-format": "darwin",
  "cpu": "core2",
  "data-layout": "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128",
  "dll-suffix": ".dylib",
  ...

The rustup​ mos​ toolchain is present:

▶ rustup toolchain list
stable-x86_64-apple-darwin
nightly-2020-07-27-x86_64-apple-darwin
nightly-2021-06-15-x86_64-apple-darwin
nightly-x86_64-apple-darwin
1.55-x86_64-apple-darwin
mos (default)
1.56.0-x86_64-apple-darwin

and mos​ links to the stage1 llvm-mos​ build:

sergibli@HephaestusMobi:~/.rustup/toolchains
▶ l
total 0
drwxr-xr-x  9 sergibli  staff  288 Apr 24 14:50 .
drwxr-xr-x  7 sergibli  staff  224 Jun 15  2021 ..
drwxr-xr-x  7 sergibli  staff  224 Feb  4 12:16 1.55-x86_64-apple-darwin
drwxr-xr-x  7 sergibli  staff  224 Feb  1 07:29 1.56.0-x86_64-apple-darwin
lrwxr-xr-x  1 sergibli  staff   88 Apr 24 14:50 mos -> /Users/sergibli/Development/3rd_party/llvm-mos/rust-mos/build/x86_64-apple-darwin/stage1
drwxr-xr-x  6 sergibli  staff  192 Dec 24  2020 nightly-2020-07-27-x86_64-apple-darwin
drwxr-xr-x  7 sergibli  staff  224 Sep 22  2021 nightly-2021-06-15-x86_64-apple-darwin
drwxr-xr-x  7 sergibli  staff  224 Apr 20 21:09 nightly-x86_64-apple-darwin
drwxr-xr-x  7 sergibli  staff  224 Apr  8 18:26 stable-x86_64-apple-darwin

The target triple is also listed:

▶ rustc --print target-list
aarch64-apple-darwin
...
mipsel-sony-psp
mipsisa64r6el-unknown-linux-gnuabi64
mos-unknown-none
msp430-none-elf
nvptx64-nvidia-cuda
...

Have you seen anything like the errors I am encountering before? Any educated guesses on where I might want to dig?

Thanks so much for your efforts with this crate!

All the best, U007D

U007D commented 2 years ago

From @mrk-its:

I followed the installation instructions and am getting a failure on the very last line. (So close!)

Indeed, you are really close. This latest step creates separate sub-targets for each supported platform. It does by modifying a bit target spec json for 'mos-unknown-none' - that's why it needs to be retrieved with rustc --print target-spec-json

Examining create_mos_targets.py shows that l.15 shells out to rustup. Running that line directly gives:

▶ rustup run mos rustc --target mos-unknown-none -Z unstable-options --print target-spec-json
rustc -Cllvm-args="..." with: Unknown command line argument '--align-large-globals=false'.  Try: 'rustc -Cllvm-args="..." with --help'
rustc -Cllvm-args="..." with: Did you mean '--asan-opt-globals=false'?

It is really strange, it looks like for some reason it verifies returned json and it doesn't like llvm-args there: { ... "llvm-args": [ "--force-precise-rotation-cost", "--jump-inst-cost=6", "--force-loop-cold-block", "--phi-node-folding-threshold=0", "--two-entry-phi-node-folding-threshold=0", "--align-large-globals=false", "--disable-spill-hoist" ], ... }

It is possible that rust-mos stopped working with the latest version of llvm-mos (I see upstream llvm changes were merged recently). I'll try to rebuild it and let you know. In the meantime you can remove this --align-large-globals=false from list of the llvm-args here:

https://github.com/mrk-its/rust-mos/blob/master/compiler/rustc_target/src/spec/mos_unknown_none.rs#L20

But I just checked clang still sets this arg:

https://github.com/llvm-mos/llvm-mos/blob/main/clang/lib/Driver/ToolChains/CommonArgs.cpp#L1666

Ok, I'll try to reproduce it locally and let you know

U007D commented 2 years ago

For @mrk-its or others who may read this in the future, @mrk-its' hint of changes in llvm-mos possibly being the culprit prompted me to try the version of llvm-mos that was current when this version of rust-mos commit SHA 65009f shipped (Mar. 30, 2022).

Through some trial and error, building llvm-mos commit SHA 84a2c0a (Mar. 14, 2022) along with llvm-mos-sdk commit SHA cfba7b0 (Mar. 15, 2022) yielded a successful (if not fully current) build. 🎉

(Note no other commit of llvm-mos-sdk I tried worked with llvm-mos@84a2c0a.)

Serentty commented 2 years ago

Even with the most recent commits (more recent than this issue), I am also getting this.

U007D commented 1 year ago

I tried again this evening with clean, latest source code builds of llvm-mos, llvm-mos-sdk and rust-mos, but got the same error in the end. Both before and after removing --align-large-globals=false from mos-unknown-none.rs, I got the following build error:

▶ export RUST_TARGET_PATH=/usr/local/rust-mos-targets/
▶ export TARGET_ARCH=x86_64-apple-darwin #for MacOS
▶ cp config.toml.example config.toml

Note config.toml:683 changed from: llvm-has-rust-patches = if llvm-config { false } else { true } to llvm-has-rust-patches = true #if llvm-config { false } else { true } (more on this below)

▶ ./x.py build -i --stage 0 src/tools/cargo
extracting /Users/Shared/Development/3rd_party/rust-mos/build/cache/2022-09-20/rust-std-beta-x86_64-apple-darwin.tar.xz
extracting /Users/Shared/Development/3rd_party/rust-mos/build/cache/2022-09-20/rustc-beta-x86_64-apple-darwin.tar.xz
extracting /Users/Shared/Development/3rd_party/rust-mos/build/cache/2022-09-20/cargo-beta-x86_64-apple-darwin.tar.xz
Building rustbuild
   Compiling libc v0.2.126
...
   Compiling cargo-credential-1password v0.1.0 (/Users/Shared/Development/3rd_party/rust-mos/src/tools/cargo/crates/credential/cargo-credential-1password)
    Finished release [optimized] target(s) in 9.63s
Build completed successfully in 1:06:27
▶ ./x.py build -i && (
   ln -s ../../stage0-tools-bin/cargo build/${TARGET_ARCH}/stage1/bin/
   cargo
      rustup toolchain link mos build/${TARGET_ARCH}/stage1
      rustup default mos
      mkdir -p $RUST_TARGET_PATH
      python3 create_mos_targets.py $RUST_TARGET_PATH
   )
Building rustbuild
...
Copying stage1 std from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building rustdoc for stage1 (x86_64-apple-darwin)
   Compiling rustdoc v0.0.0 (/Users/Shared/Development/3rd_party/rust-mos/src/librustdoc)
   Compiling rustdoc-tool v0.0.0 (/Users/Shared/Development/3rd_party/rust-mos/src/tools/rustdoc)
    Finished release [optimized] target(s) in 8.99s
Build completed successfully in 0:09:59
Rust's package manager

Usage: cargo [+toolchain] [OPTIONS] [COMMAND]

Options:
  -V, --version             Print version info and exit
      --list                List installed commands
      --explain <CODE>      Run `rustc --explain CODE`
  -v, --verbose...          Use verbose output (-vv very verbose/build.rs output)
  -q, --quiet               Do not print cargo log messages
      --color <WHEN>        Coloring: auto, always, never
      --frozen              Require Cargo.lock and cache are up to date
      --locked              Require Cargo.lock is up to date
      --offline             Run without accessing the network
      --config <KEY=VALUE>  Override a configuration value
  -Z <FLAG>                 Unstable (nightly-only) flags to Cargo, see 'cargo -Z help' for details
  -h, --help                Print help information

Some common cargo commands are (see all commands with --list):
    build, b    Compile the current package
    check, c    Analyze the current package and report errors, but don't build object files
    clean       Remove the target directory
    doc, d      Build this package's and its dependencies' documentation
    new         Create a new cargo package
    init        Create a new cargo package in an existing directory
    add         Add dependencies to a manifest file
    run, r      Run a binary or example of the local package
    test, t     Run the tests
    bench       Run the benchmarks
    update      Update dependencies listed in Cargo.lock
    search      Search registry for crates
    publish     Package and upload this package to the registry
    install     Install a Rust binary. Default location is $HOME/.cargo/bin
    uninstall   Uninstall a Rust binary

See 'cargo help <command>' for more information on a specific command.
info: default toolchain set to 'mos'
Traceback (most recent call last):
  File "/Users/Shared/Development/3rd_party/rust-mos/create_mos_targets.py", line 15, in <module>
    target_spec = json.loads(
  File "/Users/sergibli/.pyenv/versions/3.9.6/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/Users/sergibli/.pyenv/versions/3.9.6/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/sergibli/.pyenv/versions/3.9.6/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm still stuck on this. Any thoughts on how to move forward?

P.S. During the build process, I also ran into an issue with config.toml (from config.toml.example) where l.683 reads:

llvm-has-rust-patches = if llvm-config { false } else { true }

which I don't think is legal TOML? This line gave a build error, so I temporarily hard-coded this to true to proceed.