search

iarspider / cms-spack-repo

0 stars 2 forks source link

`CMSSW_13_0_X` fails in building `llvm` for `el8_amb64_gcc11` #36

Closed aandvalenzuela closed 1 year ago

aandvalenzuela commented 1 year ago

Hello,

For me and @smuzaffar, CMSSW_13_0_X branch fails in building for el8_amd64_gcc11. I open this issue for the records so that we can track the changes required for building.

Build instructions:

  1. Clone cms-spack-repo and cms-bot repositories in workspace:

    git clone git@github.com:cms-sw/cms-bot.git
git clone -b CMSSW_13_0_X git@github.com:iarspider/cms-spack-repo.git

  2. Set the required env variables:

    export WORKSPACE=$(/bin/pwd -P)
export SCRAM_ARCH=el8_amd64_gcc11
mkdir $WORKSPACE/install
export RPM_INSTALL_PREFIX=$WORKSPACE/install
export SPACK_ENV_NAME=CMSSW_13_0_X
export CMSARCH=el8_amd64_gcc11
export SPACK_DEBUG=1

  3. Run bash -ex $WORKSPACE/cms-spack-repo/bootstrap.sh script.

  4. Check compiler definition at $WORKSPACE/spack/etc/spack/linux/compilers.yaml.

    source $WORKSPACE/spack/share/spack/setup-env.sh
spack compiler find

  5. Start the build by running $WORKSPACE/cms-spack-repo/scripts/singBuild.sh script.

Environment modifications: We have performed the following updates to the environment since it was failing in building:

Build error in llvm: This new failure is in llvm. Error message:

1 error found in build log:
     689    -- Clang version: 12.0.1
     690    -- Performing Test CXX_SUPPORTS_NO_NESTED_ANON_TYPES_FLAG
     691    -- Performing Test CXX_SUPPORTS_NO_NESTED_ANON_TYPES_FLAG - Failed
     692    -- Looking for include file sys/inotify.h
     693    -- Looking for include file sys/inotify.h - found
     694    -- LLD version: 12.0.1
  >> 695    CMake Error at /build/avalenzu/workspace/install/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_place
            holder__/__spack_path_/cmake/3.18.2-dgw477zm5kweayxgxslg6j2mrqghi6n3/share/cmake-3.18/Modules/FindPackageHandleStandardAr
            gs.cmake:165 (message):
     696      Could NOT find LibEdit (missing: LibEdit_INCLUDE_DIRS LibEdit_LIBRARIES)
     697    Call Stack (most recent call first):
     698      /build/avalenzu/workspace/install/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__sp
            ack_path_/cmake/3.18.2-dgw477zm5kweayxgxslg6j2mrqghi6n3/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 
            (_FPHSA_FAILURE_MESSAGE)
     699      /build/avalenzu/workspace/spack/stage/spack-stage-llvm-12.0.1.cms-63ogdwv6p3q24ebtbmh3aqkgdhhyxfmd/spack-src/lldb/cmake
            /modules/FindLibEdit.cmake:54 (find_package_handle_standard_args)
     700      /build/avalenzu/workspace/spack/stage/spack-stage-llvm-12.0.1.cms-63ogdwv6p3q24ebtbmh3aqkgdhhyxfmd/spack-src/lldb/cmake
            /modules/LLDBConfig.cmake:48 (find_package)
     701      /build/avalenzu/workspace/spack/stage/spack-stage-llvm-12.0.1.cms-63ogdwv6p3q24ebtbmh3aqkgdhhyxfmd/spack-src/lldb/cmake
            /modules/LLDBConfig.cmake:55 (add_optional_dependency)

Spack build log at cmsdev22, path /build/avalenzu/workspace/llvm-error.

@iarspider could you confirm that the build instructions are correct? Are we missing anything so that it builds for you but not for us? cleanup-cms-repo is failing with the same xpmem and clhep errors for the time being, but it has not build xrootd or llvm yet.

Sorry for the inconvenience, Andrea.

iarspider commented 1 year ago

Sorry, my bad: forgot to sync environment file. I have updated CMSSW_13_0_X branch, please let me know if it builds now.

aandvalenzuela commented 1 year ago

Thank you @iarspider!

@smuzaffar, we also need to put a fresh tag of cmssw in the recipe, but this is for fetching cmssw.

smuzaffar commented 1 year ago

thanks I will try that

smuzaffar commented 1 year ago

for me CMSSW_13_0_X still fails. please update https://codimd.web.cern.ch/s/QL8ZVMHHQ# with a working set of instructions

[a]

lib/spack/spack/installer.py:1491 ==> [2022-12-20-17:26:00.200465] No build task for hdf5-1.10.6-mswrprp3owoubyaas2uawv6sczih743o to skip since pkg-config-0.29.2-ikpf4vg2i52fy5jwyikreiwvtn75zcpi failed
lib/spack/spack/installer.py:1799 ==> [2022-12-20-17:26:00.200708] Error: Failed to install pkg-config due to NoSuchBucket: An error occurred (NoSuchBucket) when calling the GetObject operation: The spe
cified bucket does not exist
lib/spack/spack/environment/environment.py:1501 ==> [2022-12-20-17:26:05.867744] Skip view update, this environment does not maintain a view
lib/spack/spack/main.py:1010 ==> [2022-12-20-17:26:05.868002] InstallError: Terminating after first install failure: An error occurred (NoSuchBucket) when calling the GetObject operation: The specified bucket does not exist
lib/spack/spack/error.py:54 ==> [2022-12-20-17:26:05.868273] Error: Terminating after first install failure: An error occurred (NoSuchBucket) when calling the GetObject operation: The specified bucket does not exist
Traceback (most recent call last):
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/installer.py", line 1757, in install
smuzaffar commented 1 year ago

note that I used CMSSW_13_0_X branch of iarspider/cms-spack-repo

iarspider commented 1 year ago

for me CMSSW_13_0_X still fails. please update https://codimd.web.cern.ch/s/QL8ZVMHHQ# with a working set of instructions

Hi Shahzad,

yesterday there were some problems with CERN IT infrastructure, could also affect S3. The error you see (NoSuchBucket) usually occurs when S3_ENDPOINT_URL is not set. It is set here, so it should be set when using singBuild.sh (which is a wrapper to run build.sh inside singularity container)

aandvalenzuela commented 1 year ago

Hi all,

Just to confirm, I have built all the packages except cmssw (that is building now) with branch CMSSW_13_0_X. In any case, do we expect cmssw to fail @iarspider?

iarspider commented 1 year ago

Hi all,

Just to confirm, I have built all the packages except cmssw (that is building now) with branch CMSSW_13_0_X. In any case, do we expect cmssw to fail @iarspider?

If it starts building (i.e. Spack can fetch it), it should build fine.

aandvalenzuela commented 1 year ago

For me, it fails in building cmssw with:

Traceback (most recent call last):
  File "/build/avalenzu/workspace/spack/stage/spack-stage-cmssw-CMSSW_13_0_X_2022-12-20-1100-2eapzj3txrtbs2inrrdlv4adv3a2bnx2/CMSSW_13_0_X_2022-12-20-1100/src/CondFormats/Serialization/python/condformats_serialization_generate.py", line 31, in <module>
    import clang.cindex
ModuleNotFoundError: No module named 'clang'
gmake: *** [tmp/el8_amd64_gcc11/src/CondFormats/HIObjects/src/CondFormatsHIObjects/a/Serialization.cc] Error 1

I am using:

    version("CMSSW_13_0_X_2022-12-20-1100", tag="CMSSW_13_0_X_2022-12-20-1100")

Any hint on how clang should be imported here?

smuzaffar commented 1 year ago

@iarspider , how can I build without using binary caches?

iarspider commented 1 year ago

@iarspider , how can I build without using binary caches?

@smuzaffar spack install --no-cache

For me, it fails in building cmssw with:

Traceback (most recent call last):
  File "/build/avalenzu/workspace/spack/stage/spack-stage-cmssw-CMSSW_13_0_X_2022-12-20-1100-2eapzj3txrtbs2inrrdlv4adv3a2bnx2/CMSSW_13_0_X_2022-12-20-1100/src/CondFormats/Serialization/python/condformats_serialization_generate.py", line 31, in <module>
    import clang.cindex
ModuleNotFoundError: No module named 'clang'
gmake: *** [tmp/el8_amd64_gcc11/src/CondFormats/HIObjects/src/CondFormatsHIObjects/a/Serialization.cc] Error 1

I am using:

    version("CMSSW_13_0_X_2022-12-20-1100", tag="CMSSW_13_0_X_2022-12-20-1100")

Any hint on how clang should be imported here?

@aandvalenzuela Could you please test with this commit?

iarspider commented 1 year ago

This change will rebuild llvm and it's dependents, so will take some time to get to CMSSW. There is probably a faster way to test it, maybe by creating a dummy cmssw-like SCRAM-based package that can just import clang - @smuzaffar can probably help with that.

iarspider commented 1 year ago

@smuzaffar spack install --no-cache will disable binary cache, but will still try to download sources from S3. If you don't want to use sources from S3, you need to do spack mirror --scope=site rm cms-s3. There shouldn't be any buildcache for CMSSW_13_0_X at the moment - if @aandvalenzuela didn't upload something.

smuzaffar commented 1 year ago

ok thanks @iarspider , I am tetsing --no-cache now. llvm python dir should be in PYHONPATH so that import clang works. looks like https://github.com/iarspider/cms-spack-repo/commit/b15508ade1ddb3823962e3cd6deb6e9523032ba0 should do it

aandvalenzuela commented 1 year ago

I confirm I have not uploaded anything for CMSSW_13_0_X

smuzaffar commented 1 year ago

by the way, llvm in cmsdist adds %{i}/lib64/python%{cms_python3_major_minor_version}/site-packages ( https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_13_0_X/master/llvm.spec#L3 ) , @iarspider are you sure the change https://github.com/iarspider/cms-spack-repo/commit/b15508ade1ddb3823962e3cd6deb6e9523032ba0 is correct?

iarspider commented 1 year ago

@smuzaffar I think just PYTHONPATH is actually not needed now - only PYTHON3PATH (since scram resets PYTHONPATH). But now that you ask - it may need one more fix (equivalent of this - creating .egg file), but I'm not 100% sure this is necessary (as eggs are outdated and aren't used anymore). I am running a local build now without creating egg, let's see how it goes. Is there a simple way to test import clang without building too many packages (i.e. a full set of CMSSW dependencies) .

aandvalenzuela commented 1 year ago

@iarspider https://github.com/iarspider/cms-spack-repo/commit/b15508ade1ddb3823962e3cd6deb6e9523032ba0 fails with the same ModuleNotFoundError error.

iarspider commented 1 year ago

@aandvalenzuela can you try manually creating PKG-INFO as done here, and then retry?

aandvalenzuela commented 1 year ago

@iarspider Should it be a post-install method as in https://github.com/iarspider/cms-spack-repo/blob/CMSSW_12_6_X/repos/cms/packages/llvm/package.py#L205-L217?

iarspider commented 1 year ago

@iarspider Should it be a post-install method as in https://github.com/iarspider/cms-spack-repo/blob/CMSSW_12_6_X/repos/cms/packages/llvm/package.py#L205-L217?

Just create that file by hand. It it works - yes, it should go into post-install method. Also, check that llvm python bindings are installed under lib64/python..., not under lib - I think there were problems with that.

smuzaffar commented 1 year ago

@iarspider , CMSSW_13_0_X env failed again with error

lib/spack/spack/installer.py:1930 ==> [2022-12-22-05:45:40.420398] rust: Building rust-1.57.0-ozbws4kin7hjtcyw5ssnyjk2a25clvdw [Package]
lib/spack/spack/installer.py:2042 ==> [2022-12-22-05:45:40.436947] rust: Executing phase: 'configure'
lib/spack/spack/installer.py:2042 ==> [2022-12-22-05:46:04.445861] rust: Executing phase: 'build'
lib/spack/spack/installer.py:2042 ==> [2022-12-22-06:02:53.619948] rust: Executing phase: 'install'
lib/spack/spack/error.py:54 ==> [2022-12-22-06:09:26.668421] Error: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 546551: ordinal not in range(128)

/data/cmsbld/pgo/spk/spack/lib/spack/spack/build_environment.py:1086, in _setup_pkg_and_run:
       1083        tb_string = traceback.format_exc()
       1084
       1085        # build up some context from the offending package so we can
  >>   1086        # show that, too.
       1087        package_context = get_package_context(tb)
       1088
       1089        logfile = None

See build log for details:
  /data/cmsbld/pgo/spk/spack/stage/spack-stage-rust-1.57.0-ozbws4kin7hjtcyw5ssnyjk2a25clvdw/spack-build-out.txt

Traceback (most recent call last):
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/build_environment.py", line 1070, in _setup_pkg_and_run
    return_value = function(pkg, kwargs)
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/installer.py", line 2088, in build_process
    return installer.run()
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/installer.py", line 1950, in run
    self._real_install()
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/installer.py", line 2064, in _real_install
    combine_phase_logs(pkg.phase_log_files, pkg.log_path)
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/installer.py", line 469, in combine_phase_logs
    log_file.write(phase_log.read())
  File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 546551: ordinal not in range(128)
smuzaffar commented 1 year ago

and rust build logs contains

lib/spack/spack/installer.py:2042 ==> [2022-12-22-05:45:40.436947] rust: Executing phase: 'configure'
lib/spack/spack/util/executable.py:198 ==> [2022-12-22-05:45:40.460840] './spack_bootstrap_stage/rust-1.57.0-x86_64-unknown-linux-gnu/install.sh' '--prefix=/data/cmsbld/pgo/spk/spack/stage/spack-stage-rust-1.57.0-ozbws4kin7hjtcyw5ssnyjk2a25clvdw/spack-src/spack_bootstrap'
install: creating uninstall script at /data/cmsbld/pgo/spk/spack/stage/spack-stage-rust-1.57.0-ozbws4kin7hjtcyw5ssnyjk2a25clvdw/spack-src/spack_bootstrap/lib/rustlib/uninstall.sh
install: installing component 'rustc'
install: installing component 'rust-std-x86_64-unknown-linux-gnu'
install: installing component 'rust-docs'
install: installing component 'rust-demangler-preview'
install: installing component 'cargo'
install: installing component 'rustfmt-preview'
install: installing component 'rls-preview'
install: installing component 'llvm-tools-preview'
install: installing component 'clippy-preview'
install: installing component 'rust-analysis-x86_64-unknown-linux-gnu'
install: WARNING: failed to run ldconfig. this may happen when not installing as root. run with --verbose to see the error

    rust installed.
aandvalenzuela commented 1 year ago

@iarspider Should it be a post-install method as in https://github.com/iarspider/cms-spack-repo/blob/CMSSW_12_6_X/repos/cms/packages/llvm/package.py#L205-L217?

Just create that file by hand. It it works - yes, it should go into post-install method. Also, check that llvm python bindings are installed under lib64/python..., not under lib - I think there were problems with that.

In this case it fails with the same error

iarspider commented 1 year ago

Looks like a quick fix didn't work. I will look into it later today, probably need to apply more changes to LLVM recipe.

iarspider commented 1 year ago

@iarspider , CMSSW_13_0_X env failed again with error

lib/spack/spack/installer.py:1930 ==> [2022-12-22-05:45:40.420398] rust: Building rust-1.57.0-ozbws4kin7hjtcyw5ssnyjk2a25clvdw [Package]
lib/spack/spack/installer.py:2042 ==> [2022-12-22-05:45:40.436947] rust: Executing phase: 'configure'
lib/spack/spack/installer.py:2042 ==> [2022-12-22-05:46:04.445861] rust: Executing phase: 'build'
lib/spack/spack/installer.py:2042 ==> [2022-12-22-06:02:53.619948] rust: Executing phase: 'install'
lib/spack/spack/error.py:54 ==> [2022-12-22-06:09:26.668421] Error: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 546551: ordinal not in range(128)

/data/cmsbld/pgo/spk/spack/lib/spack/spack/build_environment.py:1086, in _setup_pkg_and_run:
       1083        tb_string = traceback.format_exc()
       1084
       1085        # build up some context from the offending package so we can
  >>   1086        # show that, too.
       1087        package_context = get_package_context(tb)
       1088
       1089        logfile = None

See build log for details:
  /data/cmsbld/pgo/spk/spack/stage/spack-stage-rust-1.57.0-ozbws4kin7hjtcyw5ssnyjk2a25clvdw/spack-build-out.txt

Traceback (most recent call last):
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/build_environment.py", line 1070, in _setup_pkg_and_run
    return_value = function(pkg, kwargs)
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/installer.py", line 2088, in build_process
    return installer.run()
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/installer.py", line 1950, in run
    self._real_install()
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/installer.py", line 2064, in _real_install
    combine_phase_logs(pkg.phase_log_files, pkg.log_path)
  File "/data/cmsbld/pgo/spk/spack/lib/spack/spack/installer.py", line 469, in combine_phase_logs
    log_file.write(phase_log.read())
  File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 546551: ordinal not in range(128)

Haven't seen this one a long time. Will investigate, thanks!

aandvalenzuela commented 1 year ago

Hello @iarspider!

Looks like a quick fix didn't work. I will look into it later today, probably need to apply more changes to LLVM recipe.

Any update on this fix?

iarspider commented 1 year ago

@aandvalenzuela working on it

aandvalenzuela commented 1 year ago

Just for the records, cmssw built yesterday with the latest commit acaac7b717123c0546bd609eda79b4378010a8cb. I am starting a fresh new build from scratch so that we can confirm and get the proper build times.

iarspider commented 1 year ago

@smuzaffar regarding rust failure: I can't reproduce it. While I have seen these in the past, I think they are mostly "solved" by setting correct locale (C, instead of UTF8), something that SCRAM (I think) also does.

smuzaffar commented 1 year ago

yes @iarspider , @aandvalenzuela and myself figured it out last week :-) unsetting [a] fixed the rust install issue. But still I can not build cmssw with spack compiler wrappers

unset LC_ALL
unset LC_CTYPE
iarspider commented 1 year ago

I will now close this issue, since llvm now builds fine. We can open another for building CMSSW with compiler wrappers.