search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
21 stars 41 forks source link

Install libbufr_d.so on AWS #1078

Closed weihuang-jedi closed 2 months ago

weihuang-jedi commented 2 months ago

Package name

bufr

Package version/tag

12.0.1

Build options

please have libbufr_4,8,d all three libs.

Installation timeframe

Please install bufr on AWS. gsi_enkf packe need this lib especially, libufr_d.so.

Other information

No response

WCOSS2

WCOSS2: General questions

No response

WCOSS2: Installation and testing

No response

WCOSS2: Technical & security review list

WCOSS2: Additional comments

No response

AlexanderRichert-NOAA commented 2 months ago

As of v12, NCEPLIBS-bufr only builds the _4 library, with no plans to change that, so I think it will require some data type management on the downstream development end. Tagging @jbathegit for any further insights.

natalie-perlin commented 2 months ago

@AlexanderRichert-NOAA - adding gsi-addon-env should take care of installation of these libraries for the global-workflow. Are there any good instructions for that installation? That's what done so far on a newly started AWS CentOS instance:

source /contrib/admin/basic_setup.sh 
cd /contrib/spack-stack/spack-stack-1.6.0/
spack stack create env --site noaa-aws --template gsi-addon-dev --name gsi-addon-env
cd envs/gsi-addon-env/
spack env activate -p .
spack concretize 2>&1 | tee log.spack.concretize.gsi-addon-env.001
 # (nothing to concretize, empty log file)
 spack install --verbose --fail-fast 2>&1 | tee log.spack.install.gsi-addon-env.001

A log file for the install says: ==> /contrib/spack-stack/spack-stack-1.6.0/envs/gsi-addon-env environment has no specs to install

What's missing in the above recipe?..

AlexanderRichert-NOAA commented 2 months ago

Did you add compilers to spack.yaml? I seem to recall the compiler list is empty by default for the gsi-addon-dev template.

natalie-perlin commented 2 months ago

Did you add compilers to spack.yaml? I seem to recall the compiler list is empty by default for the gsi-addon-dev template.

Yes, that must be it! It was empty. On the other hand, it seemed to start installing other packages as well. Does it need to be installed as a chained environments? So something like this needs to be done to create the gsi-addon-env :

spack stack create env --name gsi-addon-env --template empty --site noaa-aws --upstream /contrib/spack-stack/spack-stack-1.6.0/envs/unified-env --upstream /contrib/spack-stack/spack-stack-1.6.0/envs/unified-env/gsi-addon-env
AlexanderRichert-NOAA commented 2 months ago

Yes it does, sorry I missed that-- You want to add --upstream /contrib/spack-stack/spack-stack-1.6.0/envs/unified-env/install to the spack stack create env command, then you should see lots of '[^]' in your concretization output.

jbathegit commented 2 months ago

As of v12, NCEPLIBS-bufr only builds the _4 library, with no plans to change that, so I think it will require some data type management on the downstream development end. Tagging @jbathegit for any further insights.

Alex is correct. As of v12.0.0 there's only a single build of the NCEPLIBS-bufr library, equivalent to what was previously labeled as the _4 build. But that one single build should still be compatible with all application codes, including those that previously linked to separate _d or _8 builds of the library.

natalie-perlin commented 2 months ago

Yes it does, sorry I missed that-- You want to add --upstream /contrib/spack-stack/spack-stack-1.6.0/envs/unified-env/install to the spack stack create env command, then you should see lots of '[^]' in your concretization output.

Thank you, @AlexanderRichert-NOAA , will take this route!!

natalie-perlin commented 2 months ago

@AlexanderRichert-NOAA The following was added to ./spack-stack-1.6.0/envs/gsi-addon-env/spack.yaml file after creating the environment gsi-addon-env:

  definitions:
  # Note: Set 'compilers' manually; must match upstream list
  - compilers: ['%intel']
  - packages:
    - global-workflow-env ^bufr@11.7.0 ^metplus@3.1.1 ^met@9.1.3
    - ufs-weather-model-env
    - gsi-env

  specs:
  - matrix:
    - [$packages]
    - [$compilers]

Adding only "definitions:", "compilers" and "packages" (no "specs" and python packages) did not concretize anything. The concretization went OK after adding 

specs:
  - matrix:
    - [$packages]
    - [$compilers]

It looks like installation has started to build everything, except using cached sources when found:

==> Warning: upstream not found: /contrib/spack-stack/spack-stack-1.6.0/envs/unified-env/.spack-db/index.json
[+] /usr (external gmake-3.82-pj7m7hvhwnm6b6u4p5w6it4ejvjxmygn)
[+] /usr (external pkg-config-0.27.1-lrp5spnn3npv7x5ol6dblkpkerlaj7es)
==> Installing ca-certificates-mozilla-2023-05-30-2325alg2swzcuhoo6q4hya5cqzd76k7j [3/134]
==> No binary for ca-certificates-mozilla-2023-05-30-2325alg2swzcuhoo6q4hya5cqzd76k7j found: installing from source
==> Using cached archive: /tmp/spack-stack/cache/source_cache/_source-cache/archive/5f/5fadcae90aa4ae041150f8e2d26c37d980522cdb49f923fc1e1b5eb8d74e71ad
==> No patches needed for ca-certificates-mozilla
==> ca-certificates-mozilla: Executing phase: 'install'
==> [2024-04-16-19:41:01.965121] Installing cacert-2023-05-30.pem to /contrib/spack-stack/spack-stack-1.6.0/envs/gsi-addon-env/install/intel/2021.3.0/ca-certificates-mozilla-2023-05-30-2325alg/share/cacert.pem
==> ca-certificates-mozilla: Successfully installed ca-certificates-mozilla-2023-05-30-2325alg2swzcuhoo6q4hya5cqzd76k7j
  Stage: 0.02s.  Install: 0.06s.  Post-install: 0.56s.  Total: 0.80s
[+] /contrib/spack-stack/spack-stack-1.6.0/envs/gsi-addon-env/install/intel/2021.3.0/ca-certificates-mozilla-2023-05-30-2325alg
[+] /usr (external perl-5.16.3-uwsyiihj56b43z46p3gi3oev22wrnsuu)
[+] /usr (external diffutils-3.3-ec76vb3x3ezjl3lracr736nw6sofoxol)
[+] /usr (external gettext-0.19.8.1-coxdw3exkqp4g4dimy5cbkuthn3pjlep)
==> intel-oneapi-mpi@2021.3.0 : has external module in ['impi/2021.3.0']
[+] /apps/oneapi (external intel-oneapi-mpi-2021.3.0-4abslp54c6tcnpalrhfrzddgxzkjyjty)
==> Installing crtm-fix-2.4.0.1_emc-s3z7qzellpd4fgh2vt73aed4d6dyilpm [8/134]
==> No binary for crtm-fix-2.4.0.1_emc-s3z7qzellpd4fgh2vt73aed4d6dyilpm found: installing from source
==> Using cached archive: /tmp/spack-stack/cache/source_cache/_source-cache/archive/6e/6e4005b780435c8e280d6bfa23808d8f12609dfd72f77717d046d4795cac0457.tgz
==> No patches needed for crtm-fix
==> crtm-fix: Executing phase: 'install'
....

Is it not the anticipated way for gsi-addon-env?...

AlexanderRichert-NOAA commented 2 months ago

Can you upload your spack.yaml and concretization output?

natalie-perlin commented 2 months ago

@AlexanderRichert-NOAA -sure (attached) spack.yaml_gsi-addon-env.txt

log.spack.concretize.gsi-addon-env.txt

AlexanderRichert-NOAA commented 2 months ago

"/install" needs to be added to the upstream path. The upstream path has to contain .spack-db, which in the case of spack-stack is $SPACK_ENV/install

natalie-perlin commented 2 months ago

"/install" needs to be added to the upstream path. The upstream path has to contain .spack-db, which in the case of spack-stack is $SPACK_ENV/install

... my mistake, added. The entire recipe:

source /contrib/admin/basic_setup.sh 
cd /contrib/spack-stack/spack-stack-1.6.0/
spack stack create env --site noaa-aws --template gsi-addon-dev --name gsi-addon-env  --site noaa-aws --upstream /contrib/spack-stack/spack-stack-1.6.0/envs/unified-env/install --upstream /contrib/spack-stack/spack-stack-1.6.0/envs/unified-env/gsi-addon-env/install
cd envs/gsi-addon-env/
spack env activate -p .
vim spack.yaml                   # added   - compilers: ['%intel']
spack concretize 2>&1 | tee log.spack.concretize.gsi-addon-env.001
spack install --verbose --fail-fast 2>&1 | tee log.spack.install.gsi-addon-env.001

Still appears to be building all the packages, including crtm, zlib, wgrib2, openblas... The binaries are under /contrib/spack-stack/spack-stack-1.6.0/envs/gsi-addon-env/install/intel/2021.3.0/<package>/

Logs from /contrib/spack-stack/spack-stack-1.6.0/envs/gsi-addon-env/: spack.yaml_gsi_addon.txt

log.spack.concretize.gsi-addon-env.001.txt

Is there else to take care?..

AlexanderRichert-NOAA commented 2 months ago

I don't know if it's doing anything bad, but I would get rid of the second --upstream argument there (the one with gsi-addon-env). In spack.yaml_gsi_addon.txt,

    spack-stack-1.6.0-gsi-addon-env:
      install_tree: /contrib/spack-stack/spack-stack-1.6.0/envs/gsi-addon-env/install

doesn't seem to match with what's in the command you showed. You might try deleting the gsi-addon-env directory if you're not already.

natalie-perlin commented 2 months ago

@AlexanderRichert-NOAA - thank you, eventually all worked out with the installation of gsi-addon-env on AWS

natalie-perlin commented 2 months ago

Installed gsi-addon-dev on AWS and GCP for spack-stack 1.6.0 and 1.7.0. Confirmed with Wei Huang that having bufr/11.7.0 with the libbufr_d.so library is sufficient for building global-workflow.

Installing on Azure at the moment.

natalie-perlin commented 2 months ago

Is there a way to explicitly specify the architecture - 'arch' or 'target' - when building spack-stack environments?.. The problem with Azure is that in previous builds the architecture was detected as linux-centos7-broadwell, and all the packages in spack-stack were build/optimized for that. New Azure instances started all result in arch being set to linux-centos7-x86_64_v4. Thus, when gsi-addon-env is created, it installs all the spack-stack-1.6.0 packages in addition to the those for the GSI, because those in unified-env belong to a different architecture.

What I tested so far and which did not help:

climbfuji commented 2 months ago

https://github.com/JCSDA/spack-stack/blob/353ffdf6b9d698770615b006ab2b4de5d20efe81/configs/sites/hera/packages.yaml#L8 ?

I would just rebuild everything, though

natalie-perlin commented 2 months ago

https://github.com/JCSDA/spack-stack/blob/353ffdf6b9d698770615b006ab2b4de5d20efe81/configs/sites/hera/packages.yaml#L8

? I would just rebuild everything, though

Do you suggest rebuilding for Hera? Or Azure? (or both)?

climbfuji commented 2 months ago

No, just azure. The hera link is simply to show you how to set the target

weihuang-jedi commented 2 months ago

On Azure, can we check PW_CSP?

I have:

@.*** global-workflow-cloud]$ echo $PW_CSP azure

On Fri, Apr 19, 2024 at 8:09 AM Dom Heinzeller @.***> wrote:

No, just azure. The hera link is simply to show you how to set the target

— Reply to this email directly, view it on GitHub https://github.com/JCSDA/spack-stack/issues/1078#issuecomment-2066667514, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASMCH66MNCRV5D2QCMGBUIDY6EQP5AVCNFSM6AAAAABGJVRZY6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRWGY3DONJRGQ . You are receiving this because you authored the thread.Message ID: @.***>

natalie-perlin commented 2 months ago

No, just azure. The hera link is simply to show you how to set the target

Oh, thank you!! Yes, that's what I've done, and it did not seem to help, as the architecture would still show as linux-centos7-x86_64_v4. It is also explicitly set in compilers as x86_64.
And yes, I'd most likely rebuild all of these on Azure (unless another easy fix could be found)

natalie-perlin commented 2 months ago

echo $PW_CSP

yes getting the same

weihuang-jedi commented 2 months ago

I global-workflow detect_machine.sh, it has:

if [[ ${MACHINE_ID} == "UNKNOWN" ]]; then case ${PW_CSP:-} in "aws" | "google" | "azure") MACHINE_ID=noaacloud ;; *) PW_CSP="UNKNOWN" esac fi

On Fri, Apr 19, 2024 at 8:14 AM Wei Huang - NOAA Affiliate < @.***> wrote:

On Azure, can we check PW_CSP?

I have:

@.*** global-workflow-cloud]$ echo $PW_CSP azure

On Fri, Apr 19, 2024 at 8:09 AM Dom Heinzeller @.***> wrote:

No, just azure. The hera link is simply to show you how to set the target

— Reply to this email directly, view it on GitHub https://github.com/JCSDA/spack-stack/issues/1078#issuecomment-2066667514, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASMCH66MNCRV5D2QCMGBUIDY6EQP5AVCNFSM6AAAAABGJVRZY6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRWGY3DONJRGQ . You are receiving this because you authored the thread.Message ID: @.***>

natalie-perlin commented 2 months ago

Another test of specifying target=[broadwell] in both packages.yaml and compilers.yaml resulted in this error:

==> Error: concretization failed for the following reasons:

   1. ufs-weather-model-env compiler 'intel@2021.3.0' incompatible with 'target=x86_64_v4'

So I'll be rebuilding the stacks on Azure, for spack-stack-1.6.0 and spack-stack-1.7.0.

natalie-perlin commented 2 months ago

Spack-stacks 1.6.0, 1.7.0, and their corresponding gsi-addon-env environments were successfully rebuilt on NOAA ParallelWorks Azure, /contrib/spack-stack/spack-stack-1.6.0/ /contrib/spack-stack/spack-stack-1.7.0

With that note closing the issue.