search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
21 stars 41 forks source link

Update NewSiteConfigs.rst for macOS: exclude homebrew gettext #1133

Closed climbfuji closed 1 week ago

climbfuji commented 3 weeks ago

Summary

gettext builds easily on macOS, and finding it without specifying the path means that many homebrew packages find their way into the default built environment (because /usr/local/bin etc. get added to the build environment whenever gettext is needed).

Testing

Locally on Dom's Intel macOS.

Applications affected

n/a

Systems affected

macOS user systems

Dependencies

n/a

Issue(s) addressed

n/a

Checklist

srherbener commented 2 weeks ago

I tried this branch on my Mac, and I'm getting the following errors when trying to build the musl package (which I think is a dependence of gettext:

/Users/steveherbener/spack-stack/spack/lib/spack/env/clang/clang -std=c99 -nostdinc -ffreestanding -fexcess-precision=standard -frounding-math -fno-strict-aliasing -Wa,--noexecstack -D_XOPEN_SOURCE=700 -I./arch/arm -I./arch/generic -Iobj/src/internal -I./src/include -I./src/internal -Iobj/include -I./include  -Os -pipe -fomit-frame-pointer -fno-unwind-tables -fno-asynchronous-unwind-tables -ffunction-sections -fdata-sections -w -Wno-pointer-to-int-cast -Werror=implicit-function-declaration -Werror=implicit-int -Werror=pointer-sign -Werror=pointer-arith -Werror=int-conversion -Werror=incompatible-pointer-types -Werror=discarded-qualifiers -Werror=discarded-array-qualifiers -Qunused-arguments -Waddress -Warray-bounds -Wchar-subscripts -Wduplicate-decl-specifier -Winit-self -Wreturn-type -Wsequence-point -Wstrict-aliasing -Wunused-function -Wunused-label -Wunused-variable -DBROKEN_VFP_ASM  -fPIC -c -o obj/src/process/execl.lo src/process/execl.c
crt/arm/crti.s:1:1: error: unknown directive
.syntax unified
^
crt/arm/crti.s:3:15: error: unexpected token in '.section' directive
.section .init
              ^
crt/arm/crti.s:5:1: error: unknown directive
.type _init,%function
...

I'm using Sonoma and apple-clang@15.0.0.

Any ideas?

climbfuji commented 2 weeks ago

I tried this branch on my Mac, and I'm getting the following errors when trying to build the musl package (which I think is a dependence of gettext:

/Users/steveherbener/spack-stack/spack/lib/spack/env/clang/clang -std=c99 -nostdinc -ffreestanding -fexcess-precision=standard -frounding-math -fno-strict-aliasing -Wa,--noexecstack -D_XOPEN_SOURCE=700 -I./arch/arm -I./arch/generic -Iobj/src/internal -I./src/include -I./src/internal -Iobj/include -I./include  -Os -pipe -fomit-frame-pointer -fno-unwind-tables -fno-asynchronous-unwind-tables -ffunction-sections -fdata-sections -w -Wno-pointer-to-int-cast -Werror=implicit-function-declaration -Werror=implicit-int -Werror=pointer-sign -Werror=pointer-arith -Werror=int-conversion -Werror=incompatible-pointer-types -Werror=discarded-qualifiers -Werror=discarded-array-qualifiers -Qunused-arguments -Waddress -Warray-bounds -Wchar-subscripts -Wduplicate-decl-specifier -Winit-self -Wreturn-type -Wsequence-point -Wstrict-aliasing -Wunused-function -Wunused-label -Wunused-variable -DBROKEN_VFP_ASM  -fPIC -c -o obj/src/process/execl.lo src/process/execl.c
crt/arm/crti.s:1:1: error: unknown directive
.syntax unified
^
crt/arm/crti.s:3:15: error: unexpected token in '.section' directive
.section .init
              ^
crt/arm/crti.s:5:1: error: unknown directive
.type _init,%function
...

I'm using Sonoma and apple-clang@15.0.0.

Any ideas?

that's unrelated to gettext - at least I think so. If you see the musl error, then that's because the old provider for iconv (libc) is no longer an option and it wants to build musl. You can work around this by using an external libiconv as per the documentation.

climbfuji commented 2 weeks ago

I tried this branch on my Mac, and I'm getting the following errors when trying to build the musl package (which I think is a dependence of gettext:

/Users/steveherbener/spack-stack/spack/lib/spack/env/clang/clang -std=c99 -nostdinc -ffreestanding -fexcess-precision=standard -frounding-math -fno-strict-aliasing -Wa,--noexecstack -D_XOPEN_SOURCE=700 -I./arch/arm -I./arch/generic -Iobj/src/internal -I./src/include -I./src/internal -Iobj/include -I./include  -Os -pipe -fomit-frame-pointer -fno-unwind-tables -fno-asynchronous-unwind-tables -ffunction-sections -fdata-sections -w -Wno-pointer-to-int-cast -Werror=implicit-function-declaration -Werror=implicit-int -Werror=pointer-sign -Werror=pointer-arith -Werror=int-conversion -Werror=incompatible-pointer-types -Werror=discarded-qualifiers -Werror=discarded-array-qualifiers -Qunused-arguments -Waddress -Warray-bounds -Wchar-subscripts -Wduplicate-decl-specifier -Winit-self -Wreturn-type -Wsequence-point -Wstrict-aliasing -Wunused-function -Wunused-label -Wunused-variable -DBROKEN_VFP_ASM  -fPIC -c -o obj/src/process/execl.lo src/process/execl.c
crt/arm/crti.s:1:1: error: unknown directive
.syntax unified
^
crt/arm/crti.s:3:15: error: unexpected token in '.section' directive
.section .init
              ^
crt/arm/crti.s:5:1: error: unknown directive
.type _init,%function
...

I'm using Sonoma and apple-clang@15.0.0. Any ideas?

that's unrelated to gettext - at least I think so. If you see the musl error, then that's because the old provider for iconv (libc) is no longer an option and it wants to build musl. You can work around this by using an external libiconv as per the documentation.

So I think it should be failing for you on spack-stack develop without the gettext change here

srherbener commented 2 weeks ago

I've got this in my site/packages.yaml file:

  libiconv:
    buildable: false

That forces spack to use the external libiconv, correct?

Also, I'm seeing this from spack spec gettext:

Input spec
--------------------------------
 -   gettext

Concretized
--------------------------------
 -   gettext@0.21.1%apple-clang@15.0.0 ldflags=-Wl,-ld_classic +bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=darwin-sonoma-m1
 -       ^bzip2@1.0.8%apple-clang@15.0.0 ldflags=-Wl,-ld_classic ~debug~pic+shared build_system=generic arch=darwin-sonoma-m1
 -           ^diffutils@3.10%apple-clang@15.0.0 ldflags=-Wl,-ld_classic  build_system=autotools arch=darwin-sonoma-m1
[e]      ^gmake@3.81%apple-clang@15.0.0 ldflags=-Wl,-ld_classic ~guile build_system=generic patches=ca60bd9 arch=darwin-sonoma-m1
[+]      ^gnuconfig@2022-09-17%apple-clang@15.0.0 ldflags=-Wl,-ld_classic  build_system=generic arch=darwin-sonoma-m1
 -       ^libxml2@2.10.3%apple-clang@15.0.0 ldflags=-Wl,-ld_classic +pic~python+shared build_system=autotools arch=darwin-sonoma-m1
[+]          ^pkg-config@0.29.2%apple-clang@15.0.0 ldflags=-Wl,-ld_classic +internal_glib build_system=autotools arch=darwin-sonoma-m1
[+]          ^zlib-ng@2.1.6%apple-clang@15.0.0 ldflags=-Wl,-ld_classic +compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sonoma-m1
 -       ^musl@1.2.4%apple-clang@15.0.0 ldflags=-Wl,-ld_classic  build_system=makefile arch=darwin-sonoma-m1
[+]      ^ncurses@6.5%apple-clang@15.0.0 ldflags=-Wl,-ld_classic ~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sonoma-m1
 -       ^tar@1.34%apple-clang@15.0.0 ldflags=-Wl,-ld_classic  build_system=autotools zip=pigz arch=darwin-sonoma-m1
[+]          ^pigz@2.8%apple-clang@15.0.0 ldflags=-Wl,-ld_classic  build_system=makefile arch=darwin-sonoma-m1
[+]          ^zstd@1.5.2%apple-clang@15.0.0 ldflags=-Wl,-ld_classic +pic+programs build_system=makefile compression=none libs=shared,static arch=darwin-sonoma-m1
[+]      ^xz@5.4.6%apple-clang@15.0.0 ldflags=-Wl,-ld_classic ~pic build_system=autotools libs=shared,static arch=darwin-sonoma-m1

and this in my log.install file:

123404 ==> Removing failure mark on musl-1.2.4-xuz7bu3iubnim5ndvwk6a3wdc55as66l
123405 ==> Removing failure mark on diffutils-3.10-tmqgjchyji5jhvmnl2gh6cmuwji7c4d6
123406 ==> Removing failure mark on bzip2-1.0.8-zvt4lxpgqhyhityo4iyd3t62q5nqalif
123407 ==> Removing failure mark on tar-1.34-bmsvn4nlii3lma42432br3ny7gzncurl
123408 ==> Removing failure mark on gettext-0.21.1-cxfyxqvaqkqkvpzonfxuhtv2uondoy54

Do these say that it was attempting to build gettext, and that musl is a dependency and it was building musl when the error occurred. These are the first lines of the "Removing failure mark..." messages at the end of the log.install file.

Or do I have something configured wrong with gettext?

Thanks for your help with this!

climbfuji commented 2 weeks ago

Two options: 1. explicitly set the provider for iconv to libiconv, or 2. find gettext external using an explicit call

spack external find gettext --path /path/to/gettext

not sure if path comes before the positional argument. Reason to not find it in the catchall call is that on intel macs, this adds /usr/local/bin etc to the paths with many more packages that we don't want. We need to either build gettext with an external libiconv as the iconv provider, or find the external gettext in the /usr/local/opt/gettext/... directory (or wherever brew installs the package).

srherbener commented 2 weeks ago

For the first option, do I specify the provider in the spack.yaml (where mpi is specified) or the site/packages.yaml file? Thanks

climbfuji commented 2 weeks ago

Wherever you want, can also be in common/packages.yaml

From: Stephen Herbener @.> Sent: Friday, June 14, 2024 3:06 PM To: JCSDA/spack-stack @.> Cc: Dom Heinzeller @.>; Assign @.> Subject: Re: [JCSDA/spack-stack] Update NewSiteConfigs.rst for macOS: exclude homebrew gettext (PR #1133)

For the first option, do I specify the provider in the spack.yaml (where mpi is specified) or the site/packages.yaml file? Thanks

— Reply to this email directly, view it on GitHubhttps://github.com/JCSDA/spack-stack/pull/1133#issuecomment-2168762366, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB5C2ROM3IFRM7PJJGBMJL3ZHNLNFAVCNFSM6AAAAABI7MXE4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRYG43DEMZWGY. You are receiving this because you were assigned.Message ID: @.***>

climbfuji commented 1 week ago

@srherbener Were you able to install spack-stack with this update?

srherbener commented 1 week ago

@climbfuji I added the provider spec in my spack.yaml file:

...
  packages:
    ewok-env:
      variants: +mysql
    all:
      compiler: [apple-clang@15.0.0]
      providers:
        mpi: [openmpi@5.0.3]
        iconv: [libiconv]

and re-concretized, and I'm still getting the same build failure with musl. However, it wasn't trying to build gettext yet. In this case spack was building libxml2.

I suspect that you are correct and the trouble I'm seeing with musl is not related to the gettext change. I'm inclined to approve the PR and fix the musl build later. spack-stack-1.7.0 is working on Sonoma so it should be okay to merge the PR. Do you agree?

In your macos testing, did you successfully build musl with spack-stack?

climbfuji commented 1 week ago

musl doesn't build on macos, at least not for me. I was able to force using libiconv as the iconv provider and have spack use the external libiconv for that.

This PR is unrelated to the problem you are seeing; therefore I suggest we merge it. Thanks.

srherbener commented 1 week ago

musl doesn't build on macos, at least not for me. I was able to force using libiconv as the iconv provider and have spack use the external libiconv for that.

This PR is unrelated to the problem you are seeing; therefore I suggest we merge it. Thanks.

So why is my Mac trying to build musl? Do I need to specify a provider for musl as well? Is musl coming from the libiconv, or is there another external library?

climbfuji commented 1 week ago

$ spack providers iconv iconv: glibc libiconv musl

srherbener commented 1 week ago

Thanks!