search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
24 stars 44 forks source link

Reduce dependency on hand-built packages #705

Open AlexanderRichert-NOAA opened 1 year ago

AlexanderRichert-NOAA commented 1 year ago

Is your feature request related to a problem? Please describe. The goal of spack-stack is to avoid the bad old days of manually configured package installations. Across the platforms supported by spack-stack, there are non-spack installations of packages including python, ecflow, openmpi, and mysql, which are relied on by spack-stack packages and/or the applications we support. These outside packages add complexity and risk (e.g., https://github.com/ufs-community/ufs-weather-model/pull/1707#issuecomment-1670487960), and the goal of spack-stack is to have a single, self-consistently built software stack.

Describe the solution you'd like Let's reduce the dependence on external packages as much as possible. MySQL is probably the trickiest of them, and I don't know the full extent of the story behind having separate python installations, but ecflow and openmpi for example should not be difficult to incorporate into the unified environment.

climbfuji commented 1 year ago

I agree with the intention of this issue. But let me note that ecflow is a bit tricky when using Intel. That's because it uses a boost cereal function which for many many years is known to not work with Intel (and apparently nobody cares). This is the only reason we build ecflow as an external package for spack, using the same gcc compiler (and building a static boost library) that gets used as backend for the Intel compiler.

AlexanderRichert-NOAA commented 1 year ago

Noted. I've had some success with building ecflow with intel under spack (though in the distant past I've run into the issue you describe). I'll test it out on various systems when I have time.

climbfuji commented 1 year ago

Noted. I've had some success with building ecflow with intel under spack (though in the distant past I've run into the issue you describe). I'll test it out on various systems when I have time.

Great news! Not sure we can make it part of this release, given the short time we have and all the testing it needs, but even if not then definitely for the next one. That would be hugely helpful to have going forward. Thanks for looking into this.

AlexanderRichert-NOAA commented 1 year ago

Quick update-- I've reproduced what I believe is the same issue with cereal/boost. The problem occurs not in running ecflow_server, but in running any ecflow_client command to communicate with it. I'll dig a little more, but for now the answer might be either to only use boost%gcc, or to not build ecflow with intel, as it doesn't look like the issue will be resolved (in one place, it appeared that it came down to interpretations of code standards, so intel classic may not ever support the relevant feature).

eap commented 1 month ago

Are there any updates on this issue; how should we move forward? It seems there's also tension here with the notion that we can use platform libraries to make builds more rapid and smaller, is there any consensus on the better approach?

climbfuji commented 1 month ago

mysql is no longer an issue, since the new R2D2 doesn't require it (but we need to remove it from the dependency on mysql and mysql-connector from the spack virtual packages/environments). The remaining one is ecflow, which can't be fixed with Intel classic. Once we update all site configs to use intel oneapi (icx, icpx) then this issue can be closed.