search

flux-framework / flux-sched

Fluxion Graph-based Scheduler
GNU Lesser General Public License v3.0
84 stars 39 forks source link

flux-run is not available when installing with Spack on RHEL and gcc@12.2.0 #1036

Closed stefangary closed 11 months ago

stefangary commented 11 months ago

I attempted the follow the instructions to install Flux with Spack v19.0 on an RHEL7.9 system. After I complete this process using this install script (summary below), I can spack load flux-sched and access what I understand to be flux-core tools (e.g. flux-start and flux-exec) but I cannot access higher level flux-run and flux-alloc. Also, flux --help only shows the docs available to flux-core, not the docs available to flux-sched. I've repeated these steps on a different RHEL8.6 system with the same result as well as using different OpenMPI versions.

Here is an explicit example where the flux-core tools appear to work but not the flux-sched tools:

$ spack load flux-sched
$ srun -N2 -n4 --pty -p small flux start
bash: /yum: No such file or directory
$ flux resource status
    STATUS NNODES NODELIST
     avail      4 sfgary-cloud2-00043-2-[0003,0003-0004,0004]
$ flux exec flux getattr rank
0
1
3
2
$ flux run -n4 --label-io hostname
flux: `run' is not a flux command.  See 'flux --help'

The install script basically just runs:

spack install gcc@12.2.0
spack load gcc@12.2.0
spack compiler find
spack unload
spack install flux-sched%gcc@12.2.0

Ultimately, what I really want to do is debug a PMIX initialization error I'm getting when using Flux with Parsl. However, since I can't use flux run, I can't try manual run commands with flux run -overbose=2 as suggested in this issue.

I'm new to Flux and deeply appreciate any insights you may have. Thank you.

grondo commented 11 months ago

My guess is you are using an older version of flux-core. Try flux version. flux run was added in 0.48.0.

The latst flux-core in spack is 0.51.0

https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/flux-core/package.py#L23

If you can't update to the latest for some reason, try flux mini run, which was the previous name of this command.

garlick commented 11 months ago

Ultimately, what I really want to do is debug a PMIX initialization error I'm getting when using Flux with Parsl

Once your spack issues are worked through please do open a new issue on whatever you're running into with openmpi and flux. Note that flux-pmix is also required in order for flux run to offer pmix support to openmpi applications. Integration has improved with more recent versions of flux-core so getting to the latest flux-core will be a better starting point for that work.

In the past, and I think to date in our production systems, we've configured openmpi so that it can work without pmix. I mention that just to indicate that flux-pmix has had limited testing and you may find yourself doing some of that testing for us :-)

stefangary commented 11 months ago

@grondo and @garlick - Thank you very much for the fast and helpful responses and tip about pmix. I am happy to contribute to testing; I'm working on mix of on-premise and cloud-based clusters. I will post a separate issue if I need more help.

spack install flux-core@0.51.0 indeed solved the issue with access to flux-run. I found a small missing dependency issue only when I attempted to use flux run --help which was solved by installing py-ply:

$ flux run --help
Traceback (most recent call last):
  File "/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/flux-core-0.51.0-2puq2kxgzeddwlmmr3asvyfuhxzbrdq5/libexec/flux/cmd/flux-run.py", line 15, in <module>
    import flux.cli.run as base
  File "/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/flux-core-0.51.0-2puq2kxgzeddwlmmr3asvyfuhxzbrdq5/lib/flux/python3.1/flux/cli/run.py", line 15, in <module>
    from flux.cli import base
  File "/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/flux-core-0.51.0-2puq2kxgzeddwlmmr3asvyfuhxzbrdq5/lib/flux/python3.1/flux/cli/base.py", line 39, in <module>
    from flux.constraint.parser import ConstraintParser, ConstraintSyntaxError
  File "/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/flux-core-0.51.0-2puq2kxgzeddwlmmr3asvyfuhxzbrdq5/lib/flux/python3.1/flux/constraint/parser.py", line 15, in <module>
    import ply.yacc as yacc
ModuleNotFoundError: No module named 'ply'

$ spack install py-ply
...
$ flux run --help
usage: flux run [OPTIONS...] COMMAND [ARGS...]
...
grondo commented 11 months ago

Oh, good catch, thanks!

stefangary commented 11 months ago

Sure thing! As I keep working, I find this missing dependency may apply to many different flux run invocations (not just --help).

grondo commented 11 months ago

Yes, it is a requirement for all the submission commands (they all import ply) PR opened for spack https://github.com/spack/spack/pull/38406

grondo commented 11 months ago

Hm, it appears that py-ply is already a dependency in the spack package. I do not know why it isn't being pulled in for your build

    depends_on("py-ply", type=("build", "run"), when="@0.46.1:")

I'm not sure I know enough about spack to debug this.. maybe @vsoch has some ideas?

vsoch commented 11 months ago

@stefangary can you look in your spack install directory and the environment variable for PYTHONPATH and show us what you see? I'm wondering if it's there somehow (installed by spack) but it's not properly being added to the python path. If you are starting a flux instance it would also be useful to see the paths before / after, since flux also does some changing.

stefangary commented 11 months ago

Thank you @grondo and @vsoch. Starting from scratch, my $PYTHONPATH is empty before spack load flux-sched and after that command it is:

/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyyaml-6.0-aajq667cfzsoxueyiwykcoxuy6fy2fty/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyrsistent-0.18.1-23uu7fl2xvaxuugxmcposnekm24ut25a/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-attrs-22.1.0-pfvxw6i2x4hccm2r2hjk2vytwgpch7te/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-jsonschema-4.16.0-ptg26rwkjnuekqp3pwqyiptydslrfhps/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pycparser-2.21-w2yyuzczzvywwtq3wpzdu6ygearrhqg6/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-cffi-1.15.0-v7zecbw75l3bwiso7256sloltz6hsvpv/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/flux-core-0.51.0-2puq2kxgzeddwlmmr3asvyfuhxzbrdq5/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyyaml-6.0-aajq667cfzsoxueyiwykcoxuy6fy2fty/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyrsistent-0.18.1-23uu7fl2xvaxuugxmcposnekm24ut25a/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-attrs-22.1.0-pfvxw6i2x4hccm2r2hjk2vytwgpch7te/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-jsonschema-4.16.0-ptg26rwkjnuekqp3pwqyiptydslrfhps/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pycparser-2.21-w2yyuzczzvywwtq3wpzdu6ygearrhqg6/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-cffi-1.15.0-v7zecbw75l3bwiso7256sloltz6hsvpv/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyyaml-6.0-aajq667cfzsoxueyiwykcoxuy6fy2fty/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-65.5.0-wfbdr55dd6b3j6baawtbexfvb2af5ddk/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-cython-0.29.32-yrtzbi75otr3sk2nec2cc5fu2obclc7l/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyrsistent-0.18.1-23uu7fl2xvaxuugxmcposnekm24ut25a/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-attrs-22.1.0-pfvxw6i2x4hccm2r2hjk2vytwgpch7te/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-jsonschema-4.16.0-ptg26rwkjnuekqp3pwqyiptydslrfhps/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyrsistent-0.18.1-23uu7fl2xvaxuugxmcposnekm24ut25a/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-typing-extensions-4.3.0-i33eziavtwtpcgg2inj7ehi63tl4vnmy/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-65.5.0-wfbdr55dd6b3j6baawtbexfvb2af5ddk/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-scm-7.0.5-ru7uxtzsslvf7l3hrt3mefx4g3l2oua2/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-tomli-2.0.1-aa3brbrja2fcgmut6sm7sbfbs6oyjxni/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pluggy-1.0.0-ekrvl5rqdjiiagq5tbsttbo6rqwrg57z/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pathspec-0.10.1-emo23i62swivzygxccheg42ooswg2pvu/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyparsing-3.0.9-iunsddp3j76dymrjejlpo4mkmy5rlc5r/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-packaging-21.3-h77ffuya56shukyoxwtx6qvsc6zdobis/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-editables-0.3-j6ye2rcrkw2gjkststt4k2ih42ujdjf7/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatchling-1.10.0-nfnrepoazwcn6226lgmyebt6qlkbs5qf/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatch-vcs-0.2.0-ggwougvgzeld7owlpmteyjslcn65t42w/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-tomli-2.0.1-aa3brbrja2fcgmut6sm7sbfbs6oyjxni/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pluggy-1.0.0-ekrvl5rqdjiiagq5tbsttbo6rqwrg57z/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pathspec-0.10.1-emo23i62swivzygxccheg42ooswg2pvu/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyparsing-3.0.9-iunsddp3j76dymrjejlpo4mkmy5rlc5r/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-packaging-21.3-h77ffuya56shukyoxwtx6qvsc6zdobis/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-editables-0.3-j6ye2rcrkw2gjkststt4k2ih42ujdjf7/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatchling-1.10.0-nfnrepoazwcn6226lgmyebt6qlkbs5qf/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatch-fancy-pypi-readme-22.7.0-rmylckfsq4zq3xfcddljruoaga7rimqb/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-tomli-2.0.1-aa3brbrja2fcgmut6sm7sbfbs6oyjxni/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pluggy-1.0.0-ekrvl5rqdjiiagq5tbsttbo6rqwrg57z/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pathspec-0.10.1-emo23i62swivzygxccheg42ooswg2pvu/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyparsing-3.0.9-iunsddp3j76dymrjejlpo4mkmy5rlc5r/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-packaging-21.3-h77ffuya56shukyoxwtx6qvsc6zdobis/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-editables-0.3-j6ye2rcrkw2gjkststt4k2ih42ujdjf7/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatchling-1.10.0-nfnrepoazwcn6226lgmyebt6qlkbs5qf/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pluggy-1.0.0-ekrvl5rqdjiiagq5tbsttbo6rqwrg57z/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-typing-extensions-4.3.0-i33eziavtwtpcgg2inj7ehi63tl4vnmy/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-tomli-2.0.1-aa3brbrja2fcgmut6sm7sbfbs6oyjxni/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-65.5.0-wfbdr55dd6b3j6baawtbexfvb2af5ddk/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyparsing-3.0.9-iunsddp3j76dymrjejlpo4mkmy5rlc5r/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-packaging-21.3-h77ffuya56shukyoxwtx6qvsc6zdobis/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-scm-7.0.5-ru7uxtzsslvf7l3hrt3mefx4g3l2oua2/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-typing-extensions-4.3.0-i33eziavtwtpcgg2inj7ehi63tl4vnmy/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-tomli-2.0.1-aa3brbrja2fcgmut6sm7sbfbs6oyjxni/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pathspec-0.10.1-emo23i62swivzygxccheg42ooswg2pvu/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyparsing-3.0.9-iunsddp3j76dymrjejlpo4mkmy5rlc5r/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-packaging-21.3-h77ffuya56shukyoxwtx6qvsc6zdobis/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyparsing-3.0.9-iunsddp3j76dymrjejlpo4mkmy5rlc5r/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-flit-core-3.7.1-en5ectt5emttff3i52wfgwtioyzncgpw/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-editables-0.3-j6ye2rcrkw2gjkststt4k2ih42ujdjf7/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-attrs-22.1.0-pfvxw6i2x4hccm2r2hjk2vytwgpch7te/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pycparser-2.21-w2yyuzczzvywwtq3wpzdu6ygearrhqg6/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-cffi-1.15.0-v7zecbw75l3bwiso7256sloltz6hsvpv/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pycparser-2.21-w2yyuzczzvywwtq3wpzdu6ygearrhqg6/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-wheel-0.37.1-gqowq2ixyohff6v5e3m5b3jqj3rnbqvb/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-65.5.0-wfbdr55dd6b3j6baawtbexfvb2af5ddk/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pip-22.2.2-73zas4bwqqljsjjlxx6irn3ck436mipn/lib/python3.10/site-packages

Then, I start Flux with srun -N 2 -n 2 flux start and my $PYTHONPATH is:

/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/flux-core-0.51.0-2puq2kxgzeddwlmmr3asvyfuhxzbrdq5/lib/flux/python3.1:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyyaml-6.0-aajq667cfzsoxueyiwykcoxuy6fy2fty/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyrsistent-0.18.1-23uu7fl2xvaxuugxmcposnekm24ut25a/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-attrs-22.1.0-pfvxw6i2x4hccm2r2hjk2vytwgpch7te/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-jsonschema-4.16.0-ptg26rwkjnuekqp3pwqyiptydslrfhps/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pycparser-2.21-w2yyuzczzvywwtq3wpzdu6ygearrhqg6/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-cffi-1.15.0-v7zecbw75l3bwiso7256sloltz6hsvpv/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/flux-core-0.51.0-2puq2kxgzeddwlmmr3asvyfuhxzbrdq5/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-65.5.0-wfbdr55dd6b3j6baawtbexfvb2af5ddk/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-cython-0.29.32-yrtzbi75otr3sk2nec2cc5fu2obclc7l/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-typing-extensions-4.3.0-i33eziavtwtpcgg2inj7ehi63tl4vnmy/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-scm-7.0.5-ru7uxtzsslvf7l3hrt3mefx4g3l2oua2/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-tomli-2.0.1-aa3brbrja2fcgmut6sm7sbfbs6oyjxni/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pluggy-1.0.0-ekrvl5rqdjiiagq5tbsttbo6rqwrg57z/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pathspec-0.10.1-emo23i62swivzygxccheg42ooswg2pvu/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyparsing-3.0.9-iunsddp3j76dymrjejlpo4mkmy5rlc5r/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-packaging-21.3-h77ffuya56shukyoxwtx6qvsc6zdobis/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-editables-0.3-j6ye2rcrkw2gjkststt4k2ih42ujdjf7/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatchling-1.10.0-nfnrepoazwcn6226lgmyebt6qlkbs5qf/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatch-vcs-0.2.0-ggwougvgzeld7owlpmteyjslcn65t42w/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatch-fancy-pypi-readme-22.7.0-rmylckfsq4zq3xfcddljruoaga7rimqb/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-flit-core-3.7.1-en5ectt5emttff3i52wfgwtioyzncgpw/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-wheel-0.37.1-gqowq2ixyohff6v5e3m5b3jqj3rnbqvb/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pip-22.2.2-73zas4bwqqljsjjlxx6irn3ck436mipn/lib/python3.10/site-packages

At this point, when I try flux run -N 2 a.out with my MPI application, I get Python errors that py-ply is missing. So, I then spack load py-ply and my $PYTHONPATH becomes:

/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-ply-3.11-ydotoofawtznn6mxlaycwsijigjci5kt/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-wheel-0.37.1-gqowq2ixyohff6v5e3m5b3jqj3rnbqvb/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-65.5.0-wfbdr55dd6b3j6baawtbexfvb2af5ddk/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pip-22.2.2-73zas4bwqqljsjjlxx6irn3ck436mipn/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/flux-core-0.51.0-2puq2kxgzeddwlmmr3asvyfuhxzbrdq5/lib/flux/python3.1:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyyaml-6.0-aajq667cfzsoxueyiwykcoxuy6fy2fty/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyrsistent-0.18.1-23uu7fl2xvaxuugxmcposnekm24ut25a/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-attrs-22.1.0-pfvxw6i2x4hccm2r2hjk2vytwgpch7te/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-jsonschema-4.16.0-ptg26rwkjnuekqp3pwqyiptydslrfhps/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pycparser-2.21-w2yyuzczzvywwtq3wpzdu6ygearrhqg6/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-cffi-1.15.0-v7zecbw75l3bwiso7256sloltz6hsvpv/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/flux-core-0.51.0-2puq2kxgzeddwlmmr3asvyfuhxzbrdq5/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-65.5.0-wfbdr55dd6b3j6baawtbexfvb2af5ddk/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-cython-0.29.32-yrtzbi75otr3sk2nec2cc5fu2obclc7l/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-typing-extensions-4.3.0-i33eziavtwtpcgg2inj7ehi63tl4vnmy/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-setuptools-scm-7.0.5-ru7uxtzsslvf7l3hrt3mefx4g3l2oua2/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-tomli-2.0.1-aa3brbrja2fcgmut6sm7sbfbs6oyjxni/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pluggy-1.0.0-ekrvl5rqdjiiagq5tbsttbo6rqwrg57z/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pathspec-0.10.1-emo23i62swivzygxccheg42ooswg2pvu/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pyparsing-3.0.9-iunsddp3j76dymrjejlpo4mkmy5rlc5r/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-packaging-21.3-h77ffuya56shukyoxwtx6qvsc6zdobis/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-editables-0.3-j6ye2rcrkw2gjkststt4k2ih42ujdjf7/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatchling-1.10.0-nfnrepoazwcn6226lgmyebt6qlkbs5qf/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatch-vcs-0.2.0-ggwougvgzeld7owlpmteyjslcn65t42w/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-hatch-fancy-pypi-readme-22.7.0-rmylckfsq4zq3xfcddljruoaga7rimqb/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-flit-core-3.7.1-en5ectt5emttff3i52wfgwtioyzncgpw/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-wheel-0.37.1-gqowq2ixyohff6v5e3m5b3jqj3rnbqvb/lib/python3.10/site-packages:/scratch/sfg3866/flux/spack/opt/spack/linux-rhel8-cascadelake/gcc-12.2.0/py-pip-22.2.2-73zas4bwqqljsjjlxx6irn3ck436mipn/lib/python3.10/site-packages

and then the flux run command continues without Python errors.

A potential confusing factor is that I have also installed miniconda3 in Spack and I was planning to use Python packages from that miniconda alongside Flux (i.e. in particular, Parsl from conda install -y -c conda-forge parsl). In the examples above, I explicitly did not load miniconda3 to keep things as simple as possible, so I don't think it is a direct factor in this question about py-ply. However, using Parsl and Flux together is a critical part of my project so that is on the horizon. I would deeply appreciate any best practice information about using Parsl and Flux together (e.g. is a pip install parsl into the Python in my Spack environment safer than attempting to use Conda? Could Flux's Python needs get confused with the miniconda3 Python? Or is using the Flux Conda package a better alternative?)

vsoch commented 11 months ago

@stefangary what is strange is that I don't see py-ply until you run spack load, which should not happen. I also don't see six, which is another package with an "install after this version" which leads me to think there is a bug. Could you try removing those version pins (and running the install) and seeing if it shows up? E.g.,

- depends_on("py-ply", type=("build", "run"), when="@0.46.1:")
+ depends_on("py-ply", type=("build", "run"))

If that's the bug, then for the time being you can remove them, and we can post to the spack issue board for maintainers to look into it. Another way to fix it for now would be to manually add the py-ply path, e.g.,:

prefix=$(spack spec --format "{prefix}" py-ply)
export PYTHONPATH=$PYTHONPATH:$prefix

I think there is another way with spack find --paths --format {prefix} but I can't remember it exactly.

So, I then spack load py-ply and my $PYTHONPATH becomes:

Generally spack load won't be the right solution, because it's going to wipe your environment of the previous package paths, and you'd just have py-ply (or your chosen package). It will definitely undo any changes flux has done to paths.

However, using Parsl and Flux together is a critical part of my project so that is on the horizon. I would deeply appreciate any best practice information about using Parsl and Flux together (e.g. is a pip install parsl into the Python in my Spack environment safer than attempting to use Conda? Could Flux's Python needs get confused with the miniconda3 Python? Or is using the Flux Conda package a better alternative?)

Yeah this is a hard problem - personally speaking I tend to not recommend using spack with python libraries because it can get especially messy (as we've seen here). Spack I don't think does a good job of elegantly handling all these different installs. You can most definitely use Flux with spack + a python from another environment (I've done this with conda/mamba) but since you'll be importing flux in your scripts, you can definitely run into errors if there are two installs of a library (and it picks up the wrong one). I ran into quite a bit of trouble, for example, using Flux with a different workflow tool, snakemake, unless they were installed to the same python. What I typically do is install my Python libraries to the same system Python as flux (e.g., you could try pip install <name> --user) but I'm not sure that would work. Or I install flux to the system python, which you have even less control to do! As a third alternative you could try creating your ideal Python environment, but then using Flux bindings directly from it via our releases of flux-python https://pypi.org/project/flux-python/. You will likely need to make sure that the bindings that are installed alongside flux aren't hit, and you'll need to install a matching version. See the install instructions in the README. Those releases are fairly new and we are still looking for feedback, so any/all issues you run into, you can tell me and I'll help https://github.com/flux-framework/flux-python. TLDR, here are some options:

  1. Try installing with those directives removes to (possibly?) remove this bug (and report it)
  2. Manually grab the py-ply path from spack and add to your $PYTHONPATH
  3. Try installing to a different python than in spack / miniconda (but be very careful)
  4. Try creating your ideal environment in a Python that suites you, then install flux bindings to it.
  5. Try installing with a pip that is linked to your spack python and then use --user

That should at least be a start - I left out the ideas that I know absolutely won't work unless you have sudo.

stefangary commented 11 months ago

@vsoch, thank you for this comprehensive response!

I'm still learning Spack; which file are you referring to that has this line in it?

depends_on("py-ply", type=("build", "run"), when="@0.46.1:")

Incidentally, I do have sudo; most of the time I'm working on highly configurable, ephemeral cloud clusters where I have a lot of control (and if I break something, it's easy to start over). If there is a particularly elegant and robust way of doing this with sudo I'm very open to exploring that route. I have already experimented with installing Flux directly on the system.

vsoch commented 11 months ago

Ah sorry! It's this file here https://github.com/spack/spack/blob/99f3b9f06495b3610889d598b1c8d96f86bfe58f/var/spack/repos/builtin/packages/flux-core/package.py#L145. Every spack package has a package.py in "var/spack/repos/builtin" that is the recipe for how to install.

I have already experimented with installing Flux directly on the system.

If it were me, I'd do it this way. This is my goto for using flux with conda: https://github.com/rse-ops/flux-conda/blob/main/Dockerfile.mamba (but you don't need to clone that special branch). And then if you just use the system python, just remove the section that installs mamba/conda and instead install python3 python3-pip and remove the envars like:

 PYTHON=/opt/conda/bin/python PYTHON_PREFIX=PYTHON_EXEC_PREFIX=/opt/conda/lib/python3.8/site-packages

that are telling flux to use a special one.

stefangary commented 11 months ago

Hi @vsoch , thank you very much for these specifics. This was super helpful; at first I didn't see the depends_on line when I searched for it, e.g.:

grep ply /home/sfgary/parsl_flux/spack/var/spack/repos/builtin/packages/flux-core/package.py

but since it is listed on the current version of Spack, it prompted me to consider that the issue is due to my use of Spack v0.19. And yes, Spack v0.20 has the py-ply dependency but Spack v0.19 does not. I can also see that py-ply is missing when I run spack info flux-core with Spack v0.19 but py-ply is listed when the same operation is run with Spack v0.20. When I rebuilt my Spack stack with Spack v0.20, py-ply was not missing and flux run was able to start up fine.

With regards to Spack-installed Flux and Python packages (in my case, Parsl) - what I found seems to work so far is to pip install parsl after I have run spack load flux-sched. I'm viewing this as the equivalent of your option #5, above (but I am not using --user so the installed package files end up inside the Spack directory instead of in my ~/.local).

At this point, it seems like my Spack and Python issues are resolved and I can get Flux to launch simple jobs from a Parsl workflow. However, when I try to get Flux to launch a simple MPI hello world (within a Parsl workflow), it fails (despite being able to run that exact executable with srun and flux run). I'm guessing this MPI issue is likely to be unrelated to the above conversation and I'll create a separate issue.

Thank you very much for your help with this issue. I've learned a great deal about Flux and Spack in the process.

vsoch commented 11 months ago

Nice! I’m definitely happy to help, and that it was just a version issue! I’m not sure what you are trying to flux run, but at least for LAMMPS (which used mpirun) I converted it like this: https://github.com/rse-ops/flux-hpc/blob/f919ca1682df193c4c7af4bb408603de78ba626e/lammps/Dockerfile#L4. And if it helps, here is the very simple work I’ve done with parsl and flux https://github.com/rse-ops/flux-hpc/tree/main/molecular-design-parsl.

Feel free to close this if you are good, and ping me in future issues if I can be helpful! And I’m interested in the workflow you are running, if you might be able to share that? Could we try it in the flux operator?

stefangary commented 11 months ago

@vsoch - Thank you very much for these additional resources! I will close this issue and open a separate issue.

I would be happy to share what I'm working on. Currently my focus is motivated by these hard-coded limits in Parsl and the MPI support in the FluxExecutor will be a good way to address those limits and I'm working with simple examples here. I would be happy to try it with flux operator provided that is supported within Parsl. I will be updating the README's based on what I've learned and improving the overall organization. Eventually, I want to run weather forecasting workflows.

vsoch commented 11 months ago

Oh fantastic! If I watch that parsl_mpi repository, is that the best way to follow your progress? I could also give a shot at porting for the operator too.

stefangary commented 11 months ago

@vsoch - yes, indeed; please feel free to explore/use it. The goal is to create a series of templates for using Parsl with Flux specifically for strengthening support for running MPI jobs within large-scale workflows. Here is an abstract. We also plan to integrate the templates with Globus Compute so these MPI processes can be "fire and forget" via FaaS. I need a day or so to organize/document so it's easier to engage with - right now it's pretty messy.

garlick commented 11 months ago

I think I already alluded to this in an earlier comment, but openpmi may require a bit of extra care with Flux. My advice is to make sure Flux can launch an MPI "hello world" with your openmpi build before getting too far in. There are basically two approaches to bootstrapping openmpi with flux

Edit: I'm posting this to make sure you don't beat your head on it for too long before opening another issue to seek help. It's a reasonable expectation that this stuff should just work, but sadly, often that is not the case.

stefangary commented 11 months ago

@garlick - Thank you very much for this update and the suggestions with OpenMPI! I have indeed verified that I can run my MPI hello world with srun and flux run but it fails when running as part of a Parsl->Flux workflow. I was just gathering all the necessary information (versions, etc.) so I can post a (new) comprehensive issue.

Also, I don't strictly need to use OpenMPI - I'm using it mostly because my initial attempts at building a Spack stack with Flux with Intel failed but my build with gcc ran to completion. I'm new to this so forgive what may be a naïve question: Is it OK to build Flux with gcc and then ask that Flux to run MPI apps compiled with Intel? If so, I can see if it works with Intel.

garlick commented 11 months ago

Yes flux and the application and it's libraries like MPI can use different compilers.

stefangary commented 11 months ago

Great, thanks for confirming. In that case, I will try Intel; I can use the discussion in https://github.com/flux-framework/flux-core/discussions/5208 to inform my testing.

stefangary commented 11 months ago

@garlick and @vsoch - Thank you very much for the help. I was able to get the Parsl -> Flux workflow to run the MPI hello-world app compiled with intel-oneapi-mpi (installed in Spack). This is probably fine for my purposes for the immediate future. Would it be helpful for you and the Flux team if I open an issue about MPI hello-world compiled with OpenMPI not working in Parsl->Flux workflows? It seems that you're already aware of issues with OpenMPI and I don't want a new issue (that is not blocking me) to distract from your efforts. Alternatively, if testing OpenMPI with Flux in a Parsl workflow would make for a useful development environment/use case scenario for you, then it seems opening an issue would be a benefit. Please let me know if you do want that issue opened; otherwise I'll assume it can wait for a while.

For now, I'm figuring out the correct mix of SLURM, Parsl, and Flux configurations to enable the control that I want over MPI-hello-world. I will be reading the docs first and may reach out later with questions. Thank you!

vsoch commented 11 months ago

I'm not particularly interested in openmpi - it sucks on Google Cloud :laughing: but I would love to see your parsl + Flux example so we can port to the Flux Operator (if you are able/willing to share). We likely would install alongside vanilla flux (sans Spack) so it would be another potentially useful setup!

Also - MPI library warning sign - it wants to spell check update to "empire" :red_square: :red_square: !!! :warning: !!!

image

stefangary commented 8 months ago

Hi @vsoch , Thank you for the info about OpenMPI and sorry about delayed response. The tests you ran in the link above are very impressive. I would like to recreate them on the cloud clusters I have access to (which are not k8s, but VMs). Please feel free to reuse the Parsl-Flux example I've been testing. This is a link to the Parsl workflow and a link the script I used to install Flux on the cluster.

vsoch commented 8 months ago

Totally no worries! yes absolutely - please use / repurpose whatever you see fit and try it out on your cloud clusters, and let me know if you want to chat about anything.