search

flux-framework / flux-sched

Fluxion Graph-based Scheduler
GNU Lesser General Public License v3.0
84 stars 39 forks source link

build: `make install` doesn't honor the install prefix in all cases #1140

Open wihobbs opened 3 months ago

wihobbs commented 3 months ago

When running on LC, I'm configuring the installation to my home directory, but it's still trying to do stuff (not sure the word really) to /usr/lib:

PYTHON=/g/g0/hobbs17/k8senv/bin/python ./configure --prefix=/g/g0/hobbs17/flux-core/install-tioga
...
make -j4 install
...
Install the project...
-- Install configuration: "RelWithDebInfo"
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/etc/flux/rc1.d/01-sched-fluxion
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/etc/flux/rc3.d/01-sched-fluxion
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/libexec/flux/cmd/flux-ion-R.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/libexec/flux/cmd/flux-ion-resource.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/__init__.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/edge.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/graph.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/multigraph.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/node.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/__pycache__
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/resourcegraph
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/resourcegraph/V1.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/resourcegraph/__init__.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/resourcegraph/__pycache__
rm: cannot remove '//usr/lib64/flux/python3.6/fluxion': Permission denied
-- linking /usr/lib64/flux/python3.6/fluxion -> /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion
ln: failed to create symbolic link '//usr/lib64/flux/python3.6/fluxion/fluxion': Permission denied
-- Installing: /usr/lib64/flux/modules/sched-fluxion-resource.so
CMake Error at resource/modules/cmake_install.cmake:60 (file):
  file INSTALL cannot copy file
  "/g/g0/hobbs17/flux-sched/resource/modules/sched-fluxion-resource.so" to
  "/usr/lib64/flux/modules/sched-fluxion-resource.so": Permission denied.
Call Stack (most recent call first):
  resource/cmake_install.cmake:57 (include)
  cmake_install.cmake:57 (include)

make: *** [Makefile:110: install] Error 1

Do I need to be specifying something else?

tpatki commented 3 months ago

I am aware that @rountree was also running into this as well yesterday. Tagging him so he can follow.

grondo commented 3 months ago

This does seem like a bug in the cmake build.

However, a workaround might be to run flux-sched make under $prefix/bin/flux start. The Fluxion build should pick up the same prefix as the first flux found in PATH, and maybe that will also fix the python install dirs.

wihobbs commented 3 months ago

I've tested and I believe setting FLUX_CORE_PREFIX and not using ./configure --prefix will get us to the place we want to be:

FLUX_CORE_PREFIX=/g/g0/hobbs17/flux-core/install-tioga ./configure
...
make -j4
...
make -j4 install
Install the project...
-- Install configuration: "RelWithDebInfo"
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/etc/flux/rc1.d/01-sched-fluxion
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/etc/flux/rc3.d/01-sched-fluxion
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/libexec/flux/cmd/flux-ion-R.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/libexec/flux/cmd/flux-ion-resource.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/__init__.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/edge.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/graph.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/multigraph.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/node.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/jsongraph/objects/__pycache__
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/resourcegraph
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/resourcegraph/V1.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/resourcegraph/__init__.py
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion/resourcegraph/__pycache__
-- linking /g/g0/hobbs17/flux-core/install-tioga/lib/flux/python3.6/fluxion -> /g/g0/hobbs17/flux-core/install-tioga/lib/python3.6/site-packages/fluxion
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/flux/modules/sched-fluxion-resource.so
-- Up-to-date: /g/g0/hobbs17/flux-core/install-tioga/lib/flux/modules/sched-fluxion-qmanager.so

(that's me re-running the commands to show that it's indeed linking and installing to the correct paths)

This appears to rely on lines 58-61 of CMakeLists.txt. Also, this appears to work whether you're inside a flux session as @grondo suggested or not. I tried it in the session first.

I'll let our sched developers decide whether this is still an issue or not.

And I'll thank @gabsillis for some helpful CMake tips that helped me parse those files :-)

grondo commented 3 months ago

FLUX_CORE_PREFIX should be automatically set to the prefix of the first flux found in PATH (which is why I suggested running under $prefix/flux start or $prefix/flux make. This is how it used to work before cmake and I think is how it is supposed to work still based on this comment in 8b503eb5d809182a48d9f100cfd5846b800c14c7:

find flux-core by finding flux, match prefix if none specified

wihobbs commented 3 months ago

By the way, setting FLUX_CORE_PREFIX is not necessary if you're inside a flux session. Cmake is able to pick up the prefix of the first flux in PATH.

wihobbs commented 3 months ago

I was just setting FLUX_CORE_PREFIX above as a workaround to having to start a flux session to build flux-sched.

grondo commented 3 months ago

Thanks for that clarification!

So I think the only thing not working here is that use of --prefix should behave the same as if you set FLUX_CORE_PREFIX. (Seems like the build is perhaps picking up some paths from /usr/bin/flux - the first flux found in PATH)

trws commented 3 months ago

It seems like this one might be a mistake in ordering the priorities. Possibly not providing the cmake enough information from the wrapper to know it should override. Does it do what you want if you explicitly set CMAKE_INSTALL_PREFIX=<prefix you want> rather than using the wrapper? That would help narrow down where I mixed up the logic.