libOmegah not installed?

[ikalash]

I turned on Omegah in one of the nightlies on mockba. It appears if you do a make install, the Omegah library does not get installed. This shows up later when building CALI on top of Albany:

https://sems-cdash-son.sandia.gov/cdash/viewBuildError.php?buildid=53182

/usr/bin/ld: cannot find -lOmega_h::omega_h

I know we don't really care about CALI anymore but I suspect this will be a problem for MALI too.

@bartgol can you please have a look?

[bartgol]

Ok, I'll take a look.

That said, I think it's time to discuss discontinuing support for CISM. I'll bring it up at the next albany telecon.

[ikalash]

Thanks! I think officially we don't support CISM, but I still build it on some of my machines. I opened this issue b/c I suspect it will be a problem for MPAS too. If/when there are changes that prevent CISM from working with Albany that are nontrivial fixes, I will drop the CALI testing.

[bartgol]

I know MALI uses export_albany.in to get the albany link flags. Does CALI do the same?

[ikalash]

I know MALI uses export_albany.in to get the albany link flags. Does CALI do the same?

It doesn't but I don't think that that's the problem. Is there supposed to be a libOmega_h that gets build by Albany and installed in the install/lib dir? It isn't there and this is why CISM is complaining. I think MALI would complain too b/c one of the libs it links according to export_albany.in is lomega_h. Where is that lib supposed to be installed when OMEGA_H is enabled?

[ikalash]

Have you tried building MALI on top of Albany with Omega_h enabled? Does it build?

[bartgol]

Omega_h is installed locally in the build tree, I never added logic for installing it along side albany. I'm looking into ways to do it.

[ikalash]

Thanks for the explanation @bartgol . It's possible the library doesn't need to be installed alongside Albany. But I think it needs to be found when linking a code against Albany, wherever it is, to avoid errors like this:

/usr/bin/ld: cannot find -lOmega_h::omega_h

[bartgol]

Thanks for the explanation @bartgol . It's possible the library doesn't need to be installed alongside Albany. But I think it needs to be found when linking a code against Albany, wherever it is, to avoid errors like this:

/usr/bin/ld: cannot find -lOmega_h::omega_h

I think it's cleaner if we put it in the installation folder. Once Albany is installed, one may think it's ok to remove the build tree. I'm not sure what spack does with the build trees once the package is installed, btw

[ikalash]

@bartgol : I agree that it's cleaner. That's a good question about spack. We will have to modify the spack build to enable omegah, and will see then what happens. I will create an issue on this in the spack repo.

[bartgol]

Ok, no rush, I was just thinking ahead. I will fix mpas first, then we can think of spack.

[ikalash]

Sounds good. I will look at spack within the next week or so. I have opened an issue on this so we don't forget in the spack repo.

[bartgol]

Ok, I pushed a fix that seems to install omegah correctly alongside Albany. I verified that this is also correct in the export_albany.in file, which is used by MALI. I don't know what the CISM build does.

Notice that Omega_h is hard-coding lib (relative to the install prefix) as the subfolder where to put its libraries. OTOH, on most machines CMake installs albany libs in the lib64 subfolder. So projects that are hard-coding library dirs (rather than relying on CMake machinery) need both lib and lib64 (for omegah and albany libs respectively) in their link line. For MALI, this doesn't matter, since we generate the file mentioned above with the full link line to use. If CALI uses that file, we're good. If it doesn't, it must list both lib and lib64 folders in the link line.

@cwsmith do you think we can get Omega_h to use paths provided by GnuInstallDirs instead of hard-coded ones? E.g., CMAKE_INSTALL_LIBDIR instead of lib?

[ikalash]

Thanks I will test it this afternoon.

[cwsmith]

@bartgol Certainly. I'll take a look.

[ikalash]

So is a libOmega_h supposed to be installed in the Albany install/lib64 dir? I am not seeing it get installed after pulling your change / rebuilding.

[bartgol]

If CMAKE_INSTALL_PREFIX is valid (i.e., writable), then yes, it should. On my workstation it does get installed there. Are you enabling omegah? The cmake output should have some info regarding omegah, including whether it was already found somewhere. Does it say anything about omegah?

[ikalash]

@bartgol : I think I know the problem. It looks like the omega_h library gets installed in install/lib whereas the other Albany libraries get installed in install/lib64. Is it possible to make them all installed in the same place, say lib64?

[bartgol]

Yeah, that's the point I was discussing above. Unfortunately, this is a detail of Omega_h installation logic. I proposed to @cwsmith to make omegah use the GnuInstallDirs rather than hardcoding lib for the install tree subfolders.

Until then, the easiest thing to do is to see if CALI can use both lib and lib64 in its link line. Based on what I saw in export_albany.in, I think MALI should work fine (though I haven't tested it).

[mperego]

It seems that @cwsmith is already working on a PR to fix this https://github.com/SCOREC/omega_h/pull/64.

[ikalash]

Yeah, that's the point I was discussing above. Unfortunately, this is a detail of Omega_h installation logic. I proposed to @cwsmith to make omegah use the GnuInstallDirs rather than hardcoding lib for the install tree subfolders.

Until then, the easiest thing to do is to see if CALI can use both lib and lib64 in its link line. Based on what I saw in export_albany.in, I think MALI should work fine (though I haven't tested it).

Sorry, I missed some of the discussion above due to travel. I can fix the CALI logic, yes - just did this. Thanks!

[ikalash]

@bartgol : can you please verify that MALI builds with the changes? I can't seem to get CALI to link properly even after mucking with the link libraries in several different ways.

[cwsmith]

@bartgol https://github.com/SCOREC/omega_h/releases/tag/scorec-v10.8.1 uses GNUInstallDirs for install destinations.

[ikalash]

@bartgol : also can you please clarify, will omega h always be optional for Albany or sometime you will make it a requirement?

[bartgol]

I don't know if we have decided that. I think it can stay optional, though it may be a requirement for our LandIce work at some point...

[mperego]

I think it's good to keep it optional for the time being.

[ikalash]

It's still not clear to me if MALI builds against Albany with Omega_h turned on... how is this error circumvented in that case, if it builds?

/usr/bin/ld: cannot find -lOmega_h::omega_h

[bartgol]

The file export_albany.in does not contain -lOmega_h::omega_h but just -lomega_h. When I look at <bld>/dummy/CMakeFiles/dummy.dir/link.txt I see all libs linked via full path. This is b/c <src>/cmake/dummy/CMakeLists.txt specifies to use NEW behavior of policy CMP0060, which does precisely that: link libs by full path even if from an implicit link directory.

I'm not sure how you generate the link line for CISM. If you use the content of export_albany.in though, you should get the correct behavior.

[ikalash]

Ok, I have confirmed that changing the -lOmega_h::omega_h to -lomega_h in the link line fixes the CALI build problem. I don't have a good fix, I can just hack build.ninja (when building with ninja) to search/replace the library name.

Thanks @bartgol for your patience and for looking at this. I think MALI will work too now but at some point we should verify, maybe once Omega_h is turned on in the PM weekly builds.

[bartgol]

I don't think we need to rush to test MALI. We cannot yet use Omega_h meshes in MALI (we don't have extruded omegah meshes), so there's not much to test other than the fact that compilation/linking go through...

If this gives us headaches, we can just disable Omega_h in mali/cali builds (especially CALI, since we don't really use omegah there)