search

PanDAWMS / panda-client

package for user tools
Apache License 2.0
13 stars 15 forks source link

Simplify the cpack options in AthenaUtils.archiveWithCpack #26

Open krasznaa opened 1 year ago

krasznaa commented 1 year ago

Since there was only a limited reaction on the corresponding ATLAS e-mail thread, I thought it would be better to open an issue here.

The following way of calling CPack is way "over engineered". :frowning:

https://github.com/PanDAWMS/panda-client/blob/master/pandaclient/AthenaUtils.py#L1067-L1070

Most of the -D arguments are unnecessary in there. But there is one that is actively disruptive: -D CPACK_INSTALL_CMAKE_PROJECTS="" That one explicitly tells CPack not to install anything.

With the way that AtlasCMake has operated for years, this option was simply ignored. But now that I'm trying to make the ATLAS builds behave in a "more standard" way, this flag prevents prun from collecting the code of the users that they are trying to execute a job with.

Please remove that argument from the CPack call. It was not needed with old ATLAS analysis releases either, and is now actively hurtful with the latest ones.

tmaeno commented 1 year ago

Removed in 1.5.45. Will be deployed on CVMFS shortly.

tmaeno commented 1 year ago

Hi t3sw,

Please deploy panda-client 1.5.45 on CVMFS. Thanks,

Tadashi

On Mon, Apr 24, 2023 at 1:38 PM Attila Krasznahorkay < @.***> wrote:

Since there was only a limited reaction on the corresponding ATLAS e-mail thread, I thought it would be better to open an issue here.

The following way of calling CPack https://cmake.org/cmake/help/latest/manual/cpack.1.html is way "over engineered". 😦

https://github.com/PanDAWMS/panda-client/blob/master/pandaclient/AthenaUtils.py#L1067-L1070

Most of the -D arguments are unnecessary in there. But there is one that is actively disruptive: -D CPACK_INSTALL_CMAKE_PROJECTS="" That one explicitly tells CPack not to install anything.

With the way that AtlasCMake https://gitlab.cern.ch/atlas/atlasexternals/-/tree/master/Build/AtlasCMake has operated for years, this option was simply ignored. But now that I'm trying to make the ATLAS builds behave in a "more standard" way, this flag prevents prun from collecting the code of the users that they are trying to execute a job with.

Please remove that argument from the CPack call. It was not needed with old ATLAS analysis releases either, and is now actively hurtful with the latest ones.

— Reply to this email directly, view it on GitHub https://github.com/PanDAWMS/panda-client/issues/26, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSIT2IX5ZPTKP3ZAVAL2L3XCZQ4BANCNFSM6AAAAAAXJOLRI4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

tmaeno commented 1 year ago

Hi Tadashi,

This simple test fails (there may be more - testing in progress).

(also seen on alma9) (Athena,22.0.50) centos7

2: panda pathena (x509) (bash) ...

cmd: pathena HelloWorldOptions.py --outDS=user.desilva.test.656cea13-c8ec-434a-8062-6f3f0e343235.x86_64-bash-pythonnone-x509 --noOutput

INFO : using CMTCONFIG=x86_64-centos7-gcc11-opt

INFO : extracting run configuration

INFO : ConfigExtractor > No Input

INFO : archiving source files

INFO : The build directory is /cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea

ERROR : cpack failed

panda pathena (x509) (bash) [FAILED]

This one on alma9 looks like a glitch and not a panda-client issue but I report for completeness.

4: panda prun container (x509) (bash) ...

Note: centos9 support is new.

  You may want to use a centos7 container if you have issues.

cmd: prun --exec='./myPunJob.sh' --outDS=user.desilva.test.2a423b6a-f5f6-480c-ada5-ea271cd350ab.x86_64-bash-pythonnone-x509 --noBuild --containerImage library/busybox:latest

INFO : gathering files under /home/desilva/tmp/desilva/.alrb/testing/tester.AlboMD/panda/bash/none/x509/prunContainer

INFO : upload sandbox

INFO : submit user.desilva.test.2a423b6a-f5f6-480c-ada5-ea271cd350ab.x86_64-bash-pythonnone-x509/

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

503 Service Unavailable

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

ERROR : task submission failed with 255

panda prun container (x509) (bash) [FAILED]

Please let me know when you have a new version.

Thanks !

regards, Asoka

On Apr 24, 2023, at 4:51 AM, Tadashi Maeno @.***> wrote:

Hi t3sw,

Please deploy panda-client 1.5.45 on CVMFS. Thanks,

Tadashi

On Mon, Apr 24, 2023 at 1:38 PM Attila Krasznahorkay @.**@.>> wrote:

Since there was only a limited reaction on the corresponding ATLAS e-mail thread, I thought it would be better to open an issue here.

The following way of calling CPackhttps://cmake.org/cmake/help/latest/manual/cpack.1.html is way "over engineered". 😦

https://github.com/PanDAWMS/panda-client/blob/master/pandaclient/AthenaUtils.py#L1067-L1070

Most of the -D arguments are unnecessary in there. But there is one that is actively disruptive: -D CPACK_INSTALL_CMAKE_PROJECTS="" That one explicitly tells CPack not to install anything.

With the way that AtlasCMakehttps://gitlab.cern.ch/atlas/atlasexternals/-/tree/master/Build/AtlasCMake has operated for years, this option was simply ignored. But now that I'm trying to make the ATLAS builds behave in a "more standard" way, this flag prevents prun from collecting the code of the users that they are trying to execute a job with.

Please remove that argument from the CPack call. It was not needed with old ATLAS analysis releases either, and is now actively hurtful with the latest ones.

— Reply to this email directly, view it on GitHubhttps://github.com/PanDAWMS/panda-client/issues/26, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABSIT2IX5ZPTKP3ZAVAL2L3XCZQ4BANCNFSM6AAAAAAXJOLRI4. You are receiving this because you are subscribed to this thread.Message ID: @.***>

tmaeno commented 1 year ago

With 22.0.50,Athena

$ cpack -B /afs/cern.ch/user/t/tmaeno/Offline/current/run/72a0f5af-245c-418b-a646-ddd44d057a88 -D CPACK_PACKAGE_FILE_NAME=sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70 -G TGZ -D CPACK_PACKAGE_NAME="" -D CPACK_PACKAGE_VERSION="" -D CPACK_PACKAGE_VERSION_MAJOR="" -D CPACK_PACKAGE_VERSION_MINOR="" -D CPACK_PACKAGE_VERSION_PATCH="" -D CPACK_PACKAGE_DESCRIPTION=""
CPack Error: Please specify build tree of the project that uses CMake using CPACK_INSTALL_CMAKE_PROJECTS, specify CPACK_INSTALL_COMMANDS, CPACK_INSTALL_SCRIPT, or CPACK_INSTALLED_DIRECTORIES.

while

$ cpack -B /afs/cern.ch/user/t/tmaeno/Offline/current/run/72a0f5af-245c-418b-a646-ddd44d057a88 -D CPACK_PACKAGE_FILE_NAME=sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70 -G TGZ -D CPACK_PACKAGE_NAME="" -D CPACK_PACKAGE_VERSION="" -D CPACK_PACKAGE_VERSION_MAJOR="" -D CPACK_PACKAGE_VERSION_MINOR="" -D CPACK_PACKAGE_VERSION_PATCH="" -D CPACK_PACKAGE_DESCRIPTION="" -D CPACK_INSTALL_CMAKE_PROJECTS=""
CPack: Create package using TGZ
CPack: Install projects
CPack: Create package
CPack: - package: /afs/cern.ch/user/t/tmaeno/Offline/current/run/72a0f5af-245c-418b-a646-ddd44d057a88/sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70.tar.gz generated.

Apparently -D CPACK_INSTALL_CMAKE_PROJECTS="" is required for old releases.

@krasznaa , what should we do then?

krasznaa commented 1 year ago

Curious. Is prun not executing the cpack command inside of the directory that it deemed as the "build directory"? Because it really should. That would make its own work a whole lot easier.

I.e. when executing the type of command that you posted Tadashi inside of my "build directory", I get:

Singularity> cpack -B /tmp/krasznaa/ -D CPACK_PACKAGE_FILE_NAME=sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70 -G TGZ -D CPACK_PACKAGE_NAME="" -D CPACK_PACKAGE_VERSION="" -D CPACK_PACKAGE_VERSION_MAJOR="" -D CPACK_PACKAGE_VERSION_MINOR="" -D CPACK_PACKAGE_VERSION_PATCH="" -D CPACK_PACKAGE_DESCRIPTION=""
CPack: Create package using TGZ
CPack: Install projects
CPack: - Install scripts: /cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea/x86_64-centos7-gcc11-opt/cmake/modules/scripts/cpack_install.cmake
CPack: - Install script: /cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea/x86_64-centos7-gcc11-opt/cmake/modules/scripts/cpack_install.cmake
CPack: Create package
CPack: - package: /tmp/krasznaa/sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70.tar.gz generated.
Singularity>

But if I try to execute it from a different directory, sure enough, I get:

Singularity> cd ..
Singularity> cpack -B /tmp/krasznaa/ -D CPACK_PACKAGE_FILE_NAME=sources.44d5a7a4-4f2d-4344-8f7c-ab6d8ac99a70 -G TGZ -D CPACK_PACKAGE_NAME="" -D CPACK_PACKAGE_VERSION="" -D CPACK_PACKAGE_VERSION_MAJOR="" -D CPACK_PACKAGE_VERSION_MINOR="" -D CPACK_PACKAGE_VERSION_PATCH="" -D CPACK_PACKAGE_DESCRIPTION=""
CPack Error: Please specify build tree of the project that uses CMake using CPACK_INSTALL_CMAKE_PROJECTS, specify CPACK_INSTALL_COMMANDS, CPACK_INSTALL_SCRIPT, or CPACK_INSTALLED_DIRECTORIES.
Singularity>

But then again, when being outside of the "build directory", and specifying a -B argument to the command to tell it to use some other directory as the target of the TGZ file, I could not make the command work at all. :confused:

So... where does Panda run these commands from exactly...?

krasznaa commented 1 year ago

Also, this line in Asoka's log is pretty worrisome:

INFO : The build directory is /cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea

The code should really be able to recognize when no build directory exists. As in that case it just does not need to package up anything with CPack...

tmaeno commented 1 year ago

I was asked to execute cpack from the first entry of $CMAKE_PREFIX_PATH.

INFO : The build directory is xyz

this message means cpack is executed from xyz. I suppose that the first entry of $CMAKE_PREFIX_PATH points to user's build dir if the user has it and sets up the environment properly. In Asoka's test, I guess he didn't have the build dir, cpack was executed from/cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea, and it used to create an empty tarball, but now it fails.

If the easiest solution is to skip cpack when the user doesn't have own build dir, how can I check the presence of user's build dir using env variables or something?

krasznaa commented 1 year ago

To be correct, you execute cpack from ${CMAKE_PREFIX_PATH[0]}/... I.e. "one directory up" from the first entry in CMAKE_PREFIX_PATH. (This is what the extra os.path.dirname(...) does in: https://github.com/PanDAWMS/panda-client/blob/master/pandaclient/AthenaUtils.py#L1063)

This is correct. When the user actually uses a "build directory", that is indeed where the directory is. But when there's no "build directory" used, just an existing ATLAS release, that directory is usually some path on CVMFS.

I believe the best thing to do is to check whether there is a file called CPackConfig.cmake in the "prospective build directory". If there is such a file there, then that does look like a "build directory". But if not, then cpack will not know what to do anyway. As by default, unless told otherwise (which your command doesn't do), it looks for a file with that name. (In the current directory, or in the directory specified with -B.)

So if I understand correctly, the issue here is really just to not try to use cpack when it shouldn't be used. I still believe that all old releases will work with the modified cpack call as long as that call is made in an actual "build directory". :wink:

tmaeno commented 1 year ago

OK, I've added the change in 1.5.46.

tmaeno commented 1 year ago

Hi Asoka,

Please deploy 1.5.46. I've added the fix following Attila's suggestion.

Tadashi

On Mon, Apr 24, 2023 at 5:47 PM Asoka De Silva @.***> wrote:

Hi Tadashi,

This simple test fails (there may be more - testing in progress).

(also seen on alma9) (Athena,22.0.50) centos7

2: panda pathena (x509) (bash) ...

cmd: pathena HelloWorldOptions.py --outDS=user.desilva.test.656cea13-c8ec-434a-8062-6f3f0e343235.x86_64-bash-pythonnone-x509 --noOutput

INFO : using CMTCONFIG=x86_64-centos7-gcc11-opt

INFO : extracting run configuration

INFO : ConfigExtractor > No Input

INFO : archiving source files

INFO : The build directory is /cvmfs/ atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea

ERROR : cpack failed

panda pathena (x509) (bash) [ FAILED]

This one on alma9 looks like a glitch and not a panda-client issue but I report for completeness.

4: panda prun container (x509) (bash) ...

Note: centos9 support is new.

  You may want to use a centos7 container if you have issues.

cmd: prun --exec='./myPunJob.sh' --outDS=user.desilva.test.2a423b6a-f5f6-480c-ada5-ea271cd350ab.x86_64-bash-pythonnone-x509 --noBuild --containerImage library/busybox:latest

INFO : gathering files under /home/desilva/tmp/desilva/.alrb/testing/tester.AlboMD/panda/bash/none/x509/prunContainer

INFO : upload sandbox

INFO : submit user.desilva.test.2a423b6a-f5f6-480c-ada5-ea271cd350ab.x86_64-bash-pythonnone-x509/

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

503 Service Unavailable

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

ERROR : task submission failed with 255

panda prun container (x509) (bash) [ FAILED]

Please let me know when you have a new version.

Thanks !

regards, Asoka

On Apr 24, 2023, at 4:51 AM, Tadashi Maeno @.***> wrote:

Hi t3sw,

Please deploy panda-client 1.5.45 on CVMFS. Thanks,

Tadashi

On Mon, Apr 24, 2023 at 1:38 PM Attila Krasznahorkay < @.***> wrote:

Since there was only a limited reaction on the corresponding ATLAS e-mail thread, I thought it would be better to open an issue here.

The following way of calling CPack https://cmake.org/cmake/help/latest/manual/cpack.1.html is way "over engineered". 😦

https://github.com/PanDAWMS/panda-client/blob/master/pandaclient/AthenaUtils.py#L1067-L1070

Most of the -D arguments are unnecessary in there. But there is one that is actively disruptive: -D CPACK_INSTALL_CMAKE_PROJECTS="" That one explicitly tells CPack not to install anything.

With the way that AtlasCMake https://gitlab.cern.ch/atlas/atlasexternals/-/tree/master/Build/AtlasCMake has operated for years, this option was simply ignored. But now that I'm trying to make the ATLAS builds behave in a "more standard" way, this flag prevents prun from collecting the code of the users that they are trying to execute a job with.

Please remove that argument from the CPack call. It was not needed with old ATLAS analysis releases either, and is now actively hurtful with the latest ones.

— Reply to this email directly, view it on GitHub https://github.com/PanDAWMS/panda-client/issues/26, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSIT2IX5ZPTKP3ZAVAL2L3XCZQ4BANCNFSM6AAAAAAXJOLRI4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

tmaeno commented 1 year ago

Hi Tadashi,

Done when you see this or newer

\tail -n 1 /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/logDir/lastUpdate

2023Apr25 19:40 | lxcvmfs169.cern.ch | 1682444438

Thanks !

regards, Asoka

On Apr 25, 2023, at 1:23 AM, Tadashi Maeno @.***> wrote:

Hi Asoka,

Please deploy 1.5.46. I've added the fix following Attila's suggestion.

Tadashi

On Mon, Apr 24, 2023 at 5:47 PM Asoka De Silva @.**@.>> wrote: Hi Tadashi,

This simple test fails (there may be more - testing in progress).

(also seen on alma9) (Athena,22.0.50) centos7 2: panda pathena (x509) (bash) ...

cmd: pathena HelloWorldOptions.py --outDS=user.desilva.test.656cea13-c8ec-434a-8062-6f3f0e343235.x86_64-bash-pythonnone-x509 --noOutput INFO : using CMTCONFIG=x86_64-centos7-gcc11-opt INFO : extracting run configuration INFO : ConfigExtractor > No Input INFO : archiving source files INFO : The build directory is /cvmfs/atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallAreahttp://atlas.cern.ch/repo/sw/software/22.0/Athena/22.0.50/InstallArea ERROR : cpack failed panda pathena (x509) (bash) [FAILED]

This one on alma9 looks like a glitch and not a panda-client issue but I report for completeness.

4: panda prun container (x509) (bash) ... Note: centos9 support is new. You may want to use a centos7 container if you have issues.

cmd: prun --exec='./myPunJob.sh' --outDS=user.desilva.test.2a423b6a-f5f6-480c-ada5-ea271cd350ab.x86_64-bash-pythonnone-x509 --noBuild --containerImage library/busybox:latest INFO : gathering files under /home/desilva/tmp/desilva/.alrb/testing/tester.AlboMD/panda/bash/none/x509/prunContainer INFO : upload sandbox INFO : submit user.desilva.test.2a423b6a-f5f6-480c-ada5-ea271cd350ab.x86_64-bash-pythonnone-x509/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

503 Service Unavailable

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

ERROR : task submission failed with 255 panda prun container (x509) (bash) [FAILED]

Please let me know when you have a new version.

Thanks !

regards, Asoka

On Apr 24, 2023, at 4:51 AM, Tadashi Maeno @.**@.>> wrote:

Hi t3sw,

Please deploy panda-client 1.5.45 on CVMFS. Thanks,

Tadashi

On Mon, Apr 24, 2023 at 1:38 PM Attila Krasznahorkay @.**@.>> wrote:

Since there was only a limited reaction on the corresponding ATLAS e-mail thread, I thought it would be better to open an issue here.

The following way of calling CPackhttps://cmake.org/cmake/help/latest/manual/cpack.1.html is way "over engineered". 😦

https://github.com/PanDAWMS/panda-client/blob/master/pandaclient/AthenaUtils.py#L1067-L1070

Most of the -D arguments are unnecessary in there. But there is one that is actively disruptive: -D CPACK_INSTALL_CMAKE_PROJECTS="" That one explicitly tells CPack not to install anything.

With the way that AtlasCMakehttps://gitlab.cern.ch/atlas/atlasexternals/-/tree/master/Build/AtlasCMake has operated for years, this option was simply ignored. But now that I'm trying to make the ATLAS builds behave in a "more standard" way, this flag prevents prun from collecting the code of the users that they are trying to execute a job with.

Please remove that argument from the CPack call. It was not needed with old ATLAS analysis releases either, and is now actively hurtful with the latest ones.

— Reply to this email directly, view it on GitHubhttps://github.com/PanDAWMS/panda-client/issues/26, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABSIT2IX5ZPTKP3ZAVAL2L3XCZQ4BANCNFSM6AAAAAAXJOLRI4. You are receiving this because you are subscribed to this thread.Message ID: @.***>