No valid pineappl installation found when running pinefarm

comane commented 1 year ago

I am running pinefarm (installed with pip) as following:

pinefarm run runcards/ATLAS_TTB_13TEV_TOT theory_200_1.yaml

The command is executed in a fresh conda environment in which I have only installed lhapdf (but no pineappl.).

The run crashes in the following way, so it seems that pinefarm is not installing PineAPPL on the fly as said in (https://pinefarm.readthedocs.io/en/latest/run.html)

Error detected in "launch auto "
write debug file /store/DAMTP/mnc33/Projects_store/PhD/nnpdf40_pheno/pinefarm_runs/results/200-ATLAS_TTB_13TEV_TOT--20231115093751/ATLAS_TTB_13TEV_TOT/run_01_tag_1_debug.log 
If you need help with this issue, please, contact us on https://answers.launchpad.net/mg5amcnlo
str : No valid pineappl installation found. 
    Please set the path to pineappl-config by using 
    MG5_aMC> set <absolute-path-to-pineappl>/bin/pineappl-config

Moreover, the .prefix/bin folder generated in the same folder in which I run pinefarm is empty

comane commented 1 year ago

Note: that in issue https://github.com/NNPDF/pinefarm/issues/47 I was running pinefarm in an environment with pineappl already installed. This however was giving an error shown below and reported at (https://bugs.launchpad.net/mg5amcnlo/+bug/2043460)

    Package pineappl_capi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `pineappl_capi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'pineappl_capi' found
    g++ -std=c++11 -O -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /store/DAMTP/mnc33/miniconda3/envs/pinefarm/include -fdebug-prefix-map=/usr/share/miniconda/envs/test/conda-bld/lhapdf_1632848490271/work=/usr/local/src/conda/lhapdf-6.4.0 -fdebug-prefix-map=/store/DAMTP/mnc33/miniconda3/envs/pinefarm=/usr/local/src/conda-prefix -I/store/DAMTP/mnc33/miniconda3/envs/pinefarm/include -c -I. pineappl_interface.cc
    pineappl_interface.cc:14:10: fatal error: pineappl_capi.h: No such file or directory
       14 | #include <pineappl_capi.h>
          | ^~~~~~~~~~~~~~~~~
    compilation terminated.
    make: *** [makefile:95: pineappl_interface.o] Error 1

felixhekhorn commented 1 year ago

oookay, so this sounds back like a pinefarm problem (and not an mg5 one) ...

when you hit run for the first time all necessary tools should indeed install themselves (and if they are already present, the script will tell you)
in your case that should be pineappl (including the CAPI) and mg5 - did you hit those installers? (from here to there )
I think those installers print to stdout - can you check whether you have cargo cinstall ... there somewhere (this one)? that should install the CAPI
is the CAPI actually present in your current env? i.e. does the file pineappl_capi.h exist somewhere? like-wise does the command pineappl-config exist somewhere? (I'm not sure at the moment how mg5 is receiving that path ...)

comane commented 1 year ago

is the CAPI actually present in your current env? i.e. does the file pineappl_capi.h

No this file doesn't exist anywhere in my env. I am not even sure it is supposed to be there since I am not installing in dev mode.

does the command pineappl-config exist somewhere? (I'm not sure at the moment how mg5 is receiving that path ...)

I can't find it in my conda environment bin. This might be the problem since mg5 crashes like so:

Error detected in "launch auto " write debug file /store/DAMTP/mnc33/Projects_store/PhD/nnpdf40_pheno/pinefarm_runs/results/200-ATLAS_TTB_13TEV_TOT--20231116151158/ATLAS_TTB_13TEV_TOT/run_01_tag_1_debug.log If you need help with this issue, please, contact us on https://answers.launchpad.net/mg5amcnlo str : No valid pineappl installation found. Please set the path to pineappl-config by using MG5_aMC> set <absolute-path-to-pineappl>/bin/pineappl-config

felixhekhorn commented 1 year ago

No this file doesn't exist anywhere in my env. I am not even sure it is supposed to be there since I am not installing in dev mode.

you mean dev mode for pinefarm? that does not matter - since mg5 will need it in any case (since it is compiling on its own) and indeed it should be installed in any case; the commands I posted above ARE pinefarm, so pinefarm can and (hopefully) will install programs on its own; and here for mg5 pinefarm should realize that he needs the CAPI and so provide the CAPI

comane commented 1 year ago

Regarding this:

I think those installers print to stdout - can you check whether you have cargo cinstall ... there somewhere (this one)? that should install the CAPI

I do not have cinstall:

cargo cinstall
error: no such command: `cinstall`

    Did you mean `install`?

    View all installed commands with `cargo --list`

felixhekhorn commented 1 year ago

This sounds like you really not went through the install.pineappl routine ...

Can you please make a new env and print the full output of pinefarm run runcards/ATLAS_TTB_13TEV_TOT theory_200_1.yaml ? specifically, I'm interested in the install outputs this call should trigger ...

comane commented 1 year ago

@felixhekhorn , I had to cancel most of the fortran error output, as the comment would have not fitted in the page

felixhekhorn commented 1 year ago

also a text file would have been fine :rofl: (you can still edit your post :innocent: )

now the relevant lines are these ones:

Could not find directory of OpenSSL installation, and this -sys crate cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it, you can set the OPENSSL_DIR environment variable for the
compilation process.

Make sure you also have the development packages of openssl installed.
For example, libssl-dev on Ubuntu or openssl-devel on Fedora.

and so cargo fails, i.e. pineappl fails and so you run into the problem - can you please try to apply the suggested fix?

comane commented 1 year ago

Here is the .txt file pinefarm_log.txt

comane commented 1 year ago

@felixhekhorn thank you so much for the assistance.

So after installing openssl it works!

I had to do the following:

wget https://www.openssl.org/source/openssl-1.1.1k.tar.gz
tar -zxvf openssl-1.1.1k.tar.gz
cd openssl-1.1.1k

./config --prefix=$HOME/usr --openssldir=$HOME/usr/openssl
make
make install

Add folder containing openssl.pc (libssl.pc) to PKG_CONFIG_PATH

the run generates the pineappl grid, however I still get an error at the end of the run:

-------------------------------------------------------------------------
bin   PineAPPL         MC        sigma      central         min      max
                                 1/100   sigma   1/1000   1/1000   1/1000
-------------------------------------------------------------------------
0  7.888729e+02  7.887245e+02   0.105   0.179   0.1882   0.1876   0.1935
Traceback (most recent call last):
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/bin/pinefarm", line 8, in <module>
    sys.exit(command())
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/cli/run.py", line 31, in subcommand
    main(dataset, theory_card, pdf)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/cli/run.py", line 67, in main
    run_dataset(runner)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/cli/run.py", line 132, in run_dataset
    runner.annotate_versions()
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/external/interface.py", line 121, in annotate_versions
    versions = self.collect_versions()
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/external/mg5/__init__.py", line 235, in collect_versions
    subprocess.run(
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/subprocess.py", line 1837, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'brz'
Thanks for using LHAPDF 6.4.0. Please make sure to cite the paper:
  Eur.Phys.J. C75 (2015) 3, 132  (http://arxiv.org/abs/1412.7420)

felixhekhorn commented 1 year ago

good we were able to solve something! :+1:

how did you solve the brz issue? (that should have been installed together with mg5 ...)

remember to close the mg5 bug (and say it actually was not a bug)

NNPDF / pinefarm

No valid pineappl installation found when running pinefarm #50