No valid pineappl installation found when running pinefarm

[comane]

I am running pinefarm (installed with pip) as following:

pinefarm run runcards/ATLAS_TTB_13TEV_TOT theory_200_1.yaml

The command is executed in a fresh conda environment in which I have only installed lhapdf (but no pineappl.).

The run crashes in the following way, so it seems that pinefarm is not installing PineAPPL on the fly as said in (https://pinefarm.readthedocs.io/en/latest/run.html)

Error detected in "launch auto "
write debug file /store/DAMTP/mnc33/Projects_store/PhD/nnpdf40_pheno/pinefarm_runs/results/200-ATLAS_TTB_13TEV_TOT--20231115093751/ATLAS_TTB_13TEV_TOT/run_01_tag_1_debug.log 
If you need help with this issue, please, contact us on https://answers.launchpad.net/mg5amcnlo
str : No valid pineappl installation found. 
    Please set the path to pineappl-config by using 
    MG5_aMC> set <absolute-path-to-pineappl>/bin/pineappl-config 

Moreover, the .prefix/bin folder generated in the same folder in which I run pinefarm is empty

[comane]

Note: that in issue https://github.com/NNPDF/pinefarm/issues/47 I was running pinefarm in an environment with pineappl already installed. This however was giving an error shown below and reported at (https://bugs.launchpad.net/mg5amcnlo/+bug/2043460)

    Package pineappl_capi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `pineappl_capi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'pineappl_capi' found
    g++ -std=c++11 -O -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /store/DAMTP/mnc33/miniconda3/envs/pinefarm/include -fdebug-prefix-map=/usr/share/miniconda/envs/test/conda-bld/lhapdf_1632848490271/work=/usr/local/src/conda/lhapdf-6.4.0 -fdebug-prefix-map=/store/DAMTP/mnc33/miniconda3/envs/pinefarm=/usr/local/src/conda-prefix -I/store/DAMTP/mnc33/miniconda3/envs/pinefarm/include -c -I. pineappl_interface.cc
    pineappl_interface.cc:14:10: fatal error: pineappl_capi.h: No such file or directory
       14 | #include <pineappl_capi.h>
          | ^~~~~~~~~~~~~~~~~
    compilation terminated.
    make: *** [makefile:95: pineappl_interface.o] Error 1

[felixhekhorn]

oookay, so this sounds back like a pinefarm problem (and not an mg5 one) ...

• when you hit run for the first time all necessary tools should indeed install themselves (and if they are already present, the script will tell you)
• in your case that should be pineappl (including the CAPI) and mg5 - did you hit those installers? (from here to there )
• I think those installers print to stdout - can you check whether you have cargo cinstall ... there somewhere (this one)? that should install the CAPI
• is the CAPI actually present in your current env? i.e. does the file pineappl_capi.h exist somewhere? like-wise does the command pineappl-config exist somewhere? (I'm not sure at the moment how mg5 is receiving that path ...)

[comane]

is the CAPI actually present in your current env? i.e. does the file pineappl_capi.h

No this file doesn't exist anywhere in my env. I am not even sure it is supposed to be there since I am not installing in dev mode.

does the command pineappl-config exist somewhere? (I'm not sure at the moment how mg5 is receiving that path ...)

I can't find it in my conda environment bin. This might be the problem since mg5 crashes like so:

Error detected in "launch auto " write debug file /store/DAMTP/mnc33/Projects_store/PhD/nnpdf40_pheno/pinefarm_runs/results/200-ATLAS_TTB_13TEV_TOT--20231116151158/ATLAS_TTB_13TEV_TOT/run_01_tag_1_debug.log If you need help with this issue, please, contact us on https://answers.launchpad.net/mg5amcnlo str : No valid pineappl installation found. Please set the path to pineappl-config by using MG5_aMC> set <absolute-path-to-pineappl>/bin/pineappl-config

[felixhekhorn]

No this file doesn't exist anywhere in my env. I am not even sure it is supposed to be there since I am not installing in dev mode.

you mean dev mode for pinefarm? that does not matter - since mg5 will need it in any case (since it is compiling on its own) and indeed it should be installed in any case; the commands I posted above ARE pinefarm, so pinefarm can and (hopefully) will install programs on its own; and here for mg5 pinefarm should realize that he needs the CAPI and so provide the CAPI

[comane]

Regarding this:

• I think those installers print to stdout - can you check whether you have cargo cinstall ... there somewhere (this one)? that should install the CAPI

I do not have cinstall:

cargo cinstall
error: no such command: `cinstall`

    Did you mean `install`?

    View all installed commands with `cargo --list`

[felixhekhorn]

This sounds like you really not went through the install.pineappl routine ...

Can you please make a new env and print the full output of pinefarm run runcards/ATLAS_TTB_13TEV_TOT theory_200_1.yaml ? specifically, I'm interested in the install outputs this call should trigger ...

[comane]

@felixhekhorn , I had to cancel most of the fortran error output, as the comment would have not fitted in the page

[felixhekhorn]

also a text file would have been fine :rofl: (you can still edit your post :innocent: )

now the relevant lines are these ones:

Could not find directory of OpenSSL installation, and this -sys crate cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it, you can set the OPENSSL_DIR environment variable for the
compilation process.

Make sure you also have the development packages of openssl installed.
For example, libssl-dev on Ubuntu or openssl-devel on Fedora.

and so cargo fails, i.e. pineappl fails and so you run into the problem - can you please try to apply the suggested fix?

[comane]

Here is the .txt file pinefarm_log.txt

[comane]

@felixhekhorn thank you so much for the assistance.

So after installing openssl it works!

I had to do the following:

wget https://www.openssl.org/source/openssl-1.1.1k.tar.gz
tar -zxvf openssl-1.1.1k.tar.gz
cd openssl-1.1.1k

./config --prefix=$HOME/usr --openssldir=$HOME/usr/openssl
make
make install

Add folder containing openssl.pc (libssl.pc) to PKG_CONFIG_PATH

the run generates the pineappl grid, however I still get an error at the end of the run:

-------------------------------------------------------------------------
bin   PineAPPL         MC        sigma      central         min      max
                                 1/100   sigma   1/1000   1/1000   1/1000
-------------------------------------------------------------------------
0  7.888729e+02  7.887245e+02   0.105   0.179   0.1882   0.1876   0.1935
Traceback (most recent call last):
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/bin/pinefarm", line 8, in <module>
    sys.exit(command())
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/cli/run.py", line 31, in subcommand
    main(dataset, theory_card, pdf)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/cli/run.py", line 67, in main
    run_dataset(runner)
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/cli/run.py", line 132, in run_dataset
    runner.annotate_versions()
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/external/interface.py", line 121, in annotate_versions
    versions = self.collect_versions()
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/site-packages/pinefarm/external/mg5/__init__.py", line 235, in collect_versions
    subprocess.run(
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/store/DAMTP/mnc33/miniconda3/envs/new_pinefarm_env/lib/python3.9/subprocess.py", line 1837, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'brz'
Thanks for using LHAPDF 6.4.0. Please make sure to cite the paper:
  Eur.Phys.J. C75 (2015) 3, 132  (http://arxiv.org/abs/1412.7420)

[felixhekhorn]

good we were able to solve something! :+1:

how did you solve the brz issue? (that should have been installed together with mg5 ...)

remember to close the mg5 bug (and say it actually was not a bug)