Complex template metaprogramming succeeds via `cppyy.cppdef`, but not with Python bindings

[taehyounpark]

• ROOT version: 6.28.06 (cppyy version: 1.6.2)

Dear experts,

I would like to report a peculiarity with C++ function calls performed from cppyy's Python-bindings versus its lower-level cppyy.cppdef() methods. I have been developing a high-level data analysis framework that, which heavily uses template metaprogramming. It was primarily developed to be used with the CERN ROOT library, and a working example (from which the issue below can be reproduced) can be found here.

The library compiles and fully runs in both clang14.0 and gcc11.2. A problem occurs, however, when trying to call the interface from Python through ROOT's supplied cppyy bindings.

Working code snippet (C++):

import cppyy
import ROOT
from ROOT import ana

df = cppyy.gbl.ana.dataflow['Tree']([],'')

cppyy.cppdef('''
  auto df = ana::dataflow<Tree>({"hww.root"},"mini");
''')

cppyy.cppdef('''
  auto runNumber = df.read<int>("runNumber");
  auto plusOne = df.define([](int x){return x+1;});
  auto runNumberPlusOne = plusOne(runNumber);
''')

print(cppyy.gbl.runNumber)
print(cppyy.gbl.plusOne)
print(cppyy.gbl.runNumberPlusOne)

# output
# <cppyy.gbl.ana.dataflow<Tree>.dataflow<Tree>::lazy<Tree::Branch<int> > object at 0x7f82d4d250e8>
# <cppyy.gbl.ana.dataflow<Tree>.dataflow<Tree>::delayed<ana::column::evaluator<ana::column::equation<int(int)> > > object at 0x7f82d4d25120>
# <cppyy.gbl.ana.dataflow<Tree>.dataflow<Tree>::lazy<ana::column::equation<int(int)> > object at 0x7f82d4d25158>

This code snippet, whether it's compiled in C++ or run from Python as above, works with the correct output.

Same, but problematic code (Python):

import cppyy
import ROOT
from ROOT import ana

df = ana.dataflow['Tree'](['hww.root'],'mini')

runNumber = df.read['int']('runNumber')

cppyy.cppdef('''auto plusOne = std::function([](int x){return x+1;});''')
print(cppyy.gbl.plusOne)

plusOne = df.define(cppyy.gbl.plusOne)
print(plusOne)

runNumberPlusOne = plusOne.evaluate(runNumber)
print(runNumberPlusOne)

# output
# <cppyy.gbl.std.function<int(int)> object at 0x7fc7e9b6d028>
# <cppyy.gbl.ana.dataflow<Tree>.dataflow<Tree>::delayed<ana::column::evaluator<ana::column::equation<int> > > object at 0x10ff6510>

As you can notice, the type of the second object is incorrect, as it is missing the argument types (int instead of int(int)), which leads to the error in trying to perform the third call (quoted below). So currently, this library would be limited to strictly use within C++ without manually reproducing essentially the entire API in Python with cppdef. I have no idea which part in cppyy is going wrong such that its cppdef call works, but not the Python binding. I understand that this is not really a minimal example and using not the latest version of cppyy, but any help or ideas to investigate the issue further would be greatly appreciated. Thank you in advance.

The error from trying to call the Python binding:

input_line_74:7:173: error: no matching member function for call to 'evaluate'
      new (ret) (ana::dataflow<Tree>::lazy<ana::column::equation<int> >) (((const ana::dataflow<Tree>::delayed<ana::column::evaluator<ana::column::equation<int> > >*)obj)->evaluate<ana::dataflow<Tree>::lazy<Tree::Branch<int> > &, ana::column::evaluator<ana::column::equation<int> >, false>((ana::dataflow<Tree>::lazy<Tree::Branch<int> >&)*(ana::dataflow<Tree>::lazy<Tree::Branch<int> >*)args[0]));
                                                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/thpark/ana/AnalysisExample/AnalysisFramework/analogical/include/ana/interface/dataflow_delayed.h:82:8: note: candidate template ignored: invalid explicitly-specified argument for template parameter 'Nodes'
  auto evaluate(Nodes &&...columns) const
       ^
input_line_74:11:112: error: no matching member function for call to 'evaluate'
      (void)(((const ana::dataflow<Tree>::delayed<ana::column::evaluator<ana::column::equation<int> > >*)obj)->evaluate<ana::dataflow<Tree>::lazy<Tree::Branch<int> > &, ana::column::evaluator<ana::column::equation<int> >, false>((ana::dataflow<Tree>::lazy<Tree::Branch<int> >&)*(ana::dataflow<Tree>::lazy<Tree::Branch<int> >*)args[0]));
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/thpark/ana/AnalysisExample/AnalysisFramework/analogical/include/ana/interface/dataflow_delayed.h:82:8: note: candidate template ignored: invalid explicitly-specified argument for template parameter 'Nodes'
  auto evaluate(Nodes &&...columns) const
       ^
input_line_75:7:181: error: no matching member function for call to 'evaluate'
      new (ret) (ana::dataflow<Tree>::lazy<ana::column::equation<int> >::varied) (((const ana::dataflow<Tree>::delayed<ana::column::evaluator<ana::column::equation<int> > >*)obj)->evaluate<ana::dataflow<Tree>::lazy<Tree::Branch<int> > *, ana::column::evaluator<ana::column::equation<int> >, false>((ana::dataflow<Tree>::lazy<Tree::Branch<int> >*&&)*(ana::dataflow<Tree>::lazy<Tree::Branch<int> >**)args[0]));
                                                                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/thpark/ana/AnalysisExample/AnalysisFramework/analogical/include/ana/interface/dataflow_delayed.h:82:8: note: candidate template ignored: invalid explicitly-specified argument for template parameter 'Nodes'
  auto evaluate(Nodes &&...columns) const
       ^
input_line_75:11:112: error: no matching member function for call to 'evaluate'
      (void)(((const ana::dataflow<Tree>::delayed<ana::column::evaluator<ana::column::equation<int> > >*)obj)->evaluate<ana::dataflow<Tree>::lazy<Tree::Branch<int> > *, ana::column::evaluator<ana::column::equation<int> >, false>((ana::dataflow<Tree>::lazy<Tree::Branch<int> >*&&)*(ana::dataflow<Tree>::lazy<Tree::Branch<int> >**)args[0]));
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/thpark/ana/AnalysisExample/AnalysisFramework/analogical/include/ana/interface/dataflow_delayed.h:82:8: note: candidate template ignored: invalid explicitly-specified argument for template parameter 'Nodes'
  auto evaluate(Nodes &&...columns) const
       ^
Traceback (most recent call last):
  File "/home/thpark/ana/AnalysisExample/run/../AnalysisFramework/AnalysisPlugins/examples/cppyy_pycall.py", line 15, in <module>
    runNumberPlusOne = plusOne.evaluate(runNumber)
ValueError: Template method resolution failed:
  ana::dataflow<Tree>::lazy<ana::column::equation<int> > ana::dataflow<Tree>::delayed<ana::column::evaluator<ana::column::equation<int> > >::evaluate(ana::dataflow<Tree>::lazy<Tree::Branch<int> >& columns) =>
    ValueError: nullptr result where temporary expected
  ana::dataflow<Tree>::lazy<ana::column::equation<int> >::varied ana::dataflow<Tree>::delayed<ana::column::evaluator<ana::column::equation<int> > >::evaluate(ana::dataflow<Tree>::lazy<Tree::Branch<int> >*&& columns) =>
    ValueError: nullptr result where temporary expected
  ana::dataflow<Tree>::lazy<ana::column::equation<int> > ana::dataflow<Tree>::delayed<ana::column::evaluator<ana::column::equation<int> > >::evaluate(ana::dataflow<Tree>::lazy<Tree::Branch<int> >&& columns) =>
    ValueError: could not convert argument 1 (object is not an rvalue)

[wlav]

For ROOT support, please ask the ROOT team. The cppyy version that ROOT uses is their local fork, it is positively ancient and has been modified for ROOT's use.

[wlav]

Closing b/c wrong repo.