Conversion from python `dict` into `nlohmann::json`

[qazi0]

Is it possible to pass a python dict into a cppyy-loaded C++ function expecting a nlohmann::json (nlohmann/json) object? This question has to have come up by now, but I wasn't able to find anything on it.

Minimal example to reproduce:

test-json.h

#include <iostream>
#include <nlohmann/json.hpp>

using nlohmann::json;

void print_name_and_age(json j) {
    std::cout << j["name"] << "\n"
              << j["age"] << "\n";
} 

test-cppyy.py

import cppyy
cppyy.include('test-json.h')

from cppyy.gbl import print_name_and_age

some_dict = {
    "name": "alfred",
    "age": 25
}

print_name_and_age(some_dict)

runs into

    print_name_and_age(some_dict)
NotImplementedError: void ::print_name_and_age(nlohmann::json j) =>
    NotImplementedError: could not convert argument 1 (this method cannot (yet) be called)

I would like to be able to pass a python dict into the C++ function, and receive it as a nlohmann::json object. I presume I would need to write some custom converter for this?

Design requirement/background (optional)

I have a reinforcement learning environment class (written in C++) that needs to accept some configuration to initialize it (in its constructor). Everything's all fine passing a nlohmann::json object into the constructor while in the C++ domain, but I have a Python wrapper around the class too, written with cppyy that provides similar functionality to the C++ interface. Uptill now, because of the aforementioned issue, I've been forced to receive a const std::map<std::string, float>& in the constructor instead of a nlohmann::json, which is what a python dict containing only str -> float mappings easily gets converted to by cppyy. But this obviously limits my input json files to only contain floats as values (my usecase requires having strings as keys but strings, ints, floats and bools as values in the JSON file). I can ofcourse write some pre-processing code to encode my heterogenous python dict into a homogenous str->float mapping on the python front (and do the same for C++) but I'd like a cleaner solution, if possible.

@wlav could you please give me some pointers on passing a python dict into the cppyy-imported C++ function and have it converted into a nlohmann::json object in the C++ function? If this requires forking cppyy to add extra converter code/too much trouble I presume I would need to use a std::map<std::string, std::any / variant> alternative? I haven't worked alot with std::any/variant, would like to ask if this would even be possible - python dict to map<string, any> - if this is the best alternative to a custom converter - in terms of performance / clean elegant code.

Environment:

Python 3.9.13

C++17

cppyy==2.3.1
cppyy-backend==1.14.8
cppyy-cling==6.25.3
cppyythonizations==1.2.1

MacOS Monterey, Apple M1

Stack Overflow

[wlav]

Saw that on stackoverflow, but am inundated with a conference atm. :)

In short, there is no automatic conversion as the connection between the types isn't known (although maybe both could be recognized as mapping objects, which could allow for some general conversion).

Yes, you can write custom converters/executors, but I've never documented it b/c such code would need to unwrap the Python object and apply whatever conversion is needed and is thus not PyPy friendly. I guess I should bite the bullet and accept it, though, esp. since the Python C-API is well supported by PyPy these days.

The examples that exist, are in the tests: https://github.com/wlav/cppyy/blob/master/test/test_api.py#L65

[qazi0]

Okay, makes sense. I'll try to look at the test code and make my way around it. Thanks alot!

[amishgujarathi]

@Siraj-Qazi - were you able to find some way around passing python dict to C++ as nlohmann::json using cppyy? I have a similar requirement to hence checking

[qazi0]

@amishgujarathi I decided to go another route instead, as like @wlav said this would require adding the custom converter code into cppyy codebase, and so I'd need to have and maintain a separate fork of it, which would affect my portability constraints. I ended up passing a simple filename str from python into C++, and then managing the resource within C++ (loading the json file and parsing it). Personally I'd say it would be amazing if there was native support for adding any type's custom converter to cppyy via its API, its a nice project idea but I'm not too well versed in llvm/clang...

Also, incase you endup doing what I did (using nlohmann::json in your C++ code only), you may come across #97 while trying to import a snippet using nlohmann::json into cppyy, so look out for that.

[amishgujarathi]

@Siraj-Qazi thanks a lot for the information, will take a look