Writing bindings for another language

[ljulliar]

I'm thinking about writing bindings for another language and my first step is to understand how Python bindings work. From what I see in bindings/py/cpp_src/ the pybind11 C++ library is used to build a Python compatible bindings from the C++ side.

Then if I take a look at, say, bindings/py/tests/encoders/rdse_test.py I see

from htm.bindings.sdr import SDR, Metrics
from htm.bindings.encoders import RDSE, RDSE_Parameters

and I'm not quite sure where the htm.bindings.sdr and htm.bindings.encoders python modules are created in the first place.

Can someone help me understand the logic of the flow here ?

Thank you everyone!

[breznak]

I'm thinking about writing bindings for another language

Cool! What language teases your heart? Btw, @dkeeney has also a long-term goal of writing bindings for C#.

and I'm not quite sure where the htm.bindings.sdr and htm.bindings.encoders python modules are created in the first place.

yes, we're a C++ library providing HTM. And pybind11 is used to make python bindings. It works by creating a wrapper which acts as python code but deep down calls the c++.

You can see examples in /bindings/py/cpp_src/bindings/ , pybind11 has some syntax how c++ methods are mapped to python. Then during setup.py we install the python (bindings) code which is located in htm.bindings.*

• Note, unlike swig I think pybind11 acts as a binding only for python (as the name suggests).

• And an organizational thing, @dkeeney @ctrl-z-9000-times @ljulliar , we'll have to discuss our stance on new bindings.

  • Personally I'd prefer a separate repo, either common htm.core.bindings or specific htm.core.<lang>. The motivation is more flexibility for both sides. We do already provide bindings for python and those will stay in the repo, but that is based on our survey, where cca 50% users cared about c++, and 50% about python.
  • other approach would be the new lang included in the repo, but "optionally". Meaning tests are not required to pass with each new PR to the (c++,py). That way, the lang-binding can be part of the repo, compiled and tested using our infrastructure. But like NAB will be bound to a specific version in git. Later versions may break unless someone again updates the lang. Thoughts?

[ljulliar]

@breznak I see the pros and cons of each scenario. Still to compile the binding (e.g. using swig) you need to get access to the C++ header files from htm.core and compile time and you need libhtmcore.so (or whatever the name is...) at runtime.

Since there are no binary distribution of HTM Core for now (in the form of debian packages for instance) it means that anyone willing to use a particular language binding will have to compile htm core any way, right ?

So there might be some benefits in having separate repositories for language bindings from the readability and decoupling standpoint but HTM core users will have to deal with both repositories anyway.

[dkeeney]

Since we have the python bindings in htm.core we should also include the other languages in there as well, but that is just my opinion.

I want to discourage the use of SWIG because it was so difficult to use and maintain...that is why we switched to pybind11.

There is another complication.. the C++ library must be able to call out into the bound language in order to implement NetworkAPI Regions that are written in that other language. I may be able to help with that if needed.

Which language did you have in mind?

[ljulliar]

@dkeeney, I had Ruby in mind. I won't be using Swig but Rice which is very much like pybind11 in spirit.

As for C++ calling in Python I realized that when I browsing the code in bindings/cpp_src/plugins . I'm not sure how to handle that properly actually... Just out of curiosity, why has this particular mechanism been put in place for the NetworkAPI ??

Thanks!

On Tue, Aug 6, 2019 at 6:22 PM David Keeney notifications@github.com wrote:

Since we have the python bindings in htm.core we should also include the other languages in there as well, but that is just my opinion.

I want to discourage the use of SWIG because it was so difficult to use and maintain...that is why we switched to pybind11.

There is another complication.. the C++ library must be able to call out into the bound language in order to implement NetworkAPI Regions that are written in that other language. I may be able to help with that if needed.

Which language did you have in mind?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/htm-community/htm.core/issues/606?email_source=notifications&email_token=AAAED4NPW4M6VXGBLIMCEYDQDGQKVA5CNFSM4IJEVD5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3VV77A#issuecomment-518742012, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAED4LCULLMAGJSEZNBHDDQDGQKVANCNFSM4IJEVD5A .

[dkeeney]

Just out of curiosity, why has this particular mechanism been put in place for the NetworkAPI ??

It has to do with the way the engine works to orchestrate the flow of data and execution of regions..

After setting up your Network object by configuring it with regions and links you would call the run( ) function on the Network object. Within this run( ) function it will setup the calling arguments and call the compute( ) function on each region implementation for each iteration. So this is C++ code calling out to the language that implemented the region...could be Python or C++. It is a callback type of thing.

This complicates the creation of region implementations but it makes it easy to create apps that use them, particularly if the regions are all built-in regions.

[dkeeney]

I had Ruby in mind.

That is cool. I would expect that Ruby had some facility to implement callbacks.

Another thing to consider is that if a region implementation is written in Python and you want to use it from Ruby, we are going to need a way to register the python region implementations.

This is the same problem I would have if I wanted to write an app in C++ and wanted to use a region that was written in Python. It is not really a callback but a pure calling into the Python library. I think there is a way to do this.

[breznak]

it means that anyone willing to use a particular language binding will have to compile htm core any way, right ?

yes, currently everyone needs to compile, but we're heading in the direction of binary distribution (not distro packages) - tar.gz for each platform and pip(y) #361

Since we have the python bindings in htm.core we should also include the other languages in there as well, but that is just my opinion.

we would discuss this when needed. I'd for now strongly prefer other (=not extremely popular) bindings moved to other repos. My reasoning is the maintanance burden on the main repo (us). Even now we have to know python and pybind11 syntax, and sometimes just the cosmetic changes to pybind (that are necessary to complete the new feature) require quite some work. We made python an exceptional position because based on "the User Survey" 50% users wanted pure-c++, and 50% python bindings.

That is not to say I want to belittle a new (Ruby is cool!) binding! Just I think the other "markets" should be kept separate. We can, and would cooperate as closely as possible.