search

eclipse-cyclonedds / cyclonedds-python

Other
54 stars 44 forks source link

Cannot run the cyclonedds idlc python plugin built from source #203

Closed simon-at-spire closed 11 months ago

simon-at-spire commented 11 months ago

Hi, after building both cyclonedds 0.10.2 and cyclonedds-python 0.10.2 from source (0.10.2 git tags), I could not use the idlc compiler to generate python code for my IDL definitions. I would like to know if that bug is known (apparently it is since there was a mitigation in the wheel file published to pypi) and if it will be fixed at some point?

The error I got:

$ idlc -l py message.idl
Cannot load generator [...]/lib/python3.8/site-packages/cyclonedds-0.10.2-py3.8-linux-x86_64.egg/cyclonedds/_idlpy.py:
[...]/lib/python3.8/site-packages/cyclonedds-0.10.2-py3.8-linux-x86_64.egg/cyclonedds/_idlpy.py:
invalid ELF header
idlc: cannot load generator py

It looks like your __idlc__.py used to find the shared object point to any file in the python package starting with _idlpy. But in the case of a source installation, this is a python script (_idlpy.py).

This file is supposed to find the shared object and load it apparently, but it looks like the .so file is incomplete?

$ python3 [...]/lib/python3.8/site-packages/cyclonedds-0.10.2-py3.8-linux-x86_64.egg/cyclonedds/_idlpy.py
Traceback (most recent call last):
  File "[...]/lib/python3.8/site-packages/cyclonedds-0.10.2-py3.8-linux-x86_64.egg/cyclonedds/_idlpy.py", line 9, in <module>
    __bootstrap__()
  File "[...]/lib/python3.8/site-packages/cyclonedds-0.10.2-py3.8-linux-x86_64.egg/cyclonedds/_idlpy.py", line 7, in __bootstrap__
    mod = importlib.util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: dynamic module does not define module export function (PyInit___main__)

I see that in the wheel that you posted on Pypi you used a workaround which was to remove _idlpy.py and put the .so file there directly. That way I guess that __idlc__.py point directly to that file:

wheel/cyclonedds/ $ ls _idlpy*
_idlpy.cpython-38-x86_64-linux-gnu.so*

I checked and when I pointed idlc directly to the plugin in the -l it worked (I saw that workaround in the idlc source code):

$ idlc -l[...]/lib/python3.8/site-packages/cyclonedds-0.10.2-py3.8-linux-x86_64.egg/cyclonedds/_idlpy.cpython-38-x86_64-linux-gnu.so message.idl

What worries me is that the wheel has some apparently non-standard versions of the compiler and ddsc libraries. Can I use the binaries that I generated myself? Or are there some specific commits that are supported? Or did I just not understand how this is supposed to work?

eboasson commented 11 months ago

I think you understand better than I do how the Python-specific details work. I had run into this myself in some form or other, assumed it had something to do with macOS-specific problems and used the same workaround so I could continue what I was trying to do.

If you take everything from the 0.10.x release branches, it should all fit. The same is true for master. There have recently been some changes in the internal interfaces between the IDL compiler's front- and back-ends, and also in the internal "type support" interfaces (named sertype/serdata in Cyclone for historical reasons). So mixing 0.10.x and master will fail, but really should be a hard fail, at compilation time. I wouldn't worry about that.

simon-at-spire commented 11 months ago

Ok, thank you, then it should be alright.