search

elijahr / python-autopxd2

A friendly fork of autopxd
MIT License
46 stars 9 forks source link

Crash on struct with array of sizeof(...) size #15

Open touilleMan opened 5 years ago

touilleMan commented 5 years ago

Consedering

#define SIZE sizeof(int)

typedef struct {
    char _internal_stuff[SIZE];
} s_foo;

Output

$ autopxd python-autopxd2/test.h 
Traceback (most recent call last):
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/bin/autopxd", line 11, in <module>
    load_entry_point('autopxd2', 'console_scripts', 'autopxd')()
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/__init__.py", line 94, in cli
    outfile.write(translate(infile.read(), infile.name, extra_cpp_args))
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/__init__.py", line 73, in translate
    p.visit(parse(code, extra_cpp_args=extra_cpp_args, whitelist=whitelist))
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 34, in visit
    rv = super(AutoPxd, self).visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 158, in visit
    return visitor(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 165, in generic_visit
    self.visit(c)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 34, in visit
    rv = super(AutoPxd, self).visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 158, in visit
    return visitor(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 180, in visit_Typedef
    decls = self.collect(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 194, in collect
    self.generic_visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 165, in generic_visit
    self.visit(c)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 34, in visit
    rv = super(AutoPxd, self).visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 158, in visit
    return visitor(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 118, in visit_TypeDecl
    decls = self.collect(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 194, in collect
    self.generic_visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 165, in generic_visit
    self.visit(c)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 34, in visit
    rv = super(AutoPxd, self).visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 158, in visit
    return visitor(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 112, in visit_Struct
    return self.visit_Block(node, 'struct')
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 59, in visit_Block
    fields = self.collect(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 194, in collect
    self.generic_visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 165, in generic_visit
    self.visit(c)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 34, in visit
    rv = super(AutoPxd, self).visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 158, in visit
    return visitor(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 129, in visit_Decl
    decls = self.collect(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 194, in collect
    self.generic_visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 165, in generic_visit
    self.visit(c)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 34, in visit
    rv = super(AutoPxd, self).visit(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/venv/lib/python3.6/site-packages/pycparser/c_ast.py", line 158, in visit
    return visitor(node)
  File "/home/emmanuel/projects/godot-python-2-electric-boogaloo/python-autopxd2/autopxd/writer.py", line 175, in visit_ArrayDecl
    assert len(decls) == 1
AssertionError
chrisjbillington commented 5 years ago

Does the same problem occur on the git version of autopxd2 and the latest released version?

I should be able to fix this I imagine, but I don't have a lot of time at the moment.

touilleMan commented 5 years ago

Does the same problem occur on the git version of autopxd2 and the latest released version?

Yep, does the same thing on the github master branch

I should be able to fix this I imagine, but I don't have a lot of time at the moment.

No worries, I've commented the assert which allowed me to generate a .pxd which was roughly good (I had to be tune it by hand a bit)

chrisjbillington commented 5 years ago

It turns out that the minimal example to trigger this issue is:

char _internal_stuff[sizeof(int)];

Because autopxd(2) can't handle anything other than a constant in the array dims, as far as I can tell.

Then again, cython does not accept:

cdef extern from 'test.h':
    char _internal_stuff[sizeof(int)]

So I'm not sure what it is desirable for autopxd(2) to produce for such code. What kind of output would you say autopxd(2) should produce in order to declare the bug 'fixed'? It kind of sounds like there is a disparity here between what is possible in C and what is possible in Cython.

Cython would happily accept

cdef extern from 'test.h':
    char _internal_stuff[]

not knowing the size of the array, so perhaps that would be fine (this is what you get just commenting out the assert) for the minimal example, but Cython is not happy with not knowing the size of the array when it is a struct member:

cdef extern from "test.h":
    struct s_foo:
        char _internal_stuff[]
Error compiling Cython file:
------------------------------------------------------------
...
cdef extern from "test.h":
    struct s_foo:
        char _internal_stuff[]                           ^
------------------------------------------------------------

test.test:3:28: Variable type 'char []' is incomplete

I'm not sure what to put in there though since sizeof(int) is not known until compile-time and Cython won't let us just put sizeof(int) in there. Perhaps this could be considered a bug or deficiency in Cython.

In what way did you 'fix up' the resulting .pxd file?

chrisjbillington commented 5 years ago

Looks like a previously reported issue in Cython:

https://mail.python.org/pipermail/cython-devel/2013-August/003787.html

Though that was in 2013, so it looks like it hasn't gotten any attention. If this issue were fixed in Cython then we could say that autopxd(2) should just reproduce the "sizeof" call inside the array dims - though that would be nontrivial since there can be arbitrary expressions in there, and autopxd is not so far in the business of translating arbitrary C expressions into Cython. So this is a potentially thorny issue.