"Syntax error at '\n'" on a valid C macro.

[GoogleCodeExporter]

Hi,

The C parser in ctypesgen (version 0.r125) gives a syntax error on some header 
files that GCC accepts. Example output:

    Error: /usr/include/libxml2/libxml/xmlstring.h:35: Syntax error at '\n'

This message comes from the ctypesgencore.parser.cgrammar.p_error function.

To reproduce, create a file named "test.h" containing:

    typedef unsigned char xmlChar;
    #define BAD_CAST (xmlChar *)

(This is extracted from xmlstring.h)
Then, run: ctypesgen.py -o test.py test.h

This error interrupts everything and test.py ends up with no content generated 
from header files. (It has no mention of xmlChar, BAD_CAST, or anything else.)

It seems that the parser sees this #define as a macro with a xmlChar parameter 
and a missing definition is missing, while it is really a parameter-less macro.

ctypesgencore.parser.yacc.Parser.parse does something like this:

    token = actions.get((statestack[-1], lookahead.type), None)

When the error is triggered, statestack[-1] is 293 and lookahead.type is 
PP_END_DEFINE. I guess that the fix should go somewhere in the cgrammar module, 
but I had no idea where to go from there.

Original issue reported on code.google.com by simon.sa...@kozea.fr on 3 Dec 2011 at 5:29

[GoogleCodeExporter]

I confirm this, but I can't see why it should go wrong, as as far as I can see, 
the line given should match the rule

              | PP_DEFINE PP_DEFINE_NAME type_name PP_END_DEFINE

but looking at a logging run, it seems to parse as

              PP_DEFINE PP_DEFINE_NAME '(' type_name ')' PP_END_DEFINE

and then fail when it gets to the newline. But adding a rule exactly as above 
doesn't improve things.

I'm not an expert with Python yacc, so my confusion is probably down to simply 
not understanding what's going on very well.

Original comment by rrt@sc3d.org on 5 Dec 2011 at 10:23

[GoogleCodeExporter]

I am not familiar with yacc in any language. But maybe what is going on is that 
macros definitions can really have any token list, not just a single type name 
or constant expression.

I’ve seen stuff like this:
#define BEGIN {
#define END }

… this and much worse has been done before.

Original comment by simon.sa...@kozea.fr on 5 Dec 2011 at 10:47

[GoogleCodeExporter]

Sure, there are no rules to parse such things in ctypesgen. But there DOES 
appear to be a rule to parse the example that fails in this issue.

Original comment by rrt@sc3d.org on 5 Dec 2011 at 10:49

[GoogleCodeExporter]

What I ended up doing is commenting the offending macro for the purpose of 
ctypesgen. I really did not need it, it just happened to be in one of the 
dependencies of the project I was interested in.

I have no clue about the actual problem reported here, but maybe it can be 
worked around. Would it be possible to have the parser not panic when it finds 
something invalid, ignore it and continue?

If this is difficult, at a higher level: on a syntax error, ignore the rest of 
the file (or even the whole file) but do not abort the whole project and 
continue with other files.

The current behavior writes a valid .py file but one that exports no symbol at 
all, for the whole project.

For the reference, here is the command I was running. (On a linux system with 
libcroco installed.)

ctypesgen.py \
    `pkg-config --cflags libcroco-0.6` `pkg-config --libs libcroco-0.6` \
    -o libcroco_.py /usr/include/libcroco-0.6/libcroco/*.h

Original comment by simon.sa...@kozea.fr on 5 Dec 2011 at 10:59

[GoogleCodeExporter]

Gah, this whole thing is a red herring.

First, the example given above does NOT match the rule I said it should match.

Secondly, ctypesgen works FINE here. The problem is nothing to do with the 
syntax error: it's that when parsing libxml headers, you need to add -I flags 
so that ctypesgen can find other libxml header files. The command line you give 
works fine on my Ubuntu system, but if I just try to wrap the problematic 
header file on its own, I get the error:

$ ctypesgen.py -o test.py /usr/include/libxml2/libxml/xmlstring.h 
...

Error: gcc -E: /usr/include/libxml2/libxml/xmlstring.h:15:31: fatal error: 
libxml/xmlversion.h: No such file or directory

If I add a suitable -I flag, then it works:

$ ctypesgen.py -I/usr/include/libxml2 -o test.py 
/usr/include/libxml2/libxml/xmlstring.h 
Warning: No libraries specified
Status: Preprocessing /tmp/tmpJy8XUm.h
Status: gcc -E -U __GNUC__ -dD -I/usr/include/libxml2 "-Dinline=" 
"-D__inline__=" "-D__extension__=" "-D_Bool=uint8_t" "-D__const=const" 
"-D__asm__(x)=" "-D__asm(x)=" "-DCTYPESGEN=1" /tmp/tmpJy8XUm.h
Status: Parsing /tmp/tmpJy8XUm.h
Error: /usr/include/libxml2/libxml/xmlstring.h:35: Syntax error at '\n'
Status: Processing description list.
Warning: Could not parse macro "#define BAD_CAST ( xmlChar * )"
Status: Writing to test.py.
Status: Wrapping complete.

and test.py contains suitable wrappings.

The fact that the command you gave worked for me and not for you suggests that 
pkg-config is not giving you the correct -I flags for some reason. But in any 
case, I don't think it has anything to do with the syntax error.

Original comment by rrt@sc3d.org on 8 Dec 2011 at 2:13

[GoogleCodeExporter]

Hi,

Indeed, I just ran again the command I gave above and got all libcroco symbols 
in spite of the syntax error in xmlstring.h, but this is a different machine. 
(Same distribution and versions, though.)

I’ll try again on my home machine where I had this bug, but I seem to recall 
that pkg-config gave the right -I flags, including /usr/include/libxml2. I’ll 
give an update in any case.

Original comment by simon.sa...@kozea.fr on 8 Dec 2011 at 2:29

[GoogleCodeExporter]

Note also that with the 2-line example above, ctypesgen indeed generates a 
binding for the typedef.

Original comment by rrt@sc3d.org on 8 Dec 2011 at 2:43

[GoogleCodeExporter]

Now on the same machine where I initially had this bug, I can not reproduce it. 
I get bindings for libcroco or for the typedef in the 2-line examples in spite 
of the syntax error.

I distinctly remember ctypesgen generating the boilerplate .py with zero 
bindings in both cases, and the problem going away when I commented the 
offending macro. But this is not the case anymore. Since then my system has had 
a micro-version update of gcc, but I don’t know if it is related.

Anyway, please close the bug as invalid.

Original comment by simon.sa...@kozea.fr on 8 Dec 2011 at 6:18

[GoogleCodeExporter]

Closing.

Original comment by rrt@sc3d.org on 8 Dec 2011 at 6:22

• Changed state: Invalid