Unions nested inside structures are not parsed

RduMarais commented 4 years ago

Hello everyone! Thank you for maintaining this library, and fixing various bugs with the original library. I am trying to parse headers with a anonymous union defined inside a struct :

typedef struct
{
  volatile union
  {
    volatile uint8_t u8;
    volatile uint16_t u16;
    volatile uint32_t u32;
  } PORT [32];
       uint32_t RESERVED0[864];
  volatile uint32_t TER;
   .....
} ITM_Type;

However, this union is not parsed as such, and i have no way of isolating the data allocated inside the union from the rest of the structure. Output, printed as another C header gives :

typedef struct ITM_Type
{
  volatile uint8_t u8;
  volatile uint16_t u16;
  volatile uint32_t u32;
  PORT ;
  uint32_t RESERVED0;
  volatile uint32_t TER;
   .....
} ITM_Type;

As you can see the resulting memory allocated is not correct.

The output in json is :

{
    "headerFileName": "report_issue/input_union.h",
    "curClass": "",
    "classes": {
        "ITM_Type": {
            "nested_classes": [],
            "abstract": false,
            "final": false,
            "namespace": "",
            "using": {},
            "name": "ITM_Type",
            "bare_name": "<anon-struct-1>",
            "inherits": [],
            "filename": "report_issue/input_union.h",
            "line_number": 2,
            "methods": {
                "public": [],
                "protected": [],
                "private": []
            },
            "properties": {
                "public": [
                    {
                        "extern": false,
                        "array": 0,
                        "filename": "report_issue/input_union.h",
                        "line_number": 6,
                        "function_pointer": 0,
                        "type": "volatile uint8_t",
                        "name": "u8",
                        "aliases": [
                            "volatile"
                        ],
                        "typedef": null,
                        "constant": 0,
                        "constexpr": 0,
                        "reference": 0,
                        "pointer": 0,
                        "static": 0,
                        "typedefs": 0,
                        "class": 0,
                        "fundamental": 0,
                        "unresolved": true,
                        "namespace": "",
                        "property_of_class": "<anon-struct-1>",
                        "mutable": false,
                        "raw_type": "volatile uint8_t"
                    },
                    {
                        "extern": false,
                        "array": 0,
                        "filename": "report_issue/input_union.h",
                        "line_number": 7,
                        "function_pointer": 0,
                        "type": "volatile uint16_t",
                        "name": "u16",
                        "aliases": [
                            "volatile"
                        ],
                        "typedef": null,
                        "constant": 0,
                        "constexpr": 0,
                        "reference": 0,
                        "pointer": 0,
                        "static": 0,
                        "typedefs": 0,
                        "class": 0,
                        "fundamental": 0,
                        "unresolved": true,
                        "namespace": "",
                        "property_of_class": "<anon-struct-1>",
                        "mutable": false,
                        "raw_type": "volatile uint16_t"
                    },
                    {
                        "extern": false,
                        "array": 0,
                        "filename": "report_issue/input_union.h",
                        "line_number": 8,
                        "function_pointer": 0,
                        "type": "volatile uint32_t",
                        "name": "u32",
                        "aliases": [
                            "volatile"
                        ],
                        "typedef": null,
                        "constant": 0,
                        "constexpr": 0,
                        "reference": 0,
                        "pointer": 0,
                        "static": 0,
                        "typedefs": 0,
                        "class": 0,
                        "fundamental": 0,
                        "unresolved": true,
                        "namespace": "",
                        "property_of_class": "<anon-struct-1>",
                        "mutable": false,
                        "raw_type": "volatile uint32_t"
                    },
                    {
                        "extern": false,
                        "array_size": "32",
                        "array": 1,
                        "filename": "report_issue/input_union.h",
                        "line_number": 9,
                        "function_pointer": 0,
                        "type": "PORT",
                        "name": "",
                        "aliases": [
                            "PORT"
                        ],
                        "typedef": null,
                        "constant": 0,
                        "constexpr": 0,
                        "reference": 0,
                        "pointer": 0,
                        "static": 0,
                        "typedefs": 0,
                        "class": 0,
                        "fundamental": 0,
                        "unresolved": true,
                        "namespace": "",
                        "property_of_class": "<anon-struct-1>",
                        "mutable": false,
                        "raw_type": "PORT"
                    },
                    {
                        "extern": false,
                        "array_size": "864",
                        "array": 1,
                        "filename": "report_issue/input_union.h",
                        "line_number": 10,
                        "function_pointer": 0,
                        "type": "uint32_t",
                        "name": "RESERVED0",
                        "aliases": [
                            "uint32_t"
                        ],
                        "typedef": null,
                        "constant": 0,
                        "constexpr": 0,
                        "reference": 0,
                        "pointer": 0,
                        "static": 0,
                        "typedefs": 0,
                        "class": 0,
                        "fundamental": 0,
                        "unresolved": true,
                        "namespace": "",
                        "property_of_class": "<anon-struct-1>",
                        "mutable": false,
                        "raw_type": "uint32_t"
                    },
                    {
                        "extern": false,
                        "array": 0,
                        "filename": "report_issue/input_union.h",
                        "line_number": 11,
                        "function_pointer": 0,
                        "type": "volatile uint32_t",
                        "name": "TER",
                        "aliases": [
                            "volatile"
                        ],
                        "typedef": null,
                        "constant": 0,
                        "constexpr": 0,
                        "reference": 0,
                        "pointer": 0,
                        "static": 0,
                        "typedefs": 0,
                        "class": 0,
                        "fundamental": 0,
                        "unresolved": true,
                        "namespace": "",
                        "property_of_class": "<anon-struct-1>",
                        "mutable": false,
                        "raw_type": "volatile uint32_t"
                    },
                ],
                "protected": [],
                "private": []
            },
            "enums": {
                "public": [],
                "protected": [],
                "private": []
            },
            "typedefs": {
                "public": [],
                "protected": [],
                "private": []
            },
            "forward_declares": {
                "public": [],
                "protected": [],
                "private": []
            },
            "declaration_method": "struct"
        }
    },
    "functions": [],
    "pragmas": [],
    "defines": [],
    "includes": [],
    "headerFileNames": [
        "report_issue/input_union.h"
    ],
    "enums": [],
    "variables": [],
    "global_enums": {},
    "typedefs": {},
    "namespaces": [],
    "using": {}
}

Thanks in advance for any help

virtuald commented 4 years ago

Definitely a bug. I don't really have an interest in fixing this myself (lots of stuff going on), but I can provide guidance for debugging/fixing this if you would like to fix it.

RduMarais commented 4 years ago

I had a lookat the parsing method, it appears that the method evaluate_class_stack expects only one keyword in the declaration (in my case it was not recognized because the union is volatile. I can try to change this behavior and use if "struct" in self.nameStack etc in :

 def _evaluate_class_stack(self):
        parent = self.curClass
        if parent:
            debug_print("found nested subclass")
            self.accessSpecifierStack.append(self.curAccessSpecifier)
        if self.nameStack[0] == "typedef":
            del self.nameStack[0]
        if len(self.nameStack) == 1:
            if self.nameStack[0] == "struct":
                self.anon_struct_counter += 1
                # We cant handle more than 1 anonymous struct, so name them uniquely
                self.nameStack.append("anon-struct-%d" % self.anon_struct_counter)
            elif self.nameStack[0] == "union":
                self.anon_union_counter += 1
                # We cant handle more than 1 anonymous union, so name them uniquely
                self.nameStack.append("anon-union-%d" % self.anon_union_counter)

However, when I try with only a union declarator, the class is parsed, but not its instance in the structure. In my case, I have an array of unions, and the parser does not see it.

virtuald commented 4 years ago

I find that setting the environment variable CPPHEADERPARSER_DEBUG=1 is really useful for figuring out what the parser is doing.

RduMarais commented 4 years ago

I used this to investigate a little bit, this is my debug log : for my array

[3192] <anon-struct-10> (public) 
[3228] trace (Empty Stack)
[3181] Evaluating stack ['PORT', '[', '32', ']']
       BraceDepth: 1 (called from 2952)
[3192] <anon-struct-10> (public) 
[3312] trace
[2356] trace
[1186] var trace ['PORT', '[', '32', ']']
[1209] Array
[1221] Variable: ['PORT']

for a nested structure in the same file

[3192] <anon-union-1>::<anon-struct-2> (public) 
[3228] trace (Empty Stack)
[3181] Evaluating stack ['b']
       BraceDepth: 1 (called from 2952)
[3192] <anon-union-1> (public) 
[3312] trace
[2356] trace
[2383] DEANONYMOIZING b to type '<anon-struct-2>'
[1186] var trace ['<anon-struct-2>', 'b']
[1221] Variable: ['<anon-struct-2>', 'b']
[3181] Evaluating stack ['uint32_t', 'w']
       BraceDepth: 1 (called from 2952)

from my comprehension, the parser gets to the line 2371, where it decides what to do with the variable.

if the variable declaration has a length of 1, it checks for nested class types.
if not, it just declares a variable, assuming that the first keyword is the type. Hence it goes to the declaration of a CppVariableand sets the flags for an array of variables.

So my second issue here is that i can not parse an array of nested classes. I don't know if just changing the condition to ( len(self.nameStack) == 1 or self.nameStack[1]=="[") would correct this or if it breaks a few things in the library. I try the 2 modifications and make a few checks.

RduMarais commented 4 years ago

So I made a pull request with a few changes that should not break anything.

This bug is made of 2 problems :

The parser does not parse nested classes with preprocessor keywords
The parser does not parse arrays of nested classes

the second issue should be adressed in the PR but the first one needs deeper changes

robotpy / robotpy-cppheaderparser

Unions nested inside structures are not parsed #37