search

CastXML / pygccxml

pygccxml is a specialized XML reader that reads the output from CastXML. It provides a simple framework to navigate C++ declarations, using Python classes.
Boost Software License 1.0
132 stars 45 forks source link

Unable to find out actual class definition: ''. #45

Closed othi closed 8 years ago

othi commented 8 years ago

I seem to have stumbled on a bug while trying to use Py++ on a larger project.

Here is a snippet that reproduces the bug (edited out the namespace blunder for clearness sake):

#include <map>
using namespace std;
class apta_node;
typedef map<int, apta_node*> child_map;
class apta_node{
public:
    child_map children;
};

Here's how I try to build the module:

from pyplusplus import module_builder
mb = module_builder.module_builder_t(
    files=['test.h'],
    include_paths=["/usr/include/x86_64-linux-gnu/c++/5", "/usr/include/c++/5"],
    cflags="-std=c++11"
)

And here's the output and error message:

d@machine:~/dev/pyplusplustest$ python3 test_generate.py 
INFO Parsing source file "test.h" ... 
error: invalid argument '-std=c++11' not allowed with 'C/ObjC'         # I need to add it or --castxml-gccxml doesn't work. The XML files generates fine.
Traceback (most recent call last):
  File "test_generate.py", line 8, in <module>
    cflags="-std=c++11"
  File "/usr/local/lib/python3.4/dist-packages/pyplusplus-1.1.0-py3.4.egg/pyplusplus/module_builder/boost_python_builder.py", line 106, in __init__
  File "/usr/local/lib/python3.4/dist-packages/pyplusplus-1.1.0-py3.4.egg/pyplusplus/module_builder/boost_python_builder.py", line 149, in __parse_declarations
  File "/usr/local/lib/python3.4/dist-packages/pygccxml/parser/project_reader.py", line 249, in read_files
    return self.__parse_file_by_file(files)
  File "/usr/local/lib/python3.4/dist-packages/pygccxml/parser/project_reader.py", line 319, in __parse_file_by_file
    self._relink_declarated_types(leaved_classes, types)
  File "/usr/local/lib/python3.4/dist-packages/pygccxml/parser/project_reader.py", line 544, in _relink_declarated_types
    raise Exception(os.linesep.join(msg))
Exception: Unable to find out actual class definition: ''.
Class definition has been changed from one compilation to an other.
Why did it happen to me? Here is a short list of reasons: 
    1. There are different preprocessor definitions applied on same file during compilation
    2. Bug in pygccxml.

After a bit of fiddling, I think the problematic line is /usr/include/c++/5/bits/stl_tree.h:1407

Using Ubuntu 15.10, pygccxml 1.7.2 (and latest development branch), gccxml from Ubuntu repository (wrapper around CastXML 0.1-g8a08a44).

praetorian20 commented 8 years ago

Why are you adding things to namespace std? That makes your code ill-formed. Can the problem be reproduced if you add those definitions to your own namespace?

If you can reproduce the error with a different namespace, then paste these lines in your test_generate.py file (at the very top, immediately after you import modules) to enable debug logging and paste the output here.

import logging
pyplusplus.module_builder.set_logger_level(logging.DEBUG)
pygccxml.utils.loggers.set_level(logging.DEBUG)
othi commented 8 years ago

The namespace was a blunder from when I created the test file. The problem remains the same if I just use the std namespace. :)

Here's the output after adding the debug logging:

parsing filparsing files - started
DEBUG Reading project files: file by file
INFO Parsing source file "test.h" ... 
DEBUG Reading source file: [test.h].
DEBUG File has not been found in cache, parsing...
DEBUG gccxml cmd: /usr/bin/gccxml  -std=c++11   -I"." -I"/usr/include/x86_64-linux-gnu/c++/5" -I"/usr/include/c++/5" "test.h" -fxml="/tmp/tmpuzy87zqz.xml"
error: invalid argument '-std=c++11' not allowed with 'C/ObjC'
DEBUG CASTXML version - None ( 1.136 )
DEBUG Flushing cache... 
DEBUG Cache has been flushed in 0.0 secs
DEBUG Joining namespaces ...
DEBUG Joining declarations ...
DEBUG Relinking declared types ...
Traceback (most recent call last):
  File "test_generate.py", line 13, in <module>
    cflags="-std=c++11"
  File "/usr/local/lib/python3.4/dist-packages/pyplusplus-1.1.0-py3.4.egg/pyplusplus/module_builder/boost_python_builder.py", line 106, in __init__
  File "/usr/local/lib/python3.4/dist-packages/pyplusplus-1.1.0-py3.4.egg/pyplusplus/module_builder/boost_python_builder.py", line 149, in __parse_declarations
  File "/usr/local/lib/python3.4/dist-packages/pygccxml/parser/project_reader.py", line 249, in read_files
    return self.__parse_file_by_file(files)
  File "/usr/local/lib/python3.4/dist-packages/pygccxml/parser/project_reader.py", line 319, in __parse_file_by_file
    self._relink_declarated_types(leaved_classes, types)
  File "/usr/local/lib/python3.4/dist-packages/pygccxml/parser/project_reader.py", line 544, in _relink_declarated_types
    raise Exception(os.linesep.join(msg))
Exception: Unable to find out actual class definition: ''.
Class definition has been changed from one compilation to an other.
Why did it happen to me? Here is a short list of reasons: 
    1. There are different preprocessor definitions applied on same file during compilation
    2. Bug in pygccxml.
laxris commented 8 years ago

It also does not work when just trying to create an instance of map, without typedef or forward declaration. The debug and log output is the same.

#include <map>

using namespace std;

class mytest {
        map<int, int> mymap; // works without this
};
praetorian20 commented 8 years ago

Your problem is you're using GCCXML for parsing your header. GCCXML is a fork of a very old GCC (version 4.3 IIRC), so it's no wonder it doesn't know what -std=c++11 is, there was no C++11 standard at that time. You may be able to use -std=c++0x instead, but even then it might fail to parse the header.

You should switch over to CastXML, it's the successor to GCCXML, and works with -std=c++11.

othi commented 8 years ago

Indeed.

I switched to CastXML:

generator_path, generator_name = pygccxml.utils.find_xml_generator('castxml')
xml_generator_config = pygccxml.parser.xml_generator_configuration_t(
  cflags="-std=c++11",
  xml_generator=generator_name,
  xml_generator_path=generator_path)

mb = pyplusplus.module_builder.module_builder_t(
        files=['test.h'],
        xml_generator_config=xml_generator_config
)

The end result remains the same:

parsing files - started
DEBUG Reading project files: file by file
INFO Parsing source file "test.h" ... 
DEBUG Reading source file: [test.h].
DEBUG File has not been found in cache, parsing...
DEBUG castxml cmd: /usr/bin/castxml  -std=c++11   -I. -c -x c++ --castxml-cc-gnu  "(" /usr/bin/c++ -std=c++11 ")" --castxml-gccxml -o /tmp/tmpcy3u3zdi.xml test.h
DEBUG CASTXML version - None ( 1.136 )
DEBUG find single query execution - started
DEBUG running query: (decl type==variable_t) and (name==static_cast< size_t >)
DEBUG running non optimized query - optimization has not been done
DEBUG find single query execution - started
DEBUG running query: (decl type==variable_t) and (name==static_cast< size_t >)
DEBUG running non optimized query - optimization has not been done
DEBUG find single query execution - started
DEBUG running query: (decl type==variable_t) and (name==static_cast< size_t >)
DEBUG running non optimized query - optimization has not been done
DEBUG find single query execution - started
DEBUG running query: (decl type==variable_t) and (name==static_cast< size_t >)
DEBUG running non optimized query - optimization has not been done
DEBUG find single query execution - started
DEBUG running query: (decl type==variable_t) and (name==static_cast< size_t >)
DEBUG running non optimized query - optimization has not been done
DEBUG find single query execution - started
DEBUG running query: (decl type==variable_t) and (name==static_cast< size_t >)
DEBUG running non optimized query - optimization has not been done
DEBUG Flushing cache... 
DEBUG Cache has been flushed in 0.0 secs
DEBUG Joining namespaces ...
DEBUG Joining declarations ...
DEBUG Relinking declared types ...
Traceback (most recent call last):
  File "test_generate.py", line 17, in <module>
    xml_generator_config=xml_generator_config
  File "/usr/local/lib/python3.4/dist-packages/pyplusplus-1.1.0-py3.4.egg/pyplusplus/module_builder/boost_python_builder.py", line 106, in __init__
  File "/usr/local/lib/python3.4/dist-packages/pyplusplus-1.1.0-py3.4.egg/pyplusplus/module_builder/boost_python_builder.py", line 149, in __parse_declarations
  File "/usr/local/lib/python3.4/dist-packages/pygccxml-1.7.2-py3.4.egg/pygccxml/parser/project_reader.py", line 249, in read_files
  File "/usr/local/lib/python3.4/dist-packages/pygccxml-1.7.2-py3.4.egg/pygccxml/parser/project_reader.py", line 319, in __parse_file_by_file
  File "/usr/local/lib/python3.4/dist-packages/pygccxml-1.7.2-py3.4.egg/pygccxml/parser/project_reader.py", line 544, in _relink_declarated_types
Exception: Unable to find out actual class definition: ''.
Class definition has been changed from one compilation to an other.
Why did it happen to me? Here is a short list of reasons: 
    1. There are different preprocessor definitions applied on same file during compilation
    2. Bug in pygccxml.

Note: The module_builder completes when I remove the cflags="-std=c++11" configuration, but that is not really an option.

praetorian20 commented 8 years ago

CastXML is being invoked with this command

/usr/bin/castxml  -std=c++11   -I. -c -x c++ --castxml-cc-gnu  "(" /usr/bin/c++ -std=c++11 ")" --castxml-gccxml -o /tmp/tmpcy3u3zdi.xml test.h

Is /usr/bin/c++ an alias for gcc-5 (I assume you want to use gcc-5.x from the include paths above)? If that's not the correct path, try passing compiler_path=... to xml_generator_configuration_t.

@iMichka Can you please takeover helping with this issue? I won't be able to get back to it until next week.

iMichka commented 8 years ago

@praetorian20 Thanks for starting debugging this.

@othi thanks for reporting the problem. I hope we will be able to get that sorted out.

I just ran the code on my Mac with py++ (latest commit) and pygccxml (latest commit), using the latest castxml and clang. Everything went fine. But it should also work with older versions of py++/pygccxml.

I think knowing which compiler was used would help debug this. As asked above, is /usr/bin/c++ the correct compiler, and what version is it ? Setting the right compiler_path is recommended if you have multiple compilers installed.

I can try to install ubuntu 15.10 in a virtual box and setup the same build env as you, to try to reproduce the error. I'll do that this weekend if the debugging here is not helping.

othi commented 8 years ago

Sorry I didn't have a lot of time today to work on this.

/usr/bin/c++ is an alias for /usr/bin/g++-5. I'm going to fiddle around with the compiler path and report back.

Thanks for your quick responses!

othi commented 8 years ago

The result is the same when used with the compiler_path explicitly set to /usr/bin/g++:

generator_path, generator_name = pygccxml.utils.find_xml_generator('castxml')
xml_generator_config = pygccxml.parser.xml_generator_configuration_t(
  cflags="-std=c++11",
  compiler_path="/usr/bin/g++",
  xml_generator=generator_name,
  xml_generator_path=generator_path)
parsing files - started
DEBUG Reading project files: file by file
INFO Parsing source file "test.h" ... 
DEBUG Reading source file: [test.h].
DEBUG File has not been found in cache, parsing...
DEBUG castxml cmd: /usr/bin/castxml  -std=c++11   -I. -c -x c++ --castxml-cc-gnu  "(" /usr/bin/g++ -std=c++11 ")" --castxml-gccxml -o /tmp/tmphqxl1mt2.xml test.h
DEBUG CASTXML version - None ( 1.136 )
<snip>
  File "/usr/local/lib/python3.4/dist-packages/pygccxml-1.7.2-py3.4.egg/pygccxml/parser/project_reader.py", line 544, in _relink_declarated_types
Exception: Unable to find out actual class definition: ''.
Class definition has been changed from one compilation to an other.
Why did it happen to me? Here is a short list of reasons: 
    1. There are different preprocessor definitions applied on same file during compilation
    2. Bug in pygccxml.

Or when used with clang:

parsing files - started
DEBUG Reading project files: file by file
INFO Parsing source file "test.h" ... 
DEBUG Reading source file: [test.h].
DEBUG File has not been found in cache, parsing...
DEBUG castxml cmd: /usr/bin/castxml  -std=c++11   -I. -c -x c++ --castxml-cc-gnu  "(" /usr/bin/clang -std=c++11 ")" --castxml-gccxml -o /tmp/tmpmsq0r7kr.xml test.h
DEBUG CASTXML version - None ( 1.136 )
<snip>
  File "/usr/local/lib/python3.4/dist-packages/pygccxml-1.7.2-py3.4.egg/pygccxml/parser/project_reader.py", line 544, in _relink_declarated_types
Exception: Unable to find out actual class definition: ''.
Class definition has been changed from one compilation to an other.
Why did it happen to me? Here is a short list of reasons: 
    1. There are different preprocessor definitions applied on same file during compilation
    2. Bug in pygccxml.

pyplusplus is the latest commit from Bitbucket, pygccxml the latest commit from here, castxml version 0.1-g8a08a44, compilers from the Ubuntu repositories.

iMichka commented 8 years ago

Ok thanks. I am currently installing Wily and gcc5, I'll try to reproduce the problem there.

iMichka commented 8 years ago

Good news, I could reproduce the error. I'll try to fix this but it may take a little bit of time, depending on how complex that part of the code is.

iMichka commented 8 years ago

I pushed a fix. It's not perfect, as this part of the code is not really clear. As all the tests pass, I am quite confident this did not break anything else. Could you please test again on your side ? The fix will be in release v1.7.3.

Be aware that there are still a bunch of errors in the container traits part with gcc5, which I discovered while running the tests. If you happen to use pygccxml's container traits it may be broken. I opened another bug for this (#47) (this problem is not related to this one).

othi commented 8 years ago

This fixed the issue, thank you very much.

I may have found another one - I'll dig a little deeper and try to build a concise test case before I report it.

iMichka commented 8 years ago

Great. I'll close this issue then.

Please open a new one for the second issue. A test case is always welcome.