As per recent refactor, compdb hangs (infinite loop?) a lot

[3v1n0]

I'm running compdb over compile_commands.json files that are generated by bear or ninja.

In the previous version, when I had to generate the headersdb.json and then join it with the compile commands, it was working fine.

In the current implementation, instead, compdb hangs and it doesn't finish its job.

One project I'm currently having troubles with is https://github.com/GNOME/mutter/

When I interrupt the execution it stops at this state, but I don't think it helps. Let me know what you'd need in order to debug this.

  File "/home/marco/.local/bin/compdb", line 9, in <module>
    load_entry_point('compdb==0.1.1', 'console_scripts', 'compdb')()
  File "/home/marco/.local/lib/python2.7/site-packages/compdb/cli.py", line 291, in main
    command.execute(config, args.args)
  File "/home/marco/.local/lib/python2.7/site-packages/compdb/cli.py", line 95, in execute
    for compile_command in compile_commands:
  File "/home/marco/.local/lib/python2.7/site-packages/compdb/includedb.py", line 221, in get_all_compile_commands
    for compile_command in self.get_compile_commands(file):
  File "/home/marco/.local/lib/python2.7/site-packages/compdb/includedb.py", line 205, in get_compile_commands
    best = self._find_best(path)
  File "/home/marco/.local/lib/python2.7/site-packages/compdb/includedb.py", line 194, in _find_best
    if includer not in self._db_index:
  File "/home/marco/.local/lib/python2.7/site-packages/compdb/includedb.py", line 181, in _db_index
    self.__db_index = frozenset(self.database.get_all_files())
  File "/home/marco/.local/lib/python2.7/site-packages/compdb/backend/json.py", line 31, in get_all_files
    for entry in self._data:
  File "/home/marco/.local/lib/python2.7/site-packages/compdb/backend/json.py", line 53, in _data
    self.__data = json.load(f)
  File "/usr/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)
KeyboardInterrupt

[Sarcasm]

Indeed, I noticed this too. I need to take a look. In the meantime, you can use the version published on pypi.

[3v1n0]

The one in pypi is using the old system? Thus I should use the 2 passes?

[Sarcasm]

Ok, I have a quickfix, I will check it again to see if it is 100% correct but it should solve the issue. I invite you to test. The way I tried to make "lazy" property was incorrect, so these properties that were lazy because of their cost, were recomputed over and over, making them so much more costly...

diff --git a/compdb/backend/json.py b/compdb/backend/json.py
index 4aa94b2..caa9308 100644
--- a/compdb/backend/json.py
+++ b/compdb/backend/json.py
@@ -11,6 +11,7 @@ from compdb.models import (CompileCommand, CompilationDatabaseInterface)
 class JSONCompilationDatabase(CompilationDatabaseInterface):
     def __init__(self, json_db_path):
         self.json_db_path = json_db_path
+        self.__data = None

     @classmethod
     def probe_directory(cls, directory):
@@ -48,7 +49,7 @@ class JSONCompilationDatabase(CompilationDatabaseInterface):

     @property
     def _data(self):
-        if not hasattr(self, '__data'):
+        if self.__data is None:
             with open(self.json_db_path) as f:
                 self.__data = json.load(f)
         return self.__data
diff --git a/compdb/includedb.py b/compdb/includedb.py
index 40f3935..61c4f0c 100644
--- a/compdb/includedb.py
+++ b/compdb/includedb.py
@@ -141,6 +141,7 @@ See also https://www.python.org/doc/essays/graphs/
     def __init__(self, graph, database):
         self.graph = graph
         self.database = database
+        self.__db_index = None

     def __repr__(self):
         return '<IncludedByGraph: graph = {}, database = {}>'.format(
@@ -177,7 +178,7 @@ See also https://www.python.org/doc/essays/graphs/

     @property
     def _db_index(self):
-        if not hasattr(self, '__db_index'):
+        if self.__db_index is None:
             self.__db_index = frozenset(self.database.get_all_files())
         return self.__db_index

[3v1n0]

Yes, works great here. and everything is way faster (even for those who was able to complete before)

[Sarcasm]

The fix is okay, I submitted it. Great to hear it's faster now. There are rooms for better performances but I would say this is a reasonable baseline now.

Thanks for reporting the issue.