Assertion breaking in _search.py:356 during Trie serialization for large project

[lifflander]

We have a large codebase that uses m.css. This assertion started breaking when we happened to add a method to a C++ class in our codebase:

assert len(self.children) < 16 and len(self.results) < 2048

The values that we printed for these variables are:

children:    19
results:     212

So we are exceeding the 4 bits of allocated space for the children. When I uncomment the assertion, the code seems to run to completion, but I don't know if the rendering is off somewhere.

[cz4rs]

traceback using the latest m.css version from master (42d4a9a48f31f5df6e246c948403b54b50574a2a):

Traceback (most recent call last):
  File "./m.css/documentation/doxygen.py", line 3859, in <module>
    run(state, templates=os.path.abspath(args.templates), wildcard=args.wildcard, index_pages=args.index_pages, search_merge_subtrees=not args.search_no_subtree_merging, search_add_lookahead_barriers=not args.search_no_lookahead_barriers, search_merge_prefixes=not args.search_no_prefix_merging)
  File "./m.css/documentation/doxygen.py", line 3766, in run
    data = build_search_data(state, add_lookahead_barriers=search_add_lookahead_barriers, merge_subtrees=search_merge_subtrees, merge_prefixes=search_merge_prefixes)
  File "./m.css/documentation/doxygen.py", line 2440, in build_search_data
    return serialize_search_data(trie, map, search_type_map, symbol_count, merge_subtrees=merge_subtrees, merge_prefixes=merge_prefixes)
  File "/build/vt/m.css/documentation/_search.py", line 428, in serialize_search_data
    serialized_trie = trie.serialize(merge_subtrees=merge_subtrees)
  File "/build/vt/m.css/documentation/_search.py", line 391, in serialize
    self.root_offset_struct.pack_into(output, 0, self._serialize(hashtable, output, merge_subtrees=merge_subtrees))
  File "/build/vt/m.css/documentation/_search.py", line 345, in _serialize
    offset = child[1]._serialize(hashtable, output, merge_subtrees=merge_subtrees)
  File "/build/vt/m.css/documentation/_search.py", line 345, in _serialize
    offset = child[1]._serialize(hashtable, output, merge_subtrees=merge_subtrees)
  File "/build/vt/m.css/documentation/_search.py", line 345, in _serialize
    offset = child[1]._serialize(hashtable, output, merge_subtrees=merge_subtrees)
  [Previous line repeated 6 more times]
  File "/build/vt/m.css/documentation/_search.py", line 356, in _serialize
    assert len(self.children) < 16 and len(self.results) < 2048
AssertionError

with the values for self.children and self.results as above (19 and 212 respectively)

[mosra]

(Sorry for very late replies, finally got a chance to get back to this project.)

This is basically the same issue as described in #150 (and vaguely related to another limitation described in #123). I plan to fix this, but to clean up the issue list a bit I'll close this in favor of the older issue. Hope that's fine with you, I'll try to @ you when it's finally fixed so you can remove the patches on your side.

[mosra]

@lifflander @cz4rs This is finally fixed as of b0cf44e4ddbf42ce79a8612563e84e00e8a75808 and 0411e1812f1274bf6e49404c0dcdf649f7637076. In your particular case it should "just magically work", as the children/result count limit is now 255 and 32767, respectively.

If your project grows even further, the doc generation may abort with a message suggesting you to modify various configuration options, which then raise the default limits from 65k symbols and 16MB files to up to 4G for both. The defaults are picked like this to keep file sizes small by default, unfortunately estimating the packing options beforehand would be overly error prone so it requires manual intervention when those are reached.

[cz4rs]

@mosra Thanks for the fix and notification! Indeed, it "just works" now :ok_hand: