[kythe] automatic inter-file dependency detection

[fangism]

Currently, the verilog-kythe extractor requires an input file list that is already dependency sorted, with symbol definitions always in earlier files than references. This places the burden on the user to pass in such an ordered file list. (The consequence of incorrect ordering now is just missing def/ref edges, but aside from that things should still work.). Since we have the ability to construct partial symbol tables and detect (punt) undefined references, it should be possible to establish dependency edges between files based on root-level symbol definitions and references.

The dependency graph should not only be used internally for project fact/edge extraction, but it could be output as intermediate information for other tools to consume, e.g. unordered file-list in, ordered file-list out.

If you output lines of a b where b depends on a (an edge), you could even pipe that through UNIX tsort (topological sort).

[fangism]

blocked by #563

[fangism]

This dependency information could also be used to optimize extraction in that only anti-dependencies of modified files have to be re-extracted (incremental extraction). For low-latency on-the-fly re-indexing!

[suzizecat]

Hi ! As I see in your commit 0e00783 , I understand that verible-verilog-project file-deps should be used to scan for dependencies. Could you confirm that ?

Also, I seem to have an issue with it on a pretty big testcase. Is there known issues preventing file-deps from outputing any kind of dependency information ? (I got a lot of stuff on stderr, but nothing on stdout)

[fangism]

Hello, that is the intent, though it may not be complete. This started out as a proof-of-concept, you might be among the first to try it on larger projects and find early bugs. This mini-project would benefit greatly from building up a suite of small demonstrative test cases, working our way up to more intricate ones. Can you share any test cases that demonstrate problems, preferably reduced?

[suzizecat]

Hi, The issue seems to be due to multiple defined symbols.

Check this testcase You can see the issue by running verible-verilog-project file-deps --file_list_path flist.fls. I get :

$ verible-verilog-project file-deps --file_list_path flist.fls
[combined statuses]:
Symbol "child" is already defined in the $root scope.
Symbol "i" is already defined in the $root::child scope.
Symbol "o" is already defined in the $root::child scope.

Which is indeed true, but I would expect to have this order (with no particular order for ch1 and ch2):

mul_ref_ch2.sv
mul_ref_ch1.sv
mul_ref_top.sv

Just two points :

• Could it be possible to output the dependency graph through a Kythe schema (as Kythe propose the depends edge) ?
• With this question comes another : could it be possible to integrate this in the kythe-extractor to get file ordering (if I understand correctly, that was the original purpose of this mini-project) and the indexer run on the same go, hence being probably much more efficient than running the dependency detection and then the index ?