search

trailofbits / multiplier

Code auditing productivity multiplier.
Apache License 2.0
437 stars 27 forks source link

mx-find-linked-structures Assertion Error #565

Closed bdemick closed 2 months ago

bdemick commented 3 months ago

Running mx-find-linked-structures on an indexed Linux kernel, and running into an assertion error:

multiplier/lib/SQLiteEntityProvider.cpp:1082: virtual DeclImplPtr mx::SQLiteEntityProvider::DeclFor(const Ptr &, RawEntityId): Assertion `false' failed.

Thread 1 "mx-find-linked-" received signal SIGABRT, Aborted.

Built a debug version and it looks like it starts from FindLinkedStructures.cpp:76 in the first phase of analysis. Happy to provide more debugging info if that would be helpful.

Backtrace:

#0  __pthread_kill_implementation (no_tid=0x0, signo=0x6, threadid=0x7ffff7e4e100)
    at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=0x6, threadid=0x7ffff7e4e100) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=0x7ffff7e4e100, signo=signo@entry=0x6)
    at ./nptl/pthread_kill.c:89
#3  0x00007fffe6042476 in __GI_raise (sig=sig@entry=0x6) at ../sysdeps/posix/raise.c:26
#4  0x00007fffe60287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fffe602871b in __assert_fail_base (
    fmt=0x7fffe61dd130 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=0x7fffe90bb20b "false",
    file=0x7fffe90bdcaf "/home/user0/git/mx-work/src/multiplier/lib/SQLiteEntityProvider.cpp", line=0x43a, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007fffe6039e96 in __GI___assert_fail (assertion=0x7fffe90bb20b "false",
    file=0x7fffe90bdcaf "/home/user0/git/mx-work/src/multiplier/lib/SQLiteEntityProvider.cpp", line=0x43a,
    function=0x7fffe98353b4 "virtual DeclImplPtr mx::SQLiteEntityProvider::DeclFor(const Ptr &, RawEntityId)") at ./assert/assert.c:101
#7  0x00007ffff00165cd in mx::SQLiteEntityProvider::DeclFor (this=0x5555557bb9a0,
    self=std::shared_ptr<mx::EntityProvider> (use count 97, weak count 1) = {...},
    raw_id=0x84a000078e000001) at multiplier/lib/SQLiteEntityProvider.cpp:1075
#8  0x00007fffefd9f595 in mx::CachingEntityProvider::DeclFor (this=0x5555557e7cb0,
    self=std::shared_ptr<mx::EntityProvider> (use count 97, weak count 1) = {...},
    raw_id=0x84a000078e000001) at multiplier/lib/CachingEntityProvider.cpp:279
#9  0x00007fffef8af1c9 in mx::TagType::declaration (this=0x7fffffffde38)
    at multiplier/lib/AST/TagType.cpp:107
#10 0x0000555555592bfc in main (argc=0x3, argv=0x7fffffffe228)
    at multiplier/bin/Examples/FindLinkedStructures.cpp:76

System info

Ubuntu 22.04.4 multiplier tag: c18052b

pgoodman commented 3 months ago

Can you try the following in GDB or LLDB:

(gdb) p mx::EntityId(0x84a000078e000001).Unpack()

I'm expecting something like is_definition = true or is_definition = false in the following output:

mx::DeclId {
  fragment_id = <fragment_id>,
  kind = mx::DeclKind::RECORD,
  offset = <offset>,
  is_definition = true
}

Suppose is_definition is true. In the same debugger instance, can inverting it as follows:

(gdb) p/d mx::EntityId(mx::DeclId(<fragment_id>, mx::DeclKind::RECORD, <offset>, false)

This should print out some big decimal integer. Then:

$ mx-highlight-entity --db linux.db --entity_id <printed_id>
bdemick commented 3 months ago

Please forgive in advance my lack of experience debugging C++ in gdb :)

  1. Set a breakpoint in FindLinkedStructures.cpp:53 (after InitExample(true))
  2. Run to breakpoint (I'd expect at this point the entity_id would be valid)
  3. p mx::EntityId(0x84a000078e000001).Unpack()

This is resulting in gdb giving A syntax error in expression, near `0x84a000078e000001).Unpack()'. A colleague has suggested it might be an issue with calling constructors in gdb, or possibly some deeper stuff going on.

pgoodman commented 3 months ago

Can you try using LLDB?

$ sudo apt-get install lldb
$ lldb ./bin/mx-find-linked-structures
(lldb) r --db linux.db
<wait for it to abort>
(lldb) p mx::EntityId(0x84a000078e000001ull).Unpack()
bdemick commented 3 months ago

lldb FTW. However, that call is returning an mx::VariantId and that's pretty gnarly (see Details). I do see a (fragment_id = 1079520, kind = RECORD, offset = 1, is_definition = false) buried in there, though. Throwing that into the suggested command (and inverting the boolean):

(lldb) p/d mx::EntityId(mx::DeclID(1079520, mx::DeclKind::RECORD, 1, true))
error: expression failed to parse:
error: <user expression 1>:1:18: no member named 'DeclID' in namespace 'mx'
mx::EntityId(mx::DeclID(1079520, mx::DeclKind::RECORD, 1, true))
             ~~~~^
warning: <user expression 1>:1:38: use of enumeration in a nested name specifier is a C++11 extension
mx::EntityId(mx::DeclID(1079520, mx::DeclKind::RECORD, 1, true))
                                     ^
(lldb) p/d mx::EntityId(mx::DeclId(1079520, mx::DeclKind::RECORD, 1, true))
error: expression failed to parse:
warning: <user expression 2>:1:38: use of enumeration in a nested name specifier is a C++11 extension
mx::EntityId(mx::DeclId(1079520, mx::DeclKind::RECORD, 1, true))
                                     ^
error: <user expression 2>:1:14: no matching constructor for initialization of 'mx::DeclId'
mx::EntityId(mx::DeclId(1079520, mx::DeclKind::RECORD, 1, true))
             ^          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: candidate constructor (the implicit copy constructor) not viable: requires 1 argument, but 4 were provided

note: candidate constructor (the implicit default constructor) not viable: requires 0 arguments, but 4 were provided
Details ``` mx-find-linked-structures: /home/user0/git/mx-work/src/multiplier/lib/SQLiteEntityProvider.cpp:1082: virtual DeclImplPtr mx::SQLiteEntityProvider::DeclFor(const Ptr &, RawEntityId): Assertion `false' failed. Process 698935 stopped * thread #1, name = 'mx-find-linked-', stop reason = signal SIGABRT frame #0: 0x00007fffe60969fc libc.so.6`__GI___pthread_kill at pthread_kill.c:44:76 (lldb) p mx::EntityId(0x84a000078e000001ull).Unpack() (mx::VariantId) $0 = { std::__detail::__variant::_Variant_base = { std::__detail::__variant::_Move_assign_alias = { std::__detail::__variant::_Copy_assign_alias = { std::__detail::__variant::_Move_ctor_alias = { std::__detail::__variant::_Copy_ctor_alias = { std::__detail::__variant::_Variant_storage_alias = { _M_u = { _M_first = (_M_storage = std::monostate @ 0x0000000002e02cb0) _M_rest = { _M_first = (_M_storage = mx::FragmentId @ 0x0000000002e02cb0) _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, kind = RECORD, offset = 1, is_definition = false) } _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, kind = EXT_VECTOR_ELEMENT_EXPR, offset = 1) } _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, kind = CF_CONSUMED, offset = 1) } _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, kind = UNDEFINE_DIRECTIVE | INCLUDE_DIRECTIVE | 0x20, offset = 1) } _M_rest = { _M_first = (_M_storage = mx::TypeId @ 0x0000000002e02cb0) _M_rest = { _M_first = (_M_storage = mx::FileId @ 0x0000000002e02cb0) _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, offset = 62) } _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, offset = 62) } _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, offset = 62) } _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, offset = 62) } _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, offset = 62) } _M_rest = { _M_first = (_M_storage = mx::CompilationId @ 0x0000000002e02cb0) _M_rest = { _M_first = (_M_storage = mx::OperationId @ 0x0000000002e02cb0) _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, kind = SEMI, offset = 1) } _M_rest = { _M_first = { _M_storage = (fragment_id = 1079520, kind = SEMI, offset = 1) } _M_rest = { _M_first = { _M_storage = (file_id = 1079520, kind = SEMI, offset = 1) } _M_rest = { _M_first = { _M_storage = (type_id = 1079520, type_kind = OBJ_C_OBJECT_POINTER | OBJ_C_INTERFACE, kind = UNKNOWN, offset = 1) } _M_rest = {} } } } } } } } } } } } } } } } } } } } _M_index = '\x02' } } } } } } } ```
pgoodman commented 3 months ago

Okay lets try something else:

(lldb) p mx::EntityId(mx::FragmentId(1079520))

then take the outputted number and pass it to:

$ ./bin/mx-print-fragment --db linux.db --fragment_id <outputted_number>
pgoodman commented 3 months ago

Okay, I have done it in my LLDB ;-) Sorry, it looks like I got some of the syntax slightly wrong!

(lldb) p mx::EntityId(mx::FragmentId(1079520))
(mx::EntityId)  (opaque = 2305843009214773472)
(lldb) p mx::EntityId(mx::DeclId{1079520, mx::DeclKind::RECORD, 1, true})
(mx::EntityId)  (opaque = 9362983657750396929)

Can you run:

$ ./bin/mx-print-fragment --db linux.db --fragment_id 2305843009214773472
$ ./bin/mx-highlight-entity -db linux.db --entity_id 9362983657750396929
bdemick commented 3 months ago 
p mx::EntityId(mx::FragmentId(1079520))
(mx::EntityId) $1 = (opaque = 2305843009214773472)

Still seeing an error trying to execute this call:

p mx::EntityId(mx::DeclId{1079520, mx::DeclKind::RECORD, 1, true})
error: expression failed to parse:
error: <user expression 5>:1:24: expected '(' for function-style cast or type construction
mx::EntityId(mx::DeclId{1079520, mx::DeclKind::RECORD, 1, true})
             ~~~~~~~~~~^

Subsequently from our fragment_id above:

Fragment ``` ./mx-print-fragment --db /home/user0/linux.db --fragment_id 2305843009214773472 Parsed data from tokens: 11261250900687060992 0 KEYWORD_STRUCT KEYWORD struct 11042826318759591937 0 WHITESPACE UNKNOWN 11040574518945906690 9362983657750396928 IDENTIFIER STRUCT netns_mib 11042826318759591939 0 WHITESPACE UNKNOWN 11081106915592241156 0 L_BRACE PUNCTUATION { 11042826318759591941 0 WHITESPACE UNKNOWN 11042826318759591942 0 WHITESPACE UNKNOWN 11427884086899769351 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870664 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061001 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906698 9362983657707405312 IDENTIFIER STRUCT ipstats_mib 11078855115778555915 0 R_PARENTHESIS PUNCTUATION ) 11096869514288037900 0 STAR PUNCTUATION * 11040574518945906701 9272911665202987008 IDENTIFIER INSTANCE_MEMBER ip_statistics 11168927108325965838 0 SEMI PUNCTUATION ; 11042826318759591951 0 WHITESPACE UNKNOWN 11042826318759591952 0 WHITESPACE UNKNOWN 11042826318759591953 0 WHITESPACE UNKNOWN 11427884086899769362 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870675 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061012 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906709 9362983657707405312 IDENTIFIER STRUCT ipstats_mib 11078855115778555926 0 R_PARENTHESIS PUNCTUATION ) 11096869514288037911 0 STAR PUNCTUATION * 11040574518945906712 9272911665202987009 IDENTIFIER INSTANCE_MEMBER ipv6_statistics 11168927108325965849 0 SEMI PUNCTUATION ; 11042826318759591962 0 WHITESPACE UNKNOWN 11042826318759591963 0 WHITESPACE UNKNOWN 11042826318759591964 0 WHITESPACE UNKNOWN 11427884086899769373 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870686 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061023 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906720 9362983657719988224 IDENTIFIER STRUCT tcp_mib 11078855115778555937 0 R_PARENTHESIS PUNCTUATION ) 11096869514288037922 0 STAR PUNCTUATION * 11040574518945906723 9272911665202987010 IDENTIFIER INSTANCE_MEMBER tcp_statistics 11168927108325965860 0 SEMI PUNCTUATION ; 11042826318759591973 0 WHITESPACE UNKNOWN 11042826318759591974 0 WHITESPACE UNKNOWN 11427884086899769383 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870696 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061033 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906730 9362983657724182528 IDENTIFIER STRUCT linux_mib 11078855115778555947 0 R_PARENTHESIS PUNCTUATION ) 11096869514288037932 0 STAR PUNCTUATION * 11040574518945906733 9272911665202987011 IDENTIFIER INSTANCE_MEMBER net_statistics 11168927108325965870 0 SEMI PUNCTUATION ; 11042826318759591983 0 WHITESPACE UNKNOWN 11042826318759591984 0 WHITESPACE UNKNOWN 11427884086899769393 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870706 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061043 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906740 9362983657722085376 IDENTIFIER STRUCT udp_mib 11078855115778555957 0 R_PARENTHESIS PUNCTUATION ) 11096869514288037942 0 STAR PUNCTUATION * 11040574518945906743 9272911665202987012 IDENTIFIER INSTANCE_MEMBER udp_statistics 11168927108325965880 0 SEMI PUNCTUATION ; 11042826318759591993 0 WHITESPACE UNKNOWN 11042826318759591994 0 WHITESPACE UNKNOWN 11042826318759591995 0 WHITESPACE UNKNOWN 11427884086899769404 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870717 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061054 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906751 9362983657722085376 IDENTIFIER STRUCT udp_mib 11078855115778555968 0 R_PARENTHESIS PUNCTUATION ) 11096869514288037953 0 STAR PUNCTUATION * 11040574518945906754 9272911665202987013 IDENTIFIER INSTANCE_MEMBER udp_stats_in6 11168927108325965891 0 SEMI PUNCTUATION ; 11042826318759592004 0 WHITESPACE UNKNOWN 11042826318759592005 0 WHITESPACE UNKNOWN 11042826318759592006 0 WHITESPACE UNKNOWN 11042826318759592007 0 WHITESPACE UNKNOWN 11042826318759592008 0 WHITESPACE UNKNOWN 11042826318759592009 0 WHITESPACE UNKNOWN 11042826318759592010 0 WHITESPACE UNKNOWN 11042826318759592011 0 WHITESPACE UNKNOWN 11042826318759592012 0 WHITESPACE UNKNOWN 11427884086899769421 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870734 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061071 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906768 9362983657722085376 IDENTIFIER STRUCT udp_mib 11078855115778555985 0 R_PARENTHESIS PUNCTUATION ) 11096869514288037970 0 STAR PUNCTUATION * 11040574518945906771 9272911665202987014 IDENTIFIER INSTANCE_MEMBER udplite_statistics 11168927108325965908 0 SEMI PUNCTUATION ; 11042826318759592021 0 WHITESPACE UNKNOWN 11042826318759592022 0 WHITESPACE UNKNOWN 11042826318759592023 0 WHITESPACE UNKNOWN 11427884086899769432 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870745 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061082 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906779 9362983657722085376 IDENTIFIER STRUCT udp_mib 11078855115778555996 0 R_PARENTHESIS PUNCTUATION ) 11096869514288037981 0 STAR PUNCTUATION * 11040574518945906782 9272911665202987015 IDENTIFIER INSTANCE_MEMBER udplite_stats_in6 11168927108325965919 0 SEMI PUNCTUATION ; 11042826318759592032 0 WHITESPACE UNKNOWN 11042826318759592033 0 WHITESPACE UNKNOWN 11042826318759592034 0 WHITESPACE UNKNOWN 11427884086899769443 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870756 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061093 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906790 9362983657709502464 IDENTIFIER STRUCT icmp_mib 11078855115778556007 0 R_PARENTHESIS PUNCTUATION ) 11096869514288037992 0 STAR PUNCTUATION * 11040574518945906793 9272911665202987016 IDENTIFIER INSTANCE_MEMBER icmp_statistics 11168927108325965930 0 SEMI PUNCTUATION ; 11042826318759592043 0 WHITESPACE UNKNOWN 11042826318759592044 0 WHITESPACE UNKNOWN 11427884086899769453 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870766 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061103 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906800 9362983657711599616 IDENTIFIER STRUCT icmpmsg_mib 11078855115778556017 0 R_PARENTHESIS PUNCTUATION ) 11096869514288038002 0 STAR PUNCTUATION * 11040574518945906803 9272911665202987017 IDENTIFIER INSTANCE_MEMBER icmpmsg_statistics 11168927108325965940 0 SEMI PUNCTUATION ; 11042826318759592053 0 WHITESPACE UNKNOWN 11042826318759592054 0 WHITESPACE UNKNOWN 11042826318759592055 0 WHITESPACE UNKNOWN 11427884086899769464 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870777 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061114 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906811 9362983657712648192 IDENTIFIER STRUCT icmpv6_mib 11078855115778556028 0 R_PARENTHESIS PUNCTUATION ) 11096869514288038013 0 STAR PUNCTUATION * 11040574518945906814 9272911665202987018 IDENTIFIER INSTANCE_MEMBER icmpv6_statistics 11168927108325965951 0 SEMI PUNCTUATION ; 11042826318759592064 0 WHITESPACE UNKNOWN 11042826318759592065 0 WHITESPACE UNKNOWN 11427884086899769474 0 KEYWORD_TYPEOF KEYWORD __typeof__ 11076603315964870787 0 L_PARENTHESIS PUNCTUATION ( 11261250900687061124 0 KEYWORD_STRUCT KEYWORD struct 11040574518945906821 9362983657715793920 IDENTIFIER STRUCT icmpv6msg_mib 11078855115778556038 0 R_PARENTHESIS PUNCTUATION ) 11096869514288038023 0 STAR PUNCTUATION * 11040574518945906824 9272911665202987019 IDENTIFIER INSTANCE_MEMBER icmpv6msg_statistics 11168927108325965961 0 SEMI PUNCTUATION ; 11042826318759592074 0 WHITESPACE UNKNOWN 11042826318759592075 0 WHITESPACE UNKNOWN 11261250900687061132 0 KEYWORD_STRUCT KEYWORD struct 11042826318759592077 0 WHITESPACE UNKNOWN 11040574518945906830 9362983646608228352 IDENTIFIER STRUCT proc_dir_entry 11042826318759592079 0 WHITESPACE UNKNOWN 11096869514288038032 0 STAR PUNCTUATION * 11040574518945906833 9272911665202987020 IDENTIFIER INSTANCE_MEMBER proc_net_devsnmp6 11168927108325965970 0 SEMI PUNCTUATION ; 11042826318759592083 0 WHITESPACE UNKNOWN 11042826318759592084 0 WHITESPACE UNKNOWN 11083358715405926549 0 R_BRACE PUNCTUATION } 11168927108325965974 0 SEMI PUNCTUATION ; Parsed data from token range: struct netns_mib { __typeof__(structipstats_mib)*ip_statistics; __typeof__(structipstats_mib)*ipv6_statistics; __typeof__(structtcp_mib)*tcp_statistics; __typeof__(structlinux_mib)*net_statistics; __typeof__(structudp_mib)*udp_statistics; __typeof__(structudp_mib)*udp_stats_in6; __typeof__(structudp_mib)*udplite_statistics; __typeof__(structudp_mib)*udplite_stats_in6; __typeof__(structicmp_mib)*icmp_statistics; __typeof__(structicmpmsg_mib)*icmpmsg_statistics; __typeof__(structicmpv6_mib)*icmpv6_statistics; __typeof__(structicmpv6msg_mib)*icmpv6msg_statistics; struct proc_dir_entry *proc_net_devsnmp6; }; File data from parsed token range: struct netns_mib { DEFINE_SNMP_STAT(struct ipstats_mib, ip_statistics); #if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct ipstats_mib, ipv6_statistics); #endif DEFINE_SNMP_STAT(struct tcp_mib, tcp_statistics); DEFINE_SNMP_STAT(struct linux_mib, net_statistics); DEFINE_SNMP_STAT(struct udp_mib, udp_statistics); #if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct udp_mib, udp_stats_in6); #endif #ifdef CONFIG_XFRM_STATISTICS DEFINE_SNMP_STAT(struct linux_xfrm_mib, xfrm_statistics); #endif #if IS_ENABLED(CONFIG_TLS) DEFINE_SNMP_STAT(struct linux_tls_mib, tls_statistics); #endif #ifdef CONFIG_MPTCP DEFINE_SNMP_STAT(struct mptcp_mib, mptcp_statistics); #endif DEFINE_SNMP_STAT(struct udp_mib, udplite_statistics); #if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct udp_mib, udplite_stats_in6); #endif DEFINE_SNMP_STAT(struct icmp_mib, icmp_statistics); DEFINE_SNMP_STAT_ATOMIC(struct icmpmsg_mib, icmpmsg_statistics); #if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct icmpv6_mib, icmpv6_statistics); DEFINE_SNMP_STAT_ATOMIC(struct icmpv6msg_mib, icmpv6msg_statistics); struct proc_dir_entry *proc_net_devsnmp6; #endif }; File data from tokens: 4612575523334270469 0 IDENTIFIER IDENTIFIER struct 4612575523334270982 0 WHITESPACE UNKNOWN 4612575523334271493 0 IDENTIFIER IDENTIFIER netns_mib 4612575523334272006 0 WHITESPACE UNKNOWN 4612575523334272535 0 L_BRACE PUNCTUATION { 4612575523334273030 0 WHITESPACE UNKNOWN 4612575523334273542 0 WHITESPACE UNKNOWN 4612575523334274053 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334274581 0 L_PARENTHESIS PUNCTUATION ( 4612575523334275077 0 IDENTIFIER IDENTIFIER struct 4612575523334275590 0 WHITESPACE UNKNOWN 4612575523334276101 0 IDENTIFIER IDENTIFIER ipstats_mib 4612575523334276673 0 COMMA PUNCTUATION , 4612575523334277126 0 WHITESPACE UNKNOWN 4612575523334277637 0 IDENTIFIER IDENTIFIER ip_statistics 4612575523334278166 0 R_PARENTHESIS PUNCTUATION ) 4612575523334278718 0 SEMI PUNCTUATION ; 4612575523334279174 0 WHITESPACE UNKNOWN 4612575523334279746 0 HASH PUNCTUATION # 4612575523334280642 0 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334280710 0 WHITESPACE UNKNOWN 4612575523334281221 0 IDENTIFIER IDENTIFIER IS_ENABLED 4612575523334281749 0 L_PARENTHESIS PUNCTUATION ( 4612575523334282245 0 IDENTIFIER IDENTIFIER CONFIG_IPV6 4612575523334282774 0 R_PARENTHESIS PUNCTUATION ) 4612575523334283270 0 WHITESPACE UNKNOWN 4612575523334283782 0 WHITESPACE UNKNOWN 4612575523334284293 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334284821 0 L_PARENTHESIS PUNCTUATION ( 4612575523334285317 0 IDENTIFIER IDENTIFIER struct 4612575523334285830 0 WHITESPACE UNKNOWN 4612575523334286341 0 IDENTIFIER IDENTIFIER ipstats_mib 4612575523334286913 0 COMMA PUNCTUATION , 4612575523334287366 0 WHITESPACE UNKNOWN 4612575523334287877 0 IDENTIFIER IDENTIFIER ipv6_statistics 4612575523334288406 0 R_PARENTHESIS PUNCTUATION ) 4612575523334288958 0 SEMI PUNCTUATION ; 4612575523334289414 0 WHITESPACE UNKNOWN 4612575523334289986 0 HASH PUNCTUATION # 4612575523334290889 0 PP_ENDIF MACRO_DIRECTIVE_NAME endif 4612575523334290950 0 WHITESPACE UNKNOWN 4612575523334291462 0 WHITESPACE UNKNOWN 4612575523334291973 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334292501 0 L_PARENTHESIS PUNCTUATION ( 4612575523334292997 0 IDENTIFIER IDENTIFIER struct 4612575523334293510 0 WHITESPACE UNKNOWN 4612575523334294021 0 IDENTIFIER IDENTIFIER tcp_mib 4612575523334294593 0 COMMA PUNCTUATION , 4612575523334295046 0 WHITESPACE UNKNOWN 4612575523334295557 0 IDENTIFIER IDENTIFIER tcp_statistics 4612575523334296086 0 R_PARENTHESIS PUNCTUATION ) 4612575523334296638 0 SEMI PUNCTUATION ; 4612575523334297094 0 WHITESPACE UNKNOWN 4612575523334297606 0 WHITESPACE UNKNOWN 4612575523334298117 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334298645 0 L_PARENTHESIS PUNCTUATION ( 4612575523334299141 0 IDENTIFIER IDENTIFIER struct 4612575523334299654 0 WHITESPACE UNKNOWN 4612575523334300165 0 IDENTIFIER IDENTIFIER linux_mib 4612575523334300737 0 COMMA PUNCTUATION , 4612575523334301190 0 WHITESPACE UNKNOWN 4612575523334301701 0 IDENTIFIER IDENTIFIER net_statistics 4612575523334302230 0 R_PARENTHESIS PUNCTUATION ) 4612575523334302782 0 SEMI PUNCTUATION ; 4612575523334303238 0 WHITESPACE UNKNOWN 4612575523334303750 0 WHITESPACE UNKNOWN 4612575523334304261 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334304789 0 L_PARENTHESIS PUNCTUATION ( 4612575523334305285 0 IDENTIFIER IDENTIFIER struct 4612575523334305798 0 WHITESPACE UNKNOWN 4612575523334306309 0 IDENTIFIER IDENTIFIER udp_mib 4612575523334306881 0 COMMA PUNCTUATION , 4612575523334307334 0 WHITESPACE UNKNOWN 4612575523334307845 0 IDENTIFIER IDENTIFIER udp_statistics 4612575523334308374 0 R_PARENTHESIS PUNCTUATION ) 4612575523334308926 0 SEMI PUNCTUATION ; 4612575523334309382 0 WHITESPACE UNKNOWN 4612575523334309954 0 HASH PUNCTUATION # 4612575523334310850 0 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334310918 0 WHITESPACE UNKNOWN 4612575523334311429 0 IDENTIFIER IDENTIFIER IS_ENABLED 4612575523334311957 0 L_PARENTHESIS PUNCTUATION ( 4612575523334312453 0 IDENTIFIER IDENTIFIER CONFIG_IPV6 4612575523334312982 0 R_PARENTHESIS PUNCTUATION ) 4612575523334313478 0 WHITESPACE UNKNOWN 4612575523334313990 0 WHITESPACE UNKNOWN 4612575523334314501 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334315029 0 L_PARENTHESIS PUNCTUATION ( 4612575523334315525 0 IDENTIFIER IDENTIFIER struct 4612575523334316038 0 WHITESPACE UNKNOWN 4612575523334316549 0 IDENTIFIER IDENTIFIER udp_mib 4612575523334317121 0 COMMA PUNCTUATION , 4612575523334317574 0 WHITESPACE UNKNOWN 4612575523334318085 0 IDENTIFIER IDENTIFIER udp_stats_in6 4612575523334318614 0 R_PARENTHESIS PUNCTUATION ) 4612575523334319166 0 SEMI PUNCTUATION ; 4612575523334319622 0 WHITESPACE UNKNOWN 4612575523334320194 0 HASH PUNCTUATION # 4612575523334321097 0 PP_ENDIF MACRO_DIRECTIVE_NAME endif 4612575523334321158 0 WHITESPACE UNKNOWN 4612575523334321730 0 HASH PUNCTUATION # 4612575523334322627 0 PP_IFDEF MACRO_DIRECTIVE_NAME ifdef 4612575523334322694 0 WHITESPACE UNKNOWN 4612575523334323205 0 IDENTIFIER IDENTIFIER CONFIG_XFRM_STATISTICS 4612575523334323718 0 WHITESPACE UNKNOWN 4612575523334324230 0 WHITESPACE UNKNOWN 4612575523334324741 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334325269 0 L_PARENTHESIS PUNCTUATION ( 4612575523334325765 0 IDENTIFIER IDENTIFIER struct 4612575523334326278 0 WHITESPACE UNKNOWN 4612575523334326789 0 IDENTIFIER IDENTIFIER linux_xfrm_mib 4612575523334327361 0 COMMA PUNCTUATION , 4612575523334327814 0 WHITESPACE UNKNOWN 4612575523334328325 0 IDENTIFIER IDENTIFIER xfrm_statistics 4612575523334328854 0 R_PARENTHESIS PUNCTUATION ) 4612575523334329406 0 SEMI PUNCTUATION ; 4612575523334329862 0 WHITESPACE UNKNOWN 4612575523334330434 0 HASH PUNCTUATION # 4612575523334331337 0 PP_ENDIF MACRO_DIRECTIVE_NAME endif 4612575523334331398 0 WHITESPACE UNKNOWN 4612575523334331970 0 HASH PUNCTUATION # 4612575523334332866 0 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334332934 0 WHITESPACE UNKNOWN 4612575523334333445 0 IDENTIFIER IDENTIFIER IS_ENABLED 4612575523334333973 0 L_PARENTHESIS PUNCTUATION ( 4612575523334334469 0 IDENTIFIER IDENTIFIER CONFIG_TLS 4612575523334334998 0 R_PARENTHESIS PUNCTUATION ) 4612575523334335494 0 WHITESPACE UNKNOWN 4612575523334336006 0 WHITESPACE UNKNOWN 4612575523334336517 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334337045 0 L_PARENTHESIS PUNCTUATION ( 4612575523334337541 0 IDENTIFIER IDENTIFIER struct 4612575523334338054 0 WHITESPACE UNKNOWN 4612575523334338565 0 IDENTIFIER IDENTIFIER linux_tls_mib 4612575523334339137 0 COMMA PUNCTUATION , 4612575523334339590 0 WHITESPACE UNKNOWN 4612575523334340101 0 IDENTIFIER IDENTIFIER tls_statistics 4612575523334340630 0 R_PARENTHESIS PUNCTUATION ) 4612575523334341182 0 SEMI PUNCTUATION ; 4612575523334341638 0 WHITESPACE UNKNOWN 4612575523334342210 0 HASH PUNCTUATION # 4612575523334343113 0 PP_ENDIF MACRO_DIRECTIVE_NAME endif 4612575523334343174 0 WHITESPACE UNKNOWN 4612575523334343746 0 HASH PUNCTUATION # 4612575523334344643 0 PP_IFDEF MACRO_DIRECTIVE_NAME ifdef 4612575523334344710 0 WHITESPACE UNKNOWN 4612575523334345221 0 IDENTIFIER IDENTIFIER CONFIG_MPTCP 4612575523334345734 0 WHITESPACE UNKNOWN 4612575523334346246 0 WHITESPACE UNKNOWN 4612575523334346757 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334347285 0 L_PARENTHESIS PUNCTUATION ( 4612575523334347781 0 IDENTIFIER IDENTIFIER struct 4612575523334348294 0 WHITESPACE UNKNOWN 4612575523334348805 0 IDENTIFIER IDENTIFIER mptcp_mib 4612575523334349377 0 COMMA PUNCTUATION , 4612575523334349830 0 WHITESPACE UNKNOWN 4612575523334350341 0 IDENTIFIER IDENTIFIER mptcp_statistics 4612575523334350870 0 R_PARENTHESIS PUNCTUATION ) 4612575523334351422 0 SEMI PUNCTUATION ; 4612575523334351878 0 WHITESPACE UNKNOWN 4612575523334352450 0 HASH PUNCTUATION # 4612575523334353353 0 PP_ENDIF MACRO_DIRECTIVE_NAME endif 4612575523334353414 0 WHITESPACE UNKNOWN 4612575523334353926 0 WHITESPACE UNKNOWN 4612575523334354437 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334354965 0 L_PARENTHESIS PUNCTUATION ( 4612575523334355461 0 IDENTIFIER IDENTIFIER struct 4612575523334355974 0 WHITESPACE UNKNOWN 4612575523334356485 0 IDENTIFIER IDENTIFIER udp_mib 4612575523334357057 0 COMMA PUNCTUATION , 4612575523334357510 0 WHITESPACE UNKNOWN 4612575523334358021 0 IDENTIFIER IDENTIFIER udplite_statistics 4612575523334358550 0 R_PARENTHESIS PUNCTUATION ) 4612575523334359102 0 SEMI PUNCTUATION ; 4612575523334359558 0 WHITESPACE UNKNOWN 4612575523334360130 0 HASH PUNCTUATION # 4612575523334361026 0 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334361094 0 WHITESPACE UNKNOWN 4612575523334361605 0 IDENTIFIER IDENTIFIER IS_ENABLED 4612575523334362133 0 L_PARENTHESIS PUNCTUATION ( 4612575523334362629 0 IDENTIFIER IDENTIFIER CONFIG_IPV6 4612575523334363158 0 R_PARENTHESIS PUNCTUATION ) 4612575523334363654 0 WHITESPACE UNKNOWN 4612575523334364166 0 WHITESPACE UNKNOWN 4612575523334364677 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334365205 0 L_PARENTHESIS PUNCTUATION ( 4612575523334365701 0 IDENTIFIER IDENTIFIER struct 4612575523334366214 0 WHITESPACE UNKNOWN 4612575523334366725 0 IDENTIFIER IDENTIFIER udp_mib 4612575523334367297 0 COMMA PUNCTUATION , 4612575523334367750 0 WHITESPACE UNKNOWN 4612575523334368261 0 IDENTIFIER IDENTIFIER udplite_stats_in6 4612575523334368790 0 R_PARENTHESIS PUNCTUATION ) 4612575523334369342 0 SEMI PUNCTUATION ; 4612575523334369798 0 WHITESPACE UNKNOWN 4612575523334370370 0 HASH PUNCTUATION # 4612575523334371273 0 PP_ENDIF MACRO_DIRECTIVE_NAME endif 4612575523334371334 0 WHITESPACE UNKNOWN 4612575523334371846 0 WHITESPACE UNKNOWN 4612575523334372357 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334372885 0 L_PARENTHESIS PUNCTUATION ( 4612575523334373381 0 IDENTIFIER IDENTIFIER struct 4612575523334373894 0 WHITESPACE UNKNOWN 4612575523334374405 0 IDENTIFIER IDENTIFIER icmp_mib 4612575523334374977 0 COMMA PUNCTUATION , 4612575523334375430 0 WHITESPACE UNKNOWN 4612575523334375941 0 IDENTIFIER IDENTIFIER icmp_statistics 4612575523334376470 0 R_PARENTHESIS PUNCTUATION ) 4612575523334377022 0 SEMI PUNCTUATION ; 4612575523334377478 0 WHITESPACE UNKNOWN 4612575523334377990 0 WHITESPACE UNKNOWN 4612575523334378501 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT_ATOMIC 4612575523334379029 0 L_PARENTHESIS PUNCTUATION ( 4612575523334379525 0 IDENTIFIER IDENTIFIER struct 4612575523334380038 0 WHITESPACE UNKNOWN 4612575523334380549 0 IDENTIFIER IDENTIFIER icmpmsg_mib 4612575523334381121 0 COMMA PUNCTUATION , 4612575523334381574 0 WHITESPACE UNKNOWN 4612575523334382085 0 IDENTIFIER IDENTIFIER icmpmsg_statistics 4612575523334382614 0 R_PARENTHESIS PUNCTUATION ) 4612575523334383166 0 SEMI PUNCTUATION ; 4612575523334383622 0 WHITESPACE UNKNOWN 4612575523334384194 0 HASH PUNCTUATION # 4612575523334385090 0 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334385158 0 WHITESPACE UNKNOWN 4612575523334385669 0 IDENTIFIER IDENTIFIER IS_ENABLED 4612575523334386197 0 L_PARENTHESIS PUNCTUATION ( 4612575523334386693 0 IDENTIFIER IDENTIFIER CONFIG_IPV6 4612575523334387222 0 R_PARENTHESIS PUNCTUATION ) 4612575523334387718 0 WHITESPACE UNKNOWN 4612575523334388230 0 WHITESPACE UNKNOWN 4612575523334388741 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334389269 0 L_PARENTHESIS PUNCTUATION ( 4612575523334389765 0 IDENTIFIER IDENTIFIER struct 4612575523334390278 0 WHITESPACE UNKNOWN 4612575523334390789 0 IDENTIFIER IDENTIFIER icmpv6_mib 4612575523334391361 0 COMMA PUNCTUATION , 4612575523334391814 0 WHITESPACE UNKNOWN 4612575523334392325 0 IDENTIFIER IDENTIFIER icmpv6_statistics 4612575523334392854 0 R_PARENTHESIS PUNCTUATION ) 4612575523334393406 0 SEMI PUNCTUATION ; 4612575523334393862 0 WHITESPACE UNKNOWN 4612575523334394374 0 WHITESPACE UNKNOWN 4612575523334394885 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT_ATOMIC 4612575523334395413 0 L_PARENTHESIS PUNCTUATION ( 4612575523334395909 0 IDENTIFIER IDENTIFIER struct 4612575523334396422 0 WHITESPACE UNKNOWN 4612575523334396933 0 IDENTIFIER IDENTIFIER icmpv6msg_mib 4612575523334397505 0 COMMA PUNCTUATION , 4612575523334397958 0 WHITESPACE UNKNOWN 4612575523334398469 0 IDENTIFIER IDENTIFIER icmpv6msg_statistics 4612575523334398998 0 R_PARENTHESIS PUNCTUATION ) 4612575523334399550 0 SEMI PUNCTUATION ; 4612575523334400006 0 WHITESPACE UNKNOWN 4612575523334400518 0 WHITESPACE UNKNOWN 4612575523334401029 0 IDENTIFIER IDENTIFIER struct 4612575523334401542 0 WHITESPACE UNKNOWN 4612575523334402053 0 IDENTIFIER IDENTIFIER proc_dir_entry 4612575523334402566 0 WHITESPACE UNKNOWN 4612575523334403102 0 STAR PUNCTUATION * 4612575523334403589 0 IDENTIFIER IDENTIFIER proc_net_devsnmp6 4612575523334404158 0 SEMI PUNCTUATION ; 4612575523334404614 0 WHITESPACE UNKNOWN 4612575523334405186 0 HASH PUNCTUATION # 4612575523334406089 0 PP_ENDIF MACRO_DIRECTIVE_NAME endif 4612575523334406150 0 WHITESPACE UNKNOWN 4612575523334406680 0 R_BRACE PUNCTUATION } 4612575523334407230 0 SEMI PUNCTUATION ; File data from file token range: struct netns_mib { DEFINE_SNMP_STAT(struct ipstats_mib, ip_statistics); #if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct ipstats_mib, ipv6_statistics); #endif DEFINE_SNMP_STAT(struct tcp_mib, tcp_statistics); DEFINE_SNMP_STAT(struct linux_mib, net_statistics); DEFINE_SNMP_STAT(struct udp_mib, udp_statistics); #if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct udp_mib, udp_stats_in6); #endif #ifdef CONFIG_XFRM_STATISTICS DEFINE_SNMP_STAT(struct linux_xfrm_mib, xfrm_statistics); #endif #if IS_ENABLED(CONFIG_TLS) DEFINE_SNMP_STAT(struct linux_tls_mib, tls_statistics); #endif #ifdef CONFIG_MPTCP DEFINE_SNMP_STAT(struct mptcp_mib, mptcp_statistics); #endif DEFINE_SNMP_STAT(struct udp_mib, udplite_statistics); #if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct udp_mib, udplite_stats_in6); #endif DEFINE_SNMP_STAT(struct icmp_mib, icmp_statistics); DEFINE_SNMP_STAT_ATOMIC(struct icmpmsg_mib, icmpmsg_statistics); #if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct icmpv6_mib, icmpv6_statistics); DEFINE_SNMP_STAT_ATOMIC(struct icmpv6msg_mib, icmpv6msg_statistics); struct proc_dir_entry *proc_net_devsnmp6; #endif }; Fused data from file and fragment token ranges: struct netns_mib { DEFINE_SNMP_STAT(struct ipstats_mib, ip_statistics); # if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct ipstats_mib, ipv6_statistics); # endif DEFINE_SNMP_STAT(struct tcp_mib, tcp_statistics); DEFINE_SNMP_STAT(struct linux_mib, net_statistics); DEFINE_SNMP_STAT(struct udp_mib, udp_statistics); # if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct udp_mib, udp_stats_in6); # endif # ifdef CONFIG_XFRM_STATISTICS DEFINE_SNMP_STAT(struct linux_xfrm_mib, xfrm_statistics); # endif # if IS_ENABLED(CONFIG_TLS) DEFINE_SNMP_STAT(struct linux_tls_mib, tls_statistics); # endif # ifdef CONFIG_MPTCP DEFINE_SNMP_STAT(struct mptcp_mib, mptcp_statistics); # endif DEFINE_SNMP_STAT(struct udp_mib, udplite_statistics); # if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct udp_mib, udplite_stats_in6); # endif DEFINE_SNMP_STAT(struct icmp_mib, icmp_statistics); DEFINE_SNMP_STAT_ATOMIC(struct icmpmsg_mib, icmpmsg_statistics); # if IS_ENABLED(CONFIG_IPV6) DEFINE_SNMP_STAT(struct icmpv6_mib, icmpv6_statistics); DEFINE_SNMP_STAT_ATOMIC(struct icmpv6msg_mib, icmpv6msg_statistics); struct proc_dir_entry *proc_net_devsnmp6; # endif }; Fused data from file and fragment token ranges: 12389402607343370240 0 KEYWORD_STRUCT KEYWORD struct 12170978025415901185 0 WHITESPACE UNKNOWN 12168726225602215938 9362983657750396928 IDENTIFIER STRUCT netns_mib 12170978025415901187 0 WHITESPACE UNKNOWN 12209258622248550404 0 L_BRACE PUNCTUATION { 12170978025415901189 0 WHITESPACE UNKNOWN 12170978025415901190 0 WHITESPACE UNKNOWN 12168726225602216087 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180056 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370393 0 KEYWORD_STRUCT KEYWORD struct 4612575523334275590 0 WHITESPACE UNKNOWN 12168726225602216090 9362983657707405312 IDENTIFIER STRUCT ipstats_mib 12303834214423330971 0 COMMA PUNCTUATION , 4612575523334277126 0 WHITESPACE UNKNOWN 12168726225602216092 9272911665202987008 IDENTIFIER INSTANCE_MEMBER ip_statistics 12207006822434865309 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275086 0 SEMI PUNCTUATION ; 12170978025415901199 0 WHITESPACE UNKNOWN 12306086014221287424 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13170777142676422657 13296877932242796544 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334280710 0 WHITESPACE UNKNOWN 12168726225586487298 13314892298382737408 IDENTIFIER MACRO_NAME IS_ENABLED 12204755022605451267 0 L_PARENTHESIS PUNCTUATION ( 12168726225586487300 13314892329535930368 IDENTIFIER MACRO_NAME CONFIG_IPV6 12207006822419136518 0 R_PARENTHESIS PUNCTUATION ) 12170978025415901200 0 WHITESPACE UNKNOWN 12170978025415901201 0 WHITESPACE UNKNOWN 12168726225602216109 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180078 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370415 0 KEYWORD_STRUCT KEYWORD struct 4612575523334285830 0 WHITESPACE UNKNOWN 12168726225602216112 9362983657707405312 IDENTIFIER STRUCT ipstats_mib 12303834214423330993 0 COMMA PUNCTUATION , 4612575523334287366 0 WHITESPACE UNKNOWN 12168726225602216114 9272911665202987009 IDENTIFIER INSTANCE_MEMBER ipv6_statistics 12207006822434865331 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275097 0 SEMI PUNCTUATION ; 12170978025415901210 0 WHITESPACE UNKNOWN 12306086014222336000 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13186539741373267969 13312640530939641856 PP_ENDIF MACRO_DIRECTIVE_NAME endif 12170978025415901211 0 WHITESPACE UNKNOWN 12170978025415901212 0 WHITESPACE UNKNOWN 12168726225602216131 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180100 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370437 0 KEYWORD_STRUCT KEYWORD struct 4612575523334293510 0 WHITESPACE UNKNOWN 12168726225602216134 9362983657719988224 IDENTIFIER STRUCT tcp_mib 12303834214423331015 0 COMMA PUNCTUATION , 4612575523334295046 0 WHITESPACE UNKNOWN 12168726225602216136 9272911665202987010 IDENTIFIER INSTANCE_MEMBER tcp_statistics 12207006822434865353 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275108 0 SEMI PUNCTUATION ; 12170978025415901221 0 WHITESPACE UNKNOWN 12170978025415901222 0 WHITESPACE UNKNOWN 12168726225602216153 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180122 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370459 0 KEYWORD_STRUCT KEYWORD struct 4612575523334299654 0 WHITESPACE UNKNOWN 12168726225602216156 9362983657724182528 IDENTIFIER STRUCT linux_mib 12303834214423331037 0 COMMA PUNCTUATION , 4612575523334301190 0 WHITESPACE UNKNOWN 12168726225602216158 9272911665202987011 IDENTIFIER INSTANCE_MEMBER net_statistics 12207006822434865375 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275118 0 SEMI PUNCTUATION ; 12170978025415901231 0 WHITESPACE UNKNOWN 12170978025415901232 0 WHITESPACE UNKNOWN 12168726225602216175 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180144 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370481 0 KEYWORD_STRUCT KEYWORD struct 4612575523334305798 0 WHITESPACE UNKNOWN 12168726225602216178 9362983657722085376 IDENTIFIER STRUCT udp_mib 12303834214423331059 0 COMMA PUNCTUATION , 4612575523334307334 0 WHITESPACE UNKNOWN 12168726225602216180 9272911665202987012 IDENTIFIER INSTANCE_MEMBER udp_statistics 12207006822434865397 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275128 0 SEMI PUNCTUATION ; 12170978025415901241 0 WHITESPACE UNKNOWN 12306086014223384576 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13170777142678519809 13296877932244893696 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334310918 0 WHITESPACE UNKNOWN 12168726225588584450 13314892298382737408 IDENTIFIER MACRO_NAME IS_ENABLED 12204755022607548419 0 L_PARENTHESIS PUNCTUATION ( 12168726225588584452 13314892329535930368 IDENTIFIER MACRO_NAME CONFIG_IPV6 12207006822421233670 0 R_PARENTHESIS PUNCTUATION ) 12170978025415901242 0 WHITESPACE UNKNOWN 12170978025415901243 0 WHITESPACE UNKNOWN 12168726225602216197 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180166 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370503 0 KEYWORD_STRUCT KEYWORD struct 4612575523334316038 0 WHITESPACE UNKNOWN 12168726225602216200 9362983657722085376 IDENTIFIER STRUCT udp_mib 12303834214423331081 0 COMMA PUNCTUATION , 4612575523334317574 0 WHITESPACE UNKNOWN 12168726225602216202 9272911665202987013 IDENTIFIER INSTANCE_MEMBER udp_stats_in6 12207006822434865419 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275139 0 SEMI PUNCTUATION ; 12170978025415901252 0 WHITESPACE UNKNOWN 12306086014224433152 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13186539741375365121 13312640530941739008 PP_ENDIF MACRO_DIRECTIVE_NAME endif 12170978025415901253 0 WHITESPACE UNKNOWN 12306086014225481728 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13173028942494302209 13299129732060676096 PP_IFDEF MACRO_DIRECTIVE_NAME ifdef 4612575523334322694 0 WHITESPACE UNKNOWN 12168726225590681602 0 IDENTIFIER IDENTIFIER CONFIG_XFRM_STATISTICS 4612575523334323718 0 WHITESPACE UNKNOWN 4612575523334324230 0 WHITESPACE UNKNOWN 4612575523334324741 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334325269 0 L_PARENTHESIS PUNCTUATION ( 4612575523334325765 0 IDENTIFIER IDENTIFIER struct 4612575523334326278 0 WHITESPACE UNKNOWN 4612575523334326789 0 IDENTIFIER IDENTIFIER linux_xfrm_mib 4612575523334327361 0 COMMA PUNCTUATION , 4612575523334327814 0 WHITESPACE UNKNOWN 4612575523334328325 0 IDENTIFIER IDENTIFIER xfrm_statistics 4612575523334328854 0 R_PARENTHESIS PUNCTUATION ) 4612575523334329406 0 SEMI PUNCTUATION ; 12170978025415901254 0 WHITESPACE UNKNOWN 12306086014226530304 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13186539741377462273 13312640530943836160 PP_ENDIF MACRO_DIRECTIVE_NAME endif 12170978025415901255 0 WHITESPACE UNKNOWN 12306086014227578880 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13170777142682714113 13296877932249088000 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334332934 0 WHITESPACE UNKNOWN 12168726225592778754 13314892298382737408 IDENTIFIER MACRO_NAME IS_ENABLED 12204755022611742723 0 L_PARENTHESIS PUNCTUATION ( 12168726225592778756 0 IDENTIFIER IDENTIFIER CONFIG_TLS 12207006822425427973 0 R_PARENTHESIS PUNCTUATION ) 4612575523334335494 0 WHITESPACE UNKNOWN 4612575523334336006 0 WHITESPACE UNKNOWN 4612575523334336517 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334337045 0 L_PARENTHESIS PUNCTUATION ( 4612575523334337541 0 IDENTIFIER IDENTIFIER struct 4612575523334338054 0 WHITESPACE UNKNOWN 4612575523334338565 0 IDENTIFIER IDENTIFIER linux_tls_mib 4612575523334339137 0 COMMA PUNCTUATION , 4612575523334339590 0 WHITESPACE UNKNOWN 4612575523334340101 0 IDENTIFIER IDENTIFIER tls_statistics 4612575523334340630 0 R_PARENTHESIS PUNCTUATION ) 4612575523334341182 0 SEMI PUNCTUATION ; 12170978025415901256 0 WHITESPACE UNKNOWN 12306086014228627456 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13186539741379559425 13312640530945933312 PP_ENDIF MACRO_DIRECTIVE_NAME endif 12170978025415901257 0 WHITESPACE UNKNOWN 12306086014229676032 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13173028942498496513 13299129732064870400 PP_IFDEF MACRO_DIRECTIVE_NAME ifdef 4612575523334344710 0 WHITESPACE UNKNOWN 12168726225594875906 0 IDENTIFIER IDENTIFIER CONFIG_MPTCP 4612575523334345734 0 WHITESPACE UNKNOWN 4612575523334346246 0 WHITESPACE UNKNOWN 4612575523334346757 0 IDENTIFIER IDENTIFIER DEFINE_SNMP_STAT 4612575523334347285 0 L_PARENTHESIS PUNCTUATION ( 4612575523334347781 0 IDENTIFIER IDENTIFIER struct 4612575523334348294 0 WHITESPACE UNKNOWN 4612575523334348805 0 IDENTIFIER IDENTIFIER mptcp_mib 4612575523334349377 0 COMMA PUNCTUATION , 4612575523334349830 0 WHITESPACE UNKNOWN 4612575523334350341 0 IDENTIFIER IDENTIFIER mptcp_statistics 4612575523334350870 0 R_PARENTHESIS PUNCTUATION ) 4612575523334351422 0 SEMI PUNCTUATION ; 12170978025415901258 0 WHITESPACE UNKNOWN 12306086014230724608 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13186539741381656577 13312640530948030464 PP_ENDIF MACRO_DIRECTIVE_NAME endif 12170978025415901259 0 WHITESPACE UNKNOWN 12170978025415901260 0 WHITESPACE UNKNOWN 12168726225602216219 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180188 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370525 0 KEYWORD_STRUCT KEYWORD struct 4612575523334355974 0 WHITESPACE UNKNOWN 12168726225602216222 9362983657722085376 IDENTIFIER STRUCT udp_mib 12303834214423331103 0 COMMA PUNCTUATION , 4612575523334357510 0 WHITESPACE UNKNOWN 12168726225602216224 9272911665202987014 IDENTIFIER INSTANCE_MEMBER udplite_statistics 12207006822434865441 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275156 0 SEMI PUNCTUATION ; 12170978025415901269 0 WHITESPACE UNKNOWN 12306086014231773184 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13170777142686908417 13296877932253282304 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334361094 0 WHITESPACE UNKNOWN 12168726225596973058 13314892298382737408 IDENTIFIER MACRO_NAME IS_ENABLED 12204755022615937027 0 L_PARENTHESIS PUNCTUATION ( 12168726225596973060 13314892329535930368 IDENTIFIER MACRO_NAME CONFIG_IPV6 12207006822429622278 0 R_PARENTHESIS PUNCTUATION ) 12170978025415901270 0 WHITESPACE UNKNOWN 12170978025415901271 0 WHITESPACE UNKNOWN 12168726225602216241 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180210 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370547 0 KEYWORD_STRUCT KEYWORD struct 4612575523334366214 0 WHITESPACE UNKNOWN 12168726225602216244 9362983657722085376 IDENTIFIER STRUCT udp_mib 12303834214423331125 0 COMMA PUNCTUATION , 4612575523334367750 0 WHITESPACE UNKNOWN 12168726225602216246 9272911665202987015 IDENTIFIER INSTANCE_MEMBER udplite_stats_in6 12207006822434865463 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275167 0 SEMI PUNCTUATION ; 12170978025415901280 0 WHITESPACE UNKNOWN 12306086014232821760 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13186539741383753729 13312640530950127616 PP_ENDIF MACRO_DIRECTIVE_NAME endif 12170978025415901281 0 WHITESPACE UNKNOWN 12170978025415901282 0 WHITESPACE UNKNOWN 12168726225602216263 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180232 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370569 0 KEYWORD_STRUCT KEYWORD struct 4612575523334373894 0 WHITESPACE UNKNOWN 12168726225602216266 9362983657709502464 IDENTIFIER STRUCT icmp_mib 12303834214423331147 0 COMMA PUNCTUATION , 4612575523334375430 0 WHITESPACE UNKNOWN 12168726225602216268 9272911665202987016 IDENTIFIER INSTANCE_MEMBER icmp_statistics 12207006822434865485 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275178 0 SEMI PUNCTUATION ; 12170978025415901291 0 WHITESPACE UNKNOWN 12170978025415901292 0 WHITESPACE UNKNOWN 12168726225602216285 13314892330745987072 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT_ATOMIC 12204755022621180254 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370591 0 KEYWORD_STRUCT KEYWORD struct 4612575523334380038 0 WHITESPACE UNKNOWN 12168726225602216288 9362983657711599616 IDENTIFIER STRUCT icmpmsg_mib 12303834214423331169 0 COMMA PUNCTUATION , 4612575523334381574 0 WHITESPACE UNKNOWN 12168726225602216290 9272911665202987017 IDENTIFIER INSTANCE_MEMBER icmpmsg_statistics 12207006822434865507 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275188 0 SEMI PUNCTUATION ; 12170978025415901301 0 WHITESPACE UNKNOWN 12306086014233870336 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13170777142689005569 13296877932255379456 PP_IF MACRO_DIRECTIVE_NAME if 4612575523334385158 0 WHITESPACE UNKNOWN 12168726225599070210 13314892298382737408 IDENTIFIER MACRO_NAME IS_ENABLED 12204755022618034179 0 L_PARENTHESIS PUNCTUATION ( 12168726225599070212 13314892329535930368 IDENTIFIER MACRO_NAME CONFIG_IPV6 12207006822431719430 0 R_PARENTHESIS PUNCTUATION ) 12170978025415901302 0 WHITESPACE UNKNOWN 12170978025415901303 0 WHITESPACE UNKNOWN 12168726225602216301 13314892330744938496 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT 12204755022621180270 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370607 0 KEYWORD_STRUCT KEYWORD struct 4612575523334390278 0 WHITESPACE UNKNOWN 12168726225602216304 9362983657712648192 IDENTIFIER STRUCT icmpv6_mib 12303834214423331185 0 COMMA PUNCTUATION , 4612575523334391814 0 WHITESPACE UNKNOWN 12168726225602216306 9272911665202987018 IDENTIFIER INSTANCE_MEMBER icmpv6_statistics 12207006822434865523 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275199 0 SEMI PUNCTUATION ; 12170978025415901312 0 WHITESPACE UNKNOWN 12170978025415901313 0 WHITESPACE UNKNOWN 12168726225602216323 13314892330745987072 IDENTIFIER MACRO_NAME DEFINE_SNMP_STAT_ATOMIC 12204755022621180292 0 L_PARENTHESIS PUNCTUATION ( 12389402607343370629 0 KEYWORD_STRUCT KEYWORD struct 4612575523334396422 0 WHITESPACE UNKNOWN 12168726225602216326 9362983657715793920 IDENTIFIER STRUCT icmpv6msg_mib 12303834214423331207 0 COMMA PUNCTUATION , 4612575523334397958 0 WHITESPACE UNKNOWN 12168726225602216328 9272911665202987019 IDENTIFIER INSTANCE_MEMBER icmpv6msg_statistics 12207006822434865545 0 R_PARENTHESIS PUNCTUATION ) 12297078814982275209 0 SEMI PUNCTUATION ; 12170978025415901322 0 WHITESPACE UNKNOWN 12170978025415901323 0 WHITESPACE UNKNOWN 12389402607343370380 0 KEYWORD_STRUCT KEYWORD struct 12170978025415901325 0 WHITESPACE UNKNOWN 12168726225602216078 9362983646608228352 IDENTIFIER STRUCT proc_dir_entry 12170978025415901327 0 WHITESPACE UNKNOWN 12225021220944347280 0 STAR PUNCTUATION * 12168726225602216081 9272911665202987020 IDENTIFIER INSTANCE_MEMBER proc_net_devsnmp6 12297078814982275218 0 SEMI PUNCTUATION ; 12170978025415901331 0 WHITESPACE UNKNOWN 12306086014234918912 0 HASH PUNCTUATION # 0 0 WHITESPACE UNKNOWN 13186539741385850881 13312640530952224768 PP_ENDIF MACRO_DIRECTIVE_NAME endif 12170978025415901332 0 WHITESPACE UNKNOWN 12211510422062235797 0 R_BRACE PUNCTUATION } 12297078814982275222 0 SEMI PUNCTUATION ; ```

Assuming we're interested in that first identifier for struct netns_mib, 9362983657750396928:

./mx-highlight-entity --db /home/user0/linux.db --entity_id 9362983657750396928

         /home/user0/sources/linux/include/net/netns/mib.h
         +---------------------------------------------
       7 | struct netns_mib {
       8 |  DEFINE_SNMP_STAT(struct ipstats_mib, ip_statistics);
       9 | #if IS_ENABLED(CONFIG_IPV6)
      10 |  DEFINE_SNMP_STAT(struct ipstats_mib, ipv6_statistics);
      11 | #endif
      12 |
      13 |  DEFINE_SNMP_STAT(struct tcp_mib, tcp_statistics);
      14 |  DEFINE_SNMP_STAT(struct linux_mib, net_statistics);
      15 |
      16 |  DEFINE_SNMP_STAT(struct udp_mib, udp_statistics);
      17 | #if IS_ENABLED(CONFIG_IPV6)
      18 |  DEFINE_SNMP_STAT(struct udp_mib, udp_stats_in6);
      19 | #endif
      20 |
      21 | #ifdef CONFIG_XFRM_STATISTICS
      22 |  DEFINE_SNMP_STAT(struct linux_xfrm_mib, xfrm_statistics);
      23 | #endif
      24 | #if IS_ENABLED(CONFIG_TLS)
      25 |  DEFINE_SNMP_STAT(struct linux_tls_mib, tls_statistics);
      26 | #endif
      27 | #ifdef CONFIG_MPTCP
      28 |  DEFINE_SNMP_STAT(struct mptcp_mib, mptcp_statistics);
      29 | #endif
      30 |
      31 |  DEFINE_SNMP_STAT(struct udp_mib, udplite_statistics);
      32 | #if IS_ENABLED(CONFIG_IPV6)
      33 |  DEFINE_SNMP_STAT(struct udp_mib, udplite_stats_in6);
      34 | #endif
      35 |
      36 |  DEFINE_SNMP_STAT(struct icmp_mib, icmp_statistics);
      37 |  DEFINE_SNMP_STAT_ATOMIC(struct icmpmsg_mib, icmpmsg_statistics);
      38 | #if IS_ENABLED(CONFIG_IPV6)
      39 |  DEFINE_SNMP_STAT(struct icmpv6_mib, icmpv6_statistics);
      40 |  DEFINE_SNMP_STAT_ATOMIC(struct icmpv6msg_mib, icmpv6msg_statistics);
      41 |  struct proc_dir_entry *proc_net_devsnmp6;
      42 | #endif
      43 | };
pgoodman commented 3 months ago

Alright, if you're willing to share, then can you run this script, which was added in the most recent commit to the main branch, which will extract out all files relevant to compiling this specific fragment. I can try to reproduce things locally and see if that helps.

$ python3.12 -m venv /path/to/install
$ source /path/to/install/bin/activate
(install) $ python /path/t

If you haven't built the Python bindings, then you should be able to download the latest pre-built linux release and then run the following:

$ python3.12 -m venv /path/to/extracted/release
$ source /path/to/extracted/release/bin/activate
(release) $ python /path/to/multiplier/source/CompressCompilation.py --db linux.db --entity_id 2305843009214773472 --working_dir /tmp/issue_565

Then that will create a directory /tmp/issue_565 full of copies of the relevant files from your build, and it will also create /tmp/issue_565.tar.gz. If you can attach that tarball to this issue then that would be great!

bdemick commented 3 months ago

Much appreciated - thanks! Here you go: issue_565.tar.gz

pgoodman commented 3 months ago

Alright I tried reproducing it on macOS to no avail. I was able to make the database and find the relevant structures. I'll be trying next on Linux. It's possible that the issue cannot be isolated to this one translation unit, and manifests as a function of cross-translation unit deduplication.

pgoodman commented 3 months ago

There was also a curiosity in your compile_commands.json output that I'll need to try to reproduce and chase down; this was listed as one of the command-line arguments:

      "-I\"-resource-dir /usr/lib/llvm-14/lib/clang/14.0.0",

This is most likely a PASTA issue, however. For context, PASTA is the Clang compiler wrapper that Multiplier relies on.

bdemick commented 3 months ago

Yeah I think that was part of the steps prescribed to generate the index here: https://github.com/trailofbits/multiplier/blob/main/docs/INDEXING.md, just happened to be the system clang that was installed.

Another curiosity (and what started me down this path) is that the binary release I installed segfaults, and the debug build I have been testing with aborts. It looks like it's happening with the same translation units. I think the binary release I was testing with on Linux is multiplier-c18052b.tar.xz.

pgoodman commented 3 months ago

It would make sense for it to sigsegv here where it is, unfortunately. In general, the multiplier codebase follows the practice of opportunistic assertions, i.e. ones that can have a fail-safe fallback in a release build. However, there are some places where fail-safes are impossible, and you're hitting one. For example, if a method returned std::optional<Decl> then the fail-safe is to

assert(false);
return std::nullopt;

In your case, the return value is meant to be a complete Decl object, and so there is no value that we can substitute, and so when it goes and does follow-up invocations on the Decl, it uses a null pointer. The assertion failure is just a slightly earlier manifestation of the same underlying problem.

bdemick commented 3 months ago

Ah of course - had completely slipped my mind that assert gets nop'd in release builds.

bdemick commented 3 months ago

Just for sanity's sake, I built a release mulitplier on my local Mac and am still seeing a segfault in the same spot.

pgoodman commented 3 months ago

Alright, I'm unable to reproduce things on mac/linux with just the tarball provided. If you're willing to, can you share your Linux kbuild config, and the git hash of the version of the kernel you cloned, and if it wasn't the primary torvalds one, then the repo URL? You can optionally share over email to peter ampersand trailofbits period com.

Finally, out of curiosity, have you been satisfied with the time it takes to index code? Were there unexpected hiccups along the way? General feedback on the usability of the tools would be helpful.

bdemick commented 3 months ago

If I recall correctly, it took me roughly 2 hours to generate the index for this Linux kernel, which I think is definitely well within an acceptable and usable timeframe. I used Sourcetrail (RIP) for this a few years ago and it took quite a long time (at least a day, IIRC). I've so far been very impressed with both the baseline examples/utilities. I think a brief API primer would be super useful and help to grok the codebase (especially for those of us with severely out-of-date C++ experience).

All in all, this is pretty awesome. It sure beats grep + Intellisense in terms of flexibility!

pgoodman commented 3 months ago

I think the Python API is probably where you want to live. When installed, the Python API installs some stub files, which can be used with PyCharm to support auto-completion. A nice thing about using the Python API, which is just as powerful as the C++ API, is that it auto-downcasts things, unlike in C++ where you need to do things like FunctionDecl::from.

pgoodman commented 3 months ago

Alright, I am working to reproduce things. Here's my current approach:

wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.9.10.tar.xz
tar xf linux-6.9.10.tar.xz
cd linux-6.9.10
CCC_OVERRIDE_OPTIONS="# x-Werror" CC=`which clang-14` LD=`which ld.lld-14` make LLVM=1 defconfig
CCC_OVERRIDE_OPTIONS="# x-Werror" CC=`which clang-14` LD=`which ld.lld-14` make LLVM=1 -j48
env > env_vars.txt
git clone git@github.com:amezin/vscode-linux-kernel.git .vscode
python .vscode/generate_compdb.py
mx-index --db linux.db --workspace linux.ws --target compile_commands.json --env env_vars.txt

I'm using an debug build of mx-index just in case something interesting pops up there.

pgoodman commented 3 months ago

Reproduced :-D

pgoodman commented 3 months ago

I think the issue is this. There is a declaration roughly like the following:

struct Foo {
  __typeof__(struct Bar) *bar;
};

And suppose there is not yet a forward declaration of struct Bar. Then the struct Bar within the __typeof__ will act as the forward declarator and canonical declaration for Bar. This seems a significant enough difference from the normal case we face, which is more akin to:

struct Foo {
  struct Bar *bar;
};

In this latter "normal" case, Multiplier has special logic to "hoist" that struct Bar forward declaration into its own floating fragment (the unit of code deduplication in multiplier). It does this because deduplication is partially AST structure aware. Later, there's a fixpoint step to go and add stuff to the fragment. This is a simplified explanation, but I think the bug lives somewhere in the hoisting logic not triggering, and there being different translation units where struct Bar is declared before struct Foo. I'll need to investigate further.

bdemick commented 3 months ago

Interesting. Glad you’re able to reproduce! The Linux kernel is great for edge cases 😂

pgoodman commented 2 months ago

I think I have this fixed in my branch. Along the way I noticed some things and fixed them too, and then scope creeped into adding some features lol. Hopefully by tomorrow there's be an updated main for you to test. One important note: you will need to re-generate your databases.

bdemick commented 2 months ago

Right on! I'll give that a shot as soon as it drops. You're the 💣!

pgoodman commented 2 months ago

You might overall want to do a fresh clone & rebuild, because I've changed a submodule (VAST) to point to a branch of the main repo, and I've bumped the PASTA submodule. It's also easy to go in and pull the latest there. If you go that route, then to rebuild, do the following within the build directory.

rm -rf vendor/install/vast/include
pushd vendor/vast/build ; ninja install ; popd
pushd vendor/pasta/build ; ninja install ; popd
ninja install

Let me know if the issue is resolved for you! If so, then I can close out this issue :-D

bdemick commented 2 months ago

Currently struggling to complete a clean build at the moment. Keep getting a bunch of undefined reference errors when linking:

ld.lld: error: lib/libmultiplier.so: undefined reference to llvm::EnableABIBreakingChecks [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to llvm::ilist_detail::SpecificNodeAccess<llvm::ilist_detail::node_options<mlir::Operation, true, false, void, false> >::getNodePtr(mlir::Operation*) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to llvm::ilist_detail::SpecificNodeAccess<llvm::ilist_detail::node_options<mlir::Operation, true, false, void, false> >::getValuePtr(llvm::ilist_node_impl<llvm::ilist_detail::node_options<mlir::Operation, true, false, void, false> >*) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to mlir::RewriterBase::inlineRegionBefore(mlir::Region&, mlir::Region&, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Block, true, false, void, false>, false, false>) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to mlir::RewriterBase::cloneRegionBefore(mlir::Region&, mlir::Region&, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Block, true, false, void, false>, false, false>, mlir::IRMapping&) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to mlir::RewriterBase::inlineBlockBefore(mlir::Block*, mlir::Block*, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Operation, true, false, void, false>, false, false>, mlir::ValueRange) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to mlir::RewriterBase::splitBlock(mlir::Block*, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Operation, true, false, void, false>, false, false>) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to llvm::ilist_traits<mlir::Block>::transferNodesFromList(llvm::ilist_traits<mlir::Block>&, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Block, true, false, void, false>, false, false>, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Block, true, false, void, false>, false, false>) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to llvm::ilist_traits<mlir::Operation>::transferNodesFromList(llvm::ilist_traits<mlir::Operation>&, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Operation, true, false, void, false>, false, false>, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Operation, true, false, void, false>, false, false>) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to mlir::ConversionPatternRewriter::inlineRegionBefore(mlir::Region&, mlir::Region&, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Block, true, false, void, false>, false, false>) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to mlir::OpBuilder::createBlock(mlir::Region*, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Block, true, false, void, false>, false, false>, mlir::TypeRange, llvm::ArrayRef<mlir::Location>) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to mlir::RewriterBase::cloneRegionBefore(mlir::Region&, mlir::Region&, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Block, true, false, void, false>, false, false>) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to mlir::ConversionPatternRewriter::splitBlock(mlir::Block*, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Operation, true, false, void, false>, false, false>) [--no-allow-shlib-undefined]
ld.lld: error: lib/libmultiplier.so: undefined reference to mlir::Block::splitBlock(llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Operation, true, false, void, false>, false, false>) [--no-allow-shlib-undefined]
clang++-18: error: linker command failed with exit code 1 (use -v to see invocation)
[1580/3736] Building CXX object bin/Examples/C...iles/mx-list-functions.dir/ListFunctions.cpp.o
ninja: build stopped: subcommand failed.

This is coming after blowing away everything in build, install, and src and starting from scratch from the instructions in BUILD.md. Not sure exactly what's going on here.

pgoodman commented 2 months ago

Hrmm okay there's going to be a missing library references somewhere. I will try a fresh rebuild later today and see if I can replicate this.

bdemick commented 2 months ago

Retrying the build from scratch again just in case there is an environment thing going on. I had a few ssh sessions to the box I'm working on and it's possible I messed something up.

Regardless - I grabbed the Linux binary release, generated a new index from that, and ran mx-find-linked-structures on it, and it ran to completion, so that is awesome! Will keep you posted on the build issue.

pgoodman commented 2 months ago

My guess is that it has to do with a debug build. I've got a new branch for it, with a minor change to the LLVM configuration. I'm doing a fresh debug build and will see if I can repro it.

bdemick commented 2 months ago

I think it was an environment thing on my end - just got a debug build to succeed on Linux, and mx-find-linked-structures from that build just ran to completion, so I think we can put a bow on this one.

Thank you again for all your support, and this excellent tool! It's on my Github watch list now, and I'll definitely be prodding at it some more as well as figuring out the Python bindings.