when i run prep_filter-maxdiff-trace.py,i get up with this problem,could you tell me how to solve it

[aoaowu]

the pcap file i used is https://github.com/netplier-tool/NetPlier/blob/master/data/smb2_100.pcap

[skleber]

When I call the named script with your , it gives me the following detailed output before raising the exception:

nemesys$ python src/prep_filter-maxdiff-trace.py input/smb2_100.pcap
Loading input/smb2_100.pcap
Segmentation by inflections of sigma-1.2-gauss-filtered bit-variance.
Refine segmentation (zero-slices refinements)...
Refine segmentation (base refinements)...
Filter messages...
Count Segment values...
Determine messages' commonalities...
Filtered in 0.02 s
Get groundtruth from tshark...
tshark: Error loading table 'User DLTs Table': user_dlts:2: dissector not found
Ignored sub field: ip.dsfield_tree
Ignored sub field: ip.flags_tree
Ignored sub field: tcp.flags_tree
Ignored sub field: Timestamps

 Known message dissection is
---------------------------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vlan.priority                00
delimiter                    4608
vlan.dei                     00
vlan.id                      46
vlan.etype                   0800
ip.version                   45
ip.hdr_len                   45
ip.dsfield                   00
ip.len                       0118
ip.id                        63e4
ip.flags                     40
ip.frag_offset               00
ip.ttl                       80
ip.proto                     06
ip.checksum                  39ba
ip.src                       ac120210
ip.addr                      ac120210
ip.src_host                  ac120210
ip.host                      ac120210
ip.dst                       ac12020d
ip.addr                      ac12020d
ip.dst_host                  ac12020d
ip.host                      ac12020d
tcp.srcport                  01bd
tcp.dstport                  c9e4
tcp.port                     01bd
tcp.port                     c9e4
tcp.len                      50
tcp.seq                      9e481be8
tcp.seq_raw                  9e481be8
tcp.seq                      2655525864
tcp.ack                      8d9ce782
tcp.ack_raw                  8d9ce782
tcp.ack                      2375870338
tcp.hdr_len                  50
tcp.flags                    18
tcp.window_size_value        0100
tcp.window_size              0100
tcp.window_size_scalefactor  0100
tcp.checksum                 de14
tcp.urgent_pointer           0000
tcp.payload                  000000ecfe534d424000000000000000000001000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004100010002020000ea9ec13c48364748b05d903d9c9e1136010000000000010000000100000001001e3c4f571487cc016e1dcebeca85cc0180006c00204c4d20606a06062b0601050502a060305ea030302e06092a864882f71201020206092a864886f712010202060a2a864886f71201020203060a2b06010401823702020aa32a3028a0261b246e6f745f646566696e65645f696e5f5246433431373840706c656173655f69676e6f7265
nbss.type                    00
nbss.length                  0000ec
---------------------------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Dissection is incomplete. (Compare self.getFieldValues() and self.protocolbytes):
Dissector result: 00460800460800454500011863e44000800639baac120210ac120210ac120210ac120210ac12020dac12020dac12020dac12020d01bdc9e401bdc9e4509e481be89e481be826555258648d9ce7828d9ce78223758703385018010001000100de140000000000ecfe534d424000000000000000000001000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004100010002020000ea9ec13c48364748b05d903d9c9e1136010000000000010000000100000001001e3c4f571487cc016e1dcebeca85cc0180006c00204c4d20606a06062b0601050502a060305ea030302e06092a864882f71201020206092a864886f712010202060a2a864886f71201020203060a2b06010401823702020aa32a3028a0261b246e6f745f646566696e65645f696e5f5246433431373840706c656173655f69676e6f7265000000ec
Original  packet: 00460800

The reason is that the script tries to interpret the Wireshark dissection of this packet to generate some statistics about the filtering result. The dissection is however not complete. This is likely due to a reassembled TCP payload in this case (but could also be some SMB fields for which I did not write an interpretation rule in nemere.validation.messageParser.ParsingConstants226).

Using ground truth this way is not necessary for the core function of the script and as a quick fix you could replace line 245 with except (NotImplementedError, DissectionIncomplete) as e: in src/prep_filter-maxdiff-trace.py (to ignore the exception). You will still get the notification about the incomplete dissection and the ipython shell, which you can simply close and the script will continue. I will provide this solution as a future commit sometime.

Another solution may be to remove the offending message from the trace in this case.

[skleber]

As a side note: You are trying to filter a trace of 100 messages into a list of 100 messages (which essentially will do nothing) or less with the -p parameter. Less than 100 messages is probably a rather low number of messages for any subsequent analysis in most cases. I recommend to try a larger trace (1000s of messages) and filter that down to 100 or a few hundred messages with the prep_filter-maxdiff-trace.py script.

[aoaowu]

thank you!!!!

[aoaowu]

sorry, after i replace the file nemetyl_align-segments.py,line 384 with except (NotImplementedError, DissectionIncomplete) as e: , and i run it with the file smb2_100_maxdiff-valcom-100 which handled by prep_filter-maxdiff-trace.py, i get up with the problem below . So I want to know if this tool can handle normal smb2 files from normal packet captures properly? or only it can handle over 100 of messages or special file which has handled ?

Initial clustering of the segments in the trace failed. The protocol in this trace cannot be inferred. The original exceptionmessage was:
 Dissection is incomplete:
Dissector result: 00460800460800044500011863e440004000800639baac120210ac120210ac120210ac120210ac12020dac12020dac12020dac12020d01bdc9e401bdc9e4509e481be89e481be826555258648d9ce7828d9ce78223758703385018010001000100de140000000000ecfe534d424000000000000000000001000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004100010002020000ea9ec13c48364748b05d903d9c9e1136010000000000010000000100000001001e3c4f571487cc016e1dcebeca85cc0180006c00204c4d20606a06062b0601050502a060305ea030302e06092a864882f71201020206092a864886f712010202060a2a864886f71201020203060a2b06010401823702020aa32a3028a0261b246e6f745f646566696e65645f696e5f5246433431373840706c656173655f69676e6f7265000000ec

[skleber]

In general, all of the algorithms work with any unknown protocol. However, since the program code was primarily written to do evaluations of the quality of the inference compared to ground truth, in many places dissector results are used to calculate the accuracy of the analysis. In all of those cases, you can safely comment out the offending calls to MessageComparator or other evaluation-specific components. I will look into you specific issue that you describe to provide a generic solution. However, this may take some time. Thus, you are welcome to propose pull requests to speed up the process.

[aoaowu]

thank you very much!

[skleber]

Feel free to provide a PR anytime.