search

Vadims06 / isiswatcher

History of changes in IS-IS domain
Other
4 stars 0 forks source link

Watcher crash #3

Open bleuchtang opened 1 day ago

bleuchtang commented 1 day ago

Some times isis watcher crash with the followings errors:

IS-IS watcher encounted the following error during sniffing: Traceback (most recent call last):                                                                                                                                              
  File "/home/watcher/watcher/pysniff.py", line 22, in start_sniffing                                                 
    isis_sniffer.start_sniffing()                                                                                                                                                            
  File "pysniff_helper.pyx", line 255, in pysniff_helper.ISIS_INTERFACE_SNIFFER.start_sniffing                                                                                                                                               
  File "/usr/local/lib/python3.11/site-packages/scapy/sendrecv.py", line 1311, in sniff                               
    sniffer._run(*args, **kwargs)                                                                                                                                                            
  File "/usr/local/lib/python3.11/site-packages/scapy/sendrecv.py", line 1254, in _run 
    session.on_packet_received(p)
  File "/usr/local/lib/python3.11/site-packages/scapy/sessions.py", line 109, in on_packet_received                                                                                                                                          
    result = self.prn(pkt)       
             ^^^^^^^^^^^^^                                                                                                                                                                   
  File "pysniff_helper.pyx", line 241, in pysniff_helper.ISIS_INTERFACE_SNIFFER.packet_callback                                                                                                                                              
  File "pysniff_helper.pyx", line 176, in pysniff_helper.ISIS_INTERFACE_SNIFFER.isis_packet_process                                                                                                                                          
  File "Helper.pyx", line 1642, in Helper.PDU.do_parse_isis_lsp_packet                                                                                                                       
  File "/usr/local/lib/python3.11/site-packages/scapy/packet.py", line 438, in getfieldval                                                                                                   
    return self.payload.getfieldval(attr)                                                     
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                     
  File "/usr/local/lib/python3.11/site-packages/scapy/packet.py", line 1760, in getfieldval                           
    raise AttributeError(attr)           
AttributeError: metric                                                                        

found fresh graph dump, skip doing another one
found fresh packet dump, skip doing another one                                                                       
Traceback (most recent call last):            
  File "/home/watcher/watcher/pysniff.py", line 22, in start_sniffing
    isis_sniffer.start_sniffing() 
  File "pysniff_helper.pyx", line 255, in pysniff_helper.ISIS_INTERFACE_SNIFFER.start_sniffing                                                                                                                                               
  File "/usr/local/lib/python3.11/site-packages/scapy/sendrecv.py", line 1311, in sniff                               
    sniffer._run(*args, **kwargs)                                                                                                                                                            
  File "/usr/local/lib/python3.11/site-packages/scapy/sendrecv.py", line 1254, in _run 
    session.on_packet_received(p)
  File "/usr/local/lib/python3.11/site-packages/scapy/sessions.py", line 109, in on_packet_received                                                                                                                                          
    result = self.prn(pkt)       
             ^^^^^^^^^^^^^                                                                                                                                                                   
  File "pysniff_helper.pyx", line 241, in pysniff_helper.ISIS_INTERFACE_SNIFFER.packet_callback                                                                                                                                              
  File "pysniff_helper.pyx", line 176, in pysniff_helper.ISIS_INTERFACE_SNIFFER.isis_packet_process                                                                                                                                          
  File "Helper.pyx", line 1642, in Helper.PDU.do_parse_isis_lsp_packet                                                                                                                       
  File "/usr/local/lib/python3.11/site-packages/scapy/packet.py", line 438, in getfieldval                                                                                                   
    return self.payload.getfieldval(attr)                                                     
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                     
  File "/usr/local/lib/python3.11/site-packages/scapy/packet.py", line 1760, in getfieldval                           
    raise AttributeError(attr)           
AttributeError: metric                                                                        

During handling of the above exception, another exception occurred:                                                   

Traceback (most recent call last):                         
  File "/home/watcher/watcher/pysniff.py", line 46, in <module>                                                       
    start_sniffing()                                       
  File "/home/watcher/watcher/pysniff.py", line 43, in start_sniffing                                                 
    raise ValueError(str(e))                               
ValueError: metric

As the restart policy is set to always in docker, watcher container restart without a word.

Vadims06 commented 1 day ago

Hi Emily, great job in troubleshooting, thank for deep dive into it!) The error comes from the part where I parsed IS-IS pdu packet in Helper.PDU.do_parse_isis_lsp_packet function. For some reason python scapy library couldn't parse the metric. Watcher saves broken LSP and verbose error in local logs folder. May I ask you to share such dumps with me sending them to admin @ topolograph.com or text me in direct message t.me/Vadims06