search

f5devcentral / f5-corkscrew

TMOS parser and application extractor
Apache License 2.0
13 stars 3 forks source link

[RFE] include xml stats in extraction #15

Open DumpySquare opened 3 years ago

DumpySquare commented 3 years ago

We have been talking for some time about including stats in the extraction process. These stats, like iHealth, could give valuable insight to the system and applications being parsed.

This information could be used for migrations, upgrades and troubleshooting

qkview file locations to look at including and parsing

actually, we should probably just parse every .xml file in the base directory and see what we can get from there 

{
  qkModule: {
    index: {},
    afm: {},
    apm: {},
    asm: {},
    avr: {},
    gtm: {},
    ilx: {},
    ...,
  }
}

parsing xml to json

https://stackoverflow.com/questions/52281389/convert-xml-to-json-with-nodejs/57724779

Command to search qkview files

tar xaf some_viprion.qkview --to-command 'egrep -Hn --label="$TAR_FILENAME" "Ltm::Virtual Server:" || true'

tar xaf coreltm01_2.20.2021.qkview --to-command 'egrep -Hn --label="$TAR_FILENAME" "bigiq.benlab.io_t443_vs" || true' >> bigiqVsSearch.txt

@sergitopereira, I know you have already started some of this, but wanted to get it documented so we can track the details together. Do you have any input on where you were getting some of these details from qkviews? I've reworked the unPacker function to extract more config files like certs/keys/gtm/defaults. This is probably a good opportunity to start extracting the necessary files for stats.

DumpySquare commented 3 years ago 
const htmlString = `
show /ltm virtual all-properties
-----------------------------------------------------------
Ltm::Virtual Server: /Common/VS-146.20.176.0-443
-----------------------------------------------------------
Status                         
  Availability     : offline
  State            : enabled
  Reason           : The children pool member(s) are down
  CMP              : enabled
  CMP Mode         : all-cpus
  Destination      : 172.24.201.0:443
  PVA Acceleration : none
Traffic                          ClientSide   Ephemeral      General
  Bits In                               960           0            -
  Bits Out                             1.8K           0            -
  Packets In                              3           0            -
  Packets Out                             5           0            -
  Current Connections                     0           0            -
  Maximum Connections                     2           0            -
  Total Connections                       2           0            -           
  Min Conn Duration/msec                  -           -        45.1K
  Max Conn Duration/msec                  -           -        45.1K
  Mean Conn Duration/msec                 -           -        45.1K
  Total Requests                          -           -            0
SYN Cookies                    
  Status                         not-activated
  Hardware SYN Cookie Instances           0
  Software SYN Cookie Instances           0
  Current SYN Cache                       0
  SYN Cache Overflow                      0
  Total Software                          0
  Total Software Accepted                 0
  Total Software Rejected                 0
  Total Hardware                          0
  Total Hardware Accepted                 0
CPU Usage Ratio (%)            
  Last 5 Seconds                          0
  Last 1 Minute                           0
  Last 5 Minutes                          0
-----------------------------------------------------------
Ltm::Virtual Server: /Common/VS-146.20.176.0-80
-----------------------------------------------------------
Status                         
  Availability     : offline
  State            : enabled
  Reason           : The children pool member(s) are down
  CMP              : enabled
  CMP Mode         : all-cpus
  Destination      : 172.24.201.0:80
  PVA Acceleration : none
Traffic                          ClientSide   Ephemeral      General
  Bits In                              3.2K           0            -
  Bits Out                             2.1K           0            -
  Packets In                             10           0            -
  Packets Out                             6           0            -
  Current Connections                     0           0            -
  Maximum Connections                     2           0            -
  Total Connections                       5           0            -           
  Min Conn Duration/msec                  -           -         3.3K
  Max Conn Duration/msec                  -           -         3.3K
  Mean Conn Duration/msec                 -           -         3.3K
  Total Requests                          -           -            0
SYN Cookies                    
  Status                         not-activated
  Hardware SYN Cookie Instances           0
  Software SYN Cookie Instances           0
  Current SYN Cache                       0
  SYN Cache Overflow                      0
  Total Software                          0
  Total Software Accepted                 0
  Total Software Rejected                 0
  Total Hardware                          0
  Total Hardware Accepted                 0
CPU Usage Ratio (%)            
  Last 5 Seconds                          0
  Last 1 Minute                           0
  Last 5 Minutes                          0
DumpySquare commented 3 years ago

stat_module.xml is what we are looking for

<?xml version="1.0" encoding="UTF-8"?>
<Qkproc version="1.6">
    <cluster>
        <virtual_server_stat>
            <object name="/Common/test_listener_u53_vs">
                <destination>00:00:00:00:00:00:00:00:00:00:FF:FF:C0:A8:C8:07:00:00:00:00</destination>
                <source>00:00:00:00:00:00:00:00:00:00:FF:FF:00:00:00:00:00:00:00:00</source>
                <vs_index>10</vs_index>
                <cs_min_conn_dur>28</cs_min_conn_dur>
                <cs_max_conn_dur>89545</cs_max_conn_dur>
                <cs_mean_conn_dur>31</cs_mean_conn_dur>
                <no_nodes_errors>0</no_nodes_errors>
                <cycle_count>174746574144</cycle_count>
                <delta_busy>92000481469</delta_busy>
                <delta_cycles>0</delta_cycles>
                <proxy_init_failed>0</proxy_init_failed>
                <no_acl_match>12024843</no_acl_match>
                <no_acl_match_accept>12024843</no_acl_match_accept>
                <no_acl_match_drop>0</no_acl_match_drop>
                <no_acl_match_reject>0</no_acl_match_reject>
                <last_hit_time_enforced_acl>1613904974</last_hit_time_enforced_acl>
                <no_staged_acl_match>12024843</no_staged_acl_match>
                <no_staged_acl_match_accept>12024843</no_staged_acl_match_accept>
                <no_staged_acl_match_drop>0</no_staged_acl_match_drop>
                <no_staged_acl_match_reject>0</no_staged_acl_match_reject>
                <last_hit_time_staged_acl>1613904974</last_hit_time_staged_acl>
                <clientside.pkts_in>12026403</clientside.pkts_in>
                <clientside.bytes_in>840913487</clientside.bytes_in>
                <clientside.pkts_out>10596362</clientside.pkts_out>
                <clientside.bytes_out>1367370210</clientside.bytes_out>
                <clientside.max_conns>469</clientside.max_conns>
                <clientside.tot_conns>12026403</clientside.tot_conns>
                <clientside.cur_conns>0</clientside.cur_conns>
                <clientside.slow_conns>0</clientside.slow_conns>
                <clientside.expired_conns>5888</clientside.expired_conns>
                <clientside.slow_killed>0</clientside.slow_killed>
                <clientside.evicted_conns>0</clientside.evicted_conns>
                <clientside.other_killed>0</clientside.other_killed>
                <pva.pkts_in>0</pva.pkts_in>
                <pva.bytes_in>0</pva.bytes_in>
                <pva.pkts_out>0</pva.pkts_out>
                <pva.bytes_out>0</pva.bytes_out>
                <pva.max_conns>0</pva.max_conns>
                <pva.tot_conns>0</pva.tot_conns>
                <pva.cur_conns>0</pva.cur_conns>
                <ephemeral.pkts_in>0</ephemeral.pkts_in>
                <ephemeral.bytes_in>0</ephemeral.bytes_in>
                <ephemeral.pkts_out>0</ephemeral.pkts_out>
                <ephemeral.bytes_out>0</ephemeral.bytes_out>
                <ephemeral.max_conns>0</ephemeral.max_conns>
                <ephemeral.tot_conns>0</ephemeral.tot_conns>
                <ephemeral.cur_conns>0</ephemeral.cur_conns>
                <tot_requests>11995333</tot_requests>
                <tot_pva_assist_conn>0</tot_pva_assist_conn>
                <curr_pva_assist_conn>0</curr_pva_assist_conn>
                <serverside.pkts_in>9015523</serverside.pkts_in>
                <serverside.bytes_in>1282304581</serverside.bytes_in>
                <serverside.pkts_out>9026826</serverside.pkts_out>
                <serverside.bytes_out>818575849</serverside.bytes_out>
                <serverside.max_conns>469</serverside.max_conns>
                <serverside.tot_conns>9026826</serverside.tot_conns>
                <serverside.cur_conns>0</serverside.cur_conns>
                <serverside.slow_conns>0</serverside.slow_conns>
                <serverside.expired_conns>5931</serverside.expired_conns>
                <serverside.slow_killed>0</serverside.slow_killed>
                <serverside.evicted_conns>0</serverside.evicted_conns>
                <serverside.other_killed>0</serverside.other_killed>
                <precision.tot_sync>0</precision.tot_sync>
                <precision.tot_async>0</precision.tot_async>
                <precision.delayed>0</precision.delayed>
                <precision.tot_rejected>0</precision.tot_rejected>
                <precision.tot_retry>0</precision.tot_retry>
                <precision.is_locked>0</precision.is_locked>
                <precision.tot_locked>0</precision.tot_locked>
                <precision.tot_unlocked>0</precision.tot_unlocked>
                <precision.last_locked>0</precision.last_locked>
                <precision.dura_locked>0</precision.dura_locked>
                <precision.latency>0</precision.latency>
                <precision.last_value>0</precision.last_value>
                <precision.value>0</precision.value>
                <syncookies.hwsyncookie_inst>0</syncookies.hwsyncookie_inst>
                <syncookies.swsyncookie_inst>0</syncookies.swsyncookie_inst>
                <syncookies.syncache_curr>0</syncookies.syncache_curr>
                <syncookies.syncache_over>0</syncookies.syncache_over>
                <syncookies.syncookies>0</syncookies.syncookies>
                <syncookies.accepts>0</syncookies.accepts>
                <syncookies.rejects>0</syncookies.rejects>
                <syncookies.hw_syncookies>0</syncookies.hw_syncookies>
                <syncookies.hw_accepts>0</syncookies.hw_accepts>
                <syncookies.wl_hits>0</syncookies.wl_hits>
                <syncookies.wl_accepts>0</syncookies.wl_accepts>
                <syncookies.wl_rejects>0</syncookies.wl_rejects>
                <sc_curr_valid0>0</sc_curr_valid0>
                <sc_curr_sent0>0</sc_curr_sent0>
                <sc_curr_valid1>0</sc_curr_valid1>
                <sc_curr_sent1>0</sc_curr_sent1>
                <hwdns.hwdns_inst>0</hwdns.hwdns_inst>
                <hwdns.swdns_inst>0</hwdns.swdns_inst>
                <hwdns.vlans_hw>0</hwdns.vlans_hw>
                <hwdns.vlans_sw>0</hwdns.vlans_sw>
                <hwdns.back_inserts>0</hwdns.back_inserts>
                <hwdns.back_evicts>0</hwdns.back_evicts>
                <hwdns.request_hits>0</hwdns.request_hits>
                <no_fw_nat_policy>12026403</no_fw_nat_policy>
                <no_fw_nat_rule_match>0</no_fw_nat_rule_match>
                <fw_nat_match_failures>0</fw_nat_match_failures>
                <stress_stats.stress1s>0</stress_stats.stress1s>
                <stress_stats.rexmttimeo>0</stress_stats.rexmttimeo>
                <stress_stats.sndprobe>0</stress_stats.sndprobe>
                <stress_stats.rcvecncepack>0</stress_stats.rcvecncepack>
                <stress_stats.abandons>0</stress_stats.abandons>
                <stress_stats.rxunreach>0</stress_stats.rxunreach>
                <stress_stats.serversynrtx>0</stress_stats.serversynrtx>
                <stress_stats.handshaketimeout>0</stress_stats.handshaketimeout>
                <stress_stats.expires>11819</stress_stats.expires>
                <stress_stats.dns_req>0</stress_stats.dns_req>
                <stress_stats.dns_resp>0</stress_stats.dns_resp>
                <stress_stats.context_type>0</stress_stats.context_type>
                <hornet.client_syn_received>0</hornet.client_syn_received>
                <hornet.normal_teardowns_completed>0</hornet.normal_teardowns_completed>
                <hornet.duplicate_syn_received>0</hornet.duplicate_syn_received>
                <hornet.client_rst_received>0</hornet.client_rst_received>
                <hornet.server_rst_received>0</hornet.server_rst_received>
                <hornet.port_retry_count>0</hornet.port_retry_count>
                <hornet.way_collision_count>0</hornet.way_collision_count>
                <hornet.way_full_count>0</hornet.way_full_count>
                <hornet.client_pkt_unknown_flow>0</hornet.client_pkt_unknown_flow>
                <hornet.server_pkt_unknown_flow>0</hornet.server_pkt_unknown_flow>
                <hornet.connection_created>0</hornet.connection_created>
                <hornet.embryonic_promoted>0</hornet.embryonic_promoted>
                <hornet.client_pkt_expired_embryonic>0</hornet.client_pkt_expired_embryonic>
                <hornet.server_pkt_expired_embryonic>0</hornet.server_pkt_expired_embryonic>
                <hornet.client_pkt_expired_active>0</hornet.client_pkt_expired_active>
                <hornet.server_pkt_expired_active>0</hornet.server_pkt_expired_active>
                <hornet.client_pkt_expired_timewait>0</hornet.client_pkt_expired_timewait>
                <hornet.server_pkt_expired_timewait>0</hornet.server_pkt_expired_timewait>
                <hornet.action_drop>0</hornet.action_drop>
                <hornet.action_reset>0</hornet.action_reset>
                <hornet.action_xform>0</hornet.action_xform>
                <hornet.action_fwd>0</hornet.action_fwd>
            </object>
        </virtual_server_stat>
        <virtual_server_cpu_stat>
            <object name="/Common/test_listener_u53_vs">
                <avg_5sec>0</avg_5sec>
                <avg_1min>0</avg_1min>
                <avg_5min>0</avg_5min>
                <slot_id>0</slot_id>
            </object>
        </virtual_server_cpu_stat>
DumpySquare commented 3 years ago

@sergitopereira - all the stats are in those xml files. I've updated the loadParse function to convert all the xml files to json and build a tree with all the details. Like the config tree it should be easily searchable and we can build reports or extract all the information from there. This is a ton of data...

DumpySquare commented 1 year ago

I've got an initial version of this complete and pending for the next release