search

fmadio / pcap2json

High Speed PCAP to JSON conversion utility
Other
97 stars 21 forks source link

User specified template #21

Closed fmadio closed 4 years ago

fmadio commented 5 years ago

The output format dosent not have to be in JSON format, as the runtime code only writes strings and integers into fixed spaced addresses.

As such any template could be used for the output format. Enabling more variation on output formats, both JSON, CSV, and pure binary output. Technically could output IPFIX in this way.

The overhead should be quite minimal

fmadio commented 4 years ago

Committed, examples are

Format is @KEY#CHARLENGTH#

pcap2json.config

--flow-template "{'timestamp':@TIMESTAMP#16#,'FlowCnt':@FLOWCNT#10#,'Device':@DEVICE#32#,'hash':@HASH#42#,'TotalPkt':@TOTALPKT#12#,'TotalByte':@TOTALBYTE#12#,'TotalBits':@TOTALBIT#12#,'TotalFCS':@TOTALFCS#8#,'MACSrc':@MACSRC#19#,'MACDst':@MACDST#19#,'MACProto':@MACPROTO#8#,'VLAN.0':@VLAN0#8#,'VLAN.1':@VLAN1#8#,'MPLS.0.Label':@MPLS0LABEL#8#,'MPLS.0.TC':@MPLS0TC#4#,'MPLS.1.Label':@MPLS1LABEL#8#,'MPLS.1.TC':@MPLS1TC#4#,'MPLS.2.Label':@MPLS2LABEL#8#,'MPLS.2.TC':@MPLS2TC#4#,'IPv4.Src':@IPV4SRC#20#,'IPv4.Dst':@IPV4DST#20#,'IPv4.Proto':@IPV4PROTO#8#,'IPv4.DSCP':@IPV4DSCP#8#,'UDP.Port.Src':@UDPPORTSRC#8#,'UDP.Port.Dst':@UDPPORTDST#8#,'TCP.Port.Src':@TCPPORTSRC#8#,'TCP.Port.Dst':@TCPPORTDST#8#,'TCP.FIN':@TCPFIN#8#,'TCP.SYN':@TCPSYN#8#,'TCP.RST':@TCPRST#8#,'TCP.PSH':@TCPPSH#8#,'TCP.ACK':@TCPACK#8#,'TCP.WindowMin':@TCPWINMIN#8#,'TCP.WindowMax':@TCPWINMAX#8#,'TCP.SACK':@TCPSACK#8#}"

pcap2json.lua config file

"--flow-template \"{'timestamp':@TIMESTAMP#16#,'FlowCnt':@FLOWCNT#10#,'Device':@DEVICE#32#,'hash':@HASH#42#,'TotalPkt':@TOTALPKT#12#,'TotalByte':@TOTALBYTE#12#,'TotalBits':@TOTALBIT#12#,'TotalFCS':@TOTALFCS#8#,'MACSrc':@MACSRC#19#,'MACDst':@MACDST#19#,'MACProto':@MACPROTO#8#,'VLAN.0':@VLAN0#8#,'VLAN.1':@VLAN1#8#,'MPLS.0.Label':@MPLS0LABEL#8#,'MPLS.0.TC':@MPLS0TC#4#,'MPLS.1.Label':@MPLS1LABEL#8#,'MPLS.1.TC':@MPLS1TC#4#,'MPLS.2.Label':@MPLS2LABEL#8#,'MPLS.2.TC':@MPLS2TC#4#,'IPv4.Src':@IPV4SRC#20#,'IPv4.Dst':@IPV4DST#20#,'IPv4.Proto':@IPV4PROTO#8#,'IPv4.DSCP':@IPV4DSCP#8#,'UDP.Port.Src':@UDPPORTSRC#8#,'UDP.Port.Dst':@UDPPORTDST#8#,'TCP.Port.Src':@TCPPORTSRC#8#,'TCP.Port.Dst':@TCPPORTDST#8#,'TCP.FIN':@TCPFIN#8#,'TCP.SYN':@TCPSYN#8#,'TCP.RST':@TCPRST#8#,'TCP.PSH':@TCPPSH#8#,'TCP.ACK':@TCPACK#8#,'TCP.WindowMin':@TCPWINMIN#8#,'TCP.WindowMax':@TCPWINMAX#8#,'TCP.SACK':@TCPSACK#8#}\"",