telekom / 5g-trace-visualizer

This set of Python scripts allow you to convert pcap, pcapnp or pdml 5G protocol traces (Wireshark, tcpdump, ...) into SVG sequence diagrams.
Apache License 2.0
272 stars 77 forks source link

Mime multipart boundary regex #31

Closed rtommy closed 1 year ago

rtommy commented 1 year ago

The current regex for multipart boundary (and content-type, content-id) does not consider '_' char while it can occur:

--oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC

mime_multipart_payloadregex should be extended by adding '' for the boundary.

jkolom commented 1 year ago

It should now work. At the time I only tested with the boundaries I saw. Glad to have more examples 😃

rtommy commented 1 year ago

What version of Wireshark do you use? I had 3.6.2 in Ubuntu but today I updated to 4.0.5. It seems there were changes because I get new errors. For example, it seems that the Content-Type and Content-ID in a multipart has swapped sequence. Therefore, the mime_multipart_payload_regex does not work for me anymore.

<field name="mime_multipart.boundary" showname="Boundary: \r\n--oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC\r\n" size="70" pos="1163" show="--oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC" value="0d0a2d2d6f694837554a43726d347645574d58373064497369714653415942646d65367a6e3339574d5a6674755f315f4d39443830416a686f3163544c53496b4d5653430d0a"/>

<field name="mime_multipart.part" showname="Encapsulated multipart part: (application/vnd.3gpp.5gnas)" size="71" pos="1233" show="" value="436f6e74656e742d49443a206e31536d4d73670d0a436f6e74656e742d547970653a206170706c69636174696f6e2f766e642e336770702e35676e61730d0a0d0a2e0501c10000">

<field name="mime_multipart.header.content-id" showname="Content-ID: n1SmMsg\r\n" size="21" pos="1233" show="n1SmMsg" value="436f6e74656e742d49443a206e31536d4d73670d0a"/>

<field name="mime_multipart.header.content-type" showname="Content-Type: application/vnd.3gpp.5gnas\r\n\r\n" size="44" pos="1254" show="application/vnd.3gpp.5gnas" value="436f6e74656e742d547970653a206170706c69636174696f6e2f766e642e336770702e35676e61730d0a0d0a"/>

rtommy commented 1 year ago

Also, the "Content-Id" has become "Content-ID" and I had to update the regex too.

rtommy commented 1 year ago
--oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC
Content-Type: application/json

{"supi":"imsi-520038000000001","unauthenticatedSupi":false,"pei":"imeisv-1234000000230901","gpsi":"msisdn-66943000001","pduSessionId":5,"dnn":"internet.er","sNssai":{"sst":1},"servingNfId":"5314d1a4-8015-445f-89a1-1f435529b2c0","guami":{"plmnId":{"mcc":"520","mnc":"03"},"amfId":"809101"},"servingNetwork":{"mcc":"520","mnc":"03"},"requestType":"INITIAL_REQUEST","n1SmMsg":{"contentId":"n1SmMsg"},"anType":"3GPP_ACCESS","ratType":"NR","ueLocation":{"nrLocation":{"tai":{"plmnId":{"mcc":"520","mnc":"03"},"tac":"009002"},"ncgi":{"plmnId":{"mcc":"520","mnc":"03"},"nrCellId":"003E007C7"},"ueLocationTimestamp":"2023-04-26T08:47:52Z"}},"ueTimeZone":"+07:00","smContextStatusUri":"http:\/\/10.206.108.65:8080\/callbacks\/nsmf-pdusession\/v1\/imsi-520038000000001\/sm-contexts\/-576460752302638364","selMode":"VERIFIED","epsInterworkingInd":"NONE"}
--oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC
Content-ID: n1SmMsg
Content-Type: application/vnd.3gpp.5gnas

.
--oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC--
rtommy commented 1 year ago

The error I get:

Traceback (most recent call last):
  File "trace_visualizer.py", line xxx, in parse_http_proto_stream
    header_length = multipart_lengths[idx][0] * 2
IndexError: list index out of range

It is because the m_all won't have both multiparts, only the first one while enumerate(split_payload) will have both.

Only 1 item in the list: m_all = [m for m in mime_multipart_payload_regex.finditer(data_ascii)]

Only 1 item in the list (due to m_all): multipart_lengths = [(len(m.group(1)), len(m.group(6))) for m in m_all]

split_payload will have botn multiparts: split_str = '2d2d' + boundary_hex split_payload = data_hex.split(split_str)[1:-1]

                        for idx, payload in enumerate(split_payload):
                            header_length = multipart_lengths[idx][0] * 2
                            length_to_cut = header_length - len(split_str)
                            logging.debug('Removing {0} additional bytes'.format(length_to_cut))
                            payload_clean = payload[length_to_cut:]
jkolom commented 1 year ago

Hi, I just tried some old traces with wireshark 4.0.5 and it worked, at least with the test traces I have. The regex is not used in such a way. Normally, Wireshark can decode MIME multipart by itself and it is not needed. The regex is used in some cases when the headers and the body are not in the same TCP frame. In such cases Wireshark cannot decode the multipart MIMO and the directly parses the payload and tries to make sense of it. An example would be a payload like this:

  <proto name="http2" showname="HyperText Transfer Protocol 2" size="802" pos="66">
    <field name="http2.stream" showname="Stream: DATA, Stream ID: 23, Length 793" size="802" pos="66" show="" value="">
      <field name="http2.length" showname="Length: 793" size="3" pos="66" show="793" value="000319"/>
      <field name="http2.type" showname="Type: DATA (0)" size="1" pos="69" show="0" value="00"/>
      <field name="http2.flags" showname="Flags: 0x01" size="1" pos="70" show="0x00000001" value="01">
        <field name="http2.flags.end_stream" showname=".... ...1 = End Stream: True" size="1" pos="70" show="1" value="1" unmaskedvalue="01"/>
        <field name="http2.flags.padded" showname=".... 0... = Padded: False" size="1" pos="70" show="0" value="0" unmaskedvalue="01"/>
        <field name="http2.flags.unused_data" showname="0000 .00. = Unused: 0x00" size="1" pos="70" show="0x00000000" value="0" unmaskedvalue="01"/>
      </field>
      <field name="http2.r" showname="0... .... .... .... .... .... .... .... = Reserved: 0x0" size="4" pos="71" show="0x00000000" value="0" unmaskedvalue="00000017"/>
      <field name="http2.streamid" showname=".000 0000 0000 0000 0000 0000 0001 0111 = Stream Identifier: 23" size="4" pos="71" show="23" value="17" unmaskedvalue="00000017"/>
      <field name="http2.pad_length" showname="Pad Length: 0" size="0" pos="75" show="0"/>
      <field name="http2.data.data" showname="Data: [...]" size="793" pos="75" show="2d:[...]:0a" value="2d2d676330704a7130386a55353334630d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a0d0a7b226e314d657373616765436f6e7461696e6572223a7b226e314d657373616765436c617373223a22534d222c226e314d657373616765436f6e74656e74223a7b22636f6e74656e744964223a226e31436f6e74656e74496431227d7d2c226e32496e666f436f6e7461696e6572223a7b226e32496e666f726d6174696f6e436c617373223a22534d222c22736d496e666f223a7b2270647553657373696f6e4964223a352c226e32496e666f436f6e74656e74223a7b226e676170496554797065223a225044555f5245535f53455455505f524551222c226e67617044617461223a7b22636f6e74656e744964223a226e32436f6e74656e74496431227d7d2c22734e73736169223a7b22737374223a317d7d7d2c226c6173744d7367496e6469636174696f6e223a66616c73652c2270647553657373696f6e4964223a352c226e316e324661696c7572655478664e6f746966555249223a22687474703a2f2f3137322e32372e342e333a36353530392f6e616d662d636f6d6d2f76312f75652d636f6e74657874732f3030303535313130222c22736d665265616c6c6f636174696f6e496e64223a66616c73657d0d0a2d2d676330704a7130386a55353334630d0a436f6e74656e742d547970653a206170706c69636174696f6e2f766e642e336770702e35676e61730d0a436f6e74656e742d49643a206e31436f6e74656e744964310d0a0d0a2e[...]740d0a2d2d676330704a7130386a55353334630d0a436f6e74656e742d547970653a206170706c69636174696f6e2f766e642e336770702e6e6761700d0a436f6e74656e742d49643a206e32436f6e74656e744964310d0a0d0a0000040[...]04000d0a2d2d676330704a7130386a55353334632d2d0d0a"/>
    </field>
  </proto>

The ASCII conversion of the value attribute is

--gc0pJq08jU534c
Content-Type: application/json

{"n1MessageContainer":{"n1MessageClass":"SM","n1MessageContent":{"contentId":"n1ContentId1"}},"n2InfoContainer":{"n2InformationClass":"SM","smInfo":{"pduSessionId":5,"n2InfoContent":{"ngapIeType":"PDU_RES_SETUP_REQ","ngapData":{"contentId":"n2ContentId1"}},"sNssai":{"sst":1}}},"lastMsgIndication":false,"pduSessionId":5,"n1n2FailureTxfNotifURI":"http://172.27.4.3:65509/namf-comm/v1/ue-contexts/00055110","smfReallocationInd":false}
--gc0pJq08jU534c
Content-Type: application/vnd.3gpp.5gnas
Content-Id: n1ContentId1

<binary data>

--gc0pJq08jU534c
Content-Type: application/vnd.3gpp.ngap
Content-Id: n2ContentId1

<binary data>

Which is then shown as

SMF to AMF
172.27.4.3 to 172.27.3.22 (IPs)
HTTP/2 stream 23 payload
Parsed multipart payload (missing header?)
application/json (No Content ID)
{
"n1MessageContainer": {
"n1MessageClass": "SM",
"n1MessageContent": {
"contentId": "n1ContentId1"
}
},
"n2InfoContainer": {
"n2InformationClass": "SM",
"smInfo": {
"pduSessionId": 5,
"n2InfoContent": {
"ngapIeType": "PDU_RES_SETUP_REQ",
"ngapData": {
"contentId": "n2ContentId1"
}
},
"sNssai": {
"sst": 1
}
}
},
"lastMsgIndication": false,
"pduSessionId": 5,
"n1n2FailureTxfNotifURI": "http://172.27.4.3:65509/namf-comm/v1/ue-contexts/00055110",
"smfReallocationInd": false
}
application/vnd.3gpp.5gnas (n1ContentId1)
2e[...]74
application/vnd.3gpp.ngap (n2ContentId1)
0000040[...]0400

The parsed NAS message is then visible in a subsequent NGAP message that Wireshark can decode.

Is there a chance you could share a pcap or pdml file with at least a couple of frames? Trying to reproduce your error would help with finding what exactly is happening.

rtommy commented 1 year ago

it is a bit complicated... the trace I have does not have IP proto in the packets, only "exported_pdu" because the capture was done within a 5GC product and not the normal way (capturing on an interface). Therefore, your original script does not work without some modifications to add the exported_pdu protocol into it :)

rtommy commented 1 year ago

one_packet one_packet.zip

rtommy commented 1 year ago

in my original pcap this was packet #44 which I extracted and attached.

DEBUG:root:44: IPv4 Src: 10.206.108.65
DEBUG:root:Frame 44: Joined 0 prior HTTP/2 fragments
DEBUG:root:Frame 44: could not parse HTTP/2 payload data as JSON
DEBUG:root:Found 1 MIME-multiparts by scanning payload. Boundary: "oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC (128 bytes)".
  Parts found: application/json (No Content ID)
DEBUG:root:Frame 44: Processing multipart message. Boundary= oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC (0x6f694837554a43726d347645574d58373064497369714653415942646d65367a6e3339574d5a6674755f315f4d394438304
16a686f3163544c53496b4d565343)
DEBUG:root:Frame 44: Found 2 MIME parts by scanning dissected data
DEBUG:root:Frame 44: Part 1: application/json
DEBUG:root:Frame 44: Part data: 436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a0d0a7b2273757069223a22696d73692d353230303338303030303030303031222c22756e61757468656e7469636174656453757069223a66616c73652c2270656922
3a22696d656973762d31323334303030303030323330393031222c2267707369223a226d736973646e2d3636393433303030303031222c2270647553657373696f6e4964223a352c22646e6e223a22696e7465726e65742e6572222c22734e73736169223a7b22737374223a317d2c227365
7276696e674e664964223a2235333134643161342d383031352d343435662d383961312d316634333535323962326330222c226775616d69223a7b22706c6d6e4964223a7b226d6363223a22353230222c226d6e63223a223033227d2c22616d664964223a22383039313031227d2c227365
7276696e674e6574776f726b223a7b226d6363223a22353230222c226d6e63223a223033227d2c227265717565737454797065223a22494e495449414c5f52455155455354222c226e31536d4d7367223a7b22636f6e74656e744964223a226e31536d4d7367227d2c22616e54797065223a
22334750505f414343455353222c2272617454797065223a224e52222c2275654c6f636174696f6e223a7b226e724c6f636174696f6e223a7b22746169223a7b22706c6d6e4964223a7b226d6363223a22353230222c226d6e63223a223033227d2c22746163223a22303039303032227d2c
226e636769223a7b22706c6d6e4964223a7b226d6363223a22353230222c226d6e63223a223033227d2c226e7243656c6c4964223a22303033453030374337227d2c2275654c6f636174696f6e54696d657374616d70223a22323032332d30342d32365430383a34373a35325a227d7d2c22
756554696d655a6f6e65223a222b30373a3030222c22736d436f6e74657874537461747573557269223a22687474703a5c2f5c2f31302e3230362e3130382e36353a383038305c2f63616c6c6261636b735c2f6e736d662d70647573657373696f6e5c2f76315c2f696d73692d3532303033
383030303030303030315c2f736d2d636f6e74657874735c2f2d353736343630373532333032363338333634222c2273656c4d6f6465223a225645524946494544222c22657073496e746572776f726b696e67496e64223a224e4f4e45227d
DEBUG:root:Frame 44: Part 2: application/vnd.3gpp.5gnas
DEBUG:root:Frame 44: Part data: 436f6e74656e742d49443a206e31536d4d73670d0a436f6e74656e742d547970653a206170706c69636174696f6e2f766e642e336770702e35676e61730d0a0d0a2e0501c10000
DEBUG:root:Manual boundary parsing (maybe missing header due to HPACK?). Using boundary oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC (0x6f694837554a43726d347645574d58373064497369714653415942646d65367a6e333957
4d5a6674755f315f4d39443830416a686f3163544c53496b4d565343)
DEBUG:root:Total payload length: 2312 bytes, 2266 characters. Header and payload length: 102/844
DEBUG:root:Removing 72 additional bytes
DEBUG:root:Could not manually parse payload
Traceback (most recent call last):
  File "trace_visualizer.py", line xxx, in parse_http_proto_stream
    header_length = multipart_lengths[idx][0] * 2
IndexError: list index out of range
rtommy commented 1 year ago

the puml file has the message properly, it is just that error above I did not have before....

note right of "FWD" #b3b3b3
**AMF2 to SMF**
10.206.108.65 to 10.206.108.92 (IPs)
HTTP/2 stream: 89
:scheme: http
:path: /nsmf-pdusession/v1/sm-contexts
:method: POST
:authority: 10.206.108.92:7070
content-type: multipart/related; boundary=oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC
content-length: 1156

~--First part: JSON:~--
{
    "supi": "imsi-520038000000001",
    "unauthenticatedSupi": false,
    "pei": "imeisv-1234000000230901",
    "gpsi": "msisdn-66943000001",
    "pduSessionId": 5,
    "dnn": "internet.er",
    "sNssai": {
        "sst": 1
    },
    "servingNfId": "5314d1a4-8015-445f-89a1-1f435529b2c0",
    "guami": {
        "plmnId": {
            "mcc": "520",
            "mnc": "03"
        },
        "amfId": "809101"
    },
    "servingNetwork": {
        "mcc": "520",
        "mnc": "03"
    },
    "requestType": "INITIAL_REQUEST",
    "n1SmMsg": {
        "contentId": "n1SmMsg"
    },
    "anType": "3GPP_ACCESS",
    "ratType": "NR",
    "ueLocation": {
        "nrLocation": {
            "tai": {
                "plmnId": {
                    "mcc": "520",
                    "mnc": "03"
                },
                "tac": "009002"
            },
            "ncgi": {
                "plmnId": {
                    "mcc": "520",
                    "mnc": "03"
                },
                "nrCellId": "003E007C7"
            },
            "ueLocationTimestamp": "2023-04-26T08:47:52Z"
        }
    },
    "ueTimeZone": "+07:00",
    "smContextStatusUri": "http://10.206.108.65:8080/callbacks/nsmf-pdusession/v1/imsi-520038000000001/sm-contexts/-576460752302638364",
    "selMode": "VERIFIED",
    "epsInterworkingInd": "NONE"
}

~--Next part: application/vnd.3gpp.5gnas. Content ID: n1SmMsg~--
Data:436f6e74656e742d49443a206e31536d4d73670d0a436f6e74656e742d547970653a206170706c69636174696f6e2f766e642e336770702e35676e61730d0a0d0a2e0501c10000
Non-Access-Stratum 5GS (NAS)PDU:
    Plain NAS 5GS Message:
        nas_5gs.epd: 'Extended protocol discriminator: 5G session management messages (46)'
        nas_5gs.pdu_session_id: 'PDU session identity: PDU session identity value 5 (5)'
        nas_5gs.proc_trans_id: 'Procedure transaction identity: 1'
        nas_5gs.sm.message_type: 'Message type: PDU session establishment request (0xc1)'
        Integrity protection maximum data rate:
            nas_5gs.sm.int_prot_max_data_rate_ul: 'Integrity protection maximum data rate for uplink: 64 kbps (0)'
            nas_5gs.sm.int_prot_max_data_rate_dl: 'Integrity protection maximum data rate for downlink: 64 kbps (0)'

end note
jkolom commented 1 year ago

So the PUML is getting generated but you get an error? Or do you mean that it now works?

rtommy commented 1 year ago

PUML is OK but I get that python error for multiple packets and I did not have any error before WS update. TShark (Wireshark) 4.0.3 (Git v4.0.3 packaged as 4.0.3-1~ubuntu22.04.0~ppa1).

jkolom commented 1 year ago

I did not manage today, but I think I have some idea now how/what to do

rtommy commented 1 year ago

take your time. This is not a shows-stopper and luckily has no impact so far. :)

rtommy commented 1 year ago

I have another example for you.

In this case there is a "Content-Transfer-Encoding" field too beside Content-Type and Content-ID.

Notes:

cte

content-transfer-encoding.zip

It was packet #64 in my trace.

DEBUG:root:Frame 64: Joined 0 prior HTTP/2 fragments
DEBUG:root:Frame 64: could not parse HTTP/2 payload data as JSON
DEBUG:root:Found 2 MIME-multiparts by scanning payload. Boundary: "MultipartDataListBoundary (50 bytes)".
  Parts found: application/json (No Content ID), application/vnd.3gpp.ngap (No Content ID)
DEBUG:root:Frame 64: Processing multipart message. Boundary= MultipartDataListBoundary (0x4d756c746970617274446174614c697374426f756e64617279)
DEBUG:root:Frame 64: Found 2 MIME parts by scanning dissected data
DEBUG:root:Frame 64: Part 1: application/json
DEBUG:root:Frame 64: Part data: 436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a436f6e74656e742d49443a206a736f6e0d0a436f6e74656e742d5472616e736665722d456e636f64696e673a20737472696e670d0a0d0a7b22686f5374617465223a
22505245504152494e47222c226e32536d496e666f223a7b22636f6e74656e744964223a2250647553657373696f6e5265736f757263655365747570526571756573745472616e73666572227d2c226e32536d496e666f54797065223a225044555f5245535f53455455505f524551227d  
DEBUG:root:Frame 64: could not parse HTTP/2 payload data as JSON
DEBUG:root:Frame 64: Part 2: application/vnd.3gpp.ngap
DEBUG:root:Frame 64: Part data: 436f6e74656e742d547970653a206170706c69636174696f6e2f766e642e336770702e6e6761700d0a436f6e74656e742d49443a2050647553657373696f6e5265736f757263655365747570526571756573745472616e736665720d0a436f6e7465
6e742d5472616e736665722d456e636f64696e673a2062696e6172790d0a0d0a0000040082000c1004a817c8004002540be400008b000a01f00a3832613150031f00860001000088000700050000050000
DEBUG:root:Manual boundary parsing (maybe missing header due to HPACK?). Using boundary MultipartDataListBoundary (0x4d756c746970617274446174614c697374426f756e64617279)
DEBUG:root:Total payload length: 962 bytes, 7810 characters. Header and payload length: 61/17, 70/51
DEBUG:root:Removing 68 additional bytes
DEBUG:root:Frame 64: could not parse HTTP/2 payload data as JSON
DEBUG:root:Removing 86 additional bytes
DEBUG:root:Frame 64: could not parse HTTP/2 payload data as JSON

The PUML:

note right of "10.56.20.102" #e6e6e6
**SMF to AMF**
10.56.20.96 to 192.168.189.83 (IPs)
HTTP/2 stream: 5
Request in: 63
:status: 200
content-type: multipart/related; boundary=MultipartDataListBoundary
content-length: 481

Parsed multipart payload (missing header?)

application/json (No Content ID)
436f6e74656e742d49443a206a736f6e0d0a436f6e74656e742d5472616e736665722d456e636f64696e673a20737472696e670d0a0d0a7b22686f5374617465223a22505245504152494e47222c226e32536d496e666f223a7b22636f6e74656e744964223a2250647553657373696f6e5265736f757263655365747570526571756573745472616e73666572227d2c226e32536d496e666f54797065223a225044555f5245535f53455455505f524551227d

application/vnd.3gpp.ngap (No Content ID)
436f6e74656e742d49443a2050647553657373696f6e5265736f757263655365747570526571756573745472616e736665720d0a436f6e74656e742d5472616e736665722d456e636f64696e673a2062696e6172790d0a0d0a0000040082000c1004a817c8004002540be400008b000a01f00a3832613150031f00860001000088000700050000050000
end note
jkolom commented 1 year ago

I just pushed a couple of improvements, but it is by no means finished.

The regex is improved but it has to take into account the case you mentioned and I plan to include the NAS content in the message. It now looks like this image

I also have to update the documentation.

I built in an option so that you can import traces like the ones you have. The command I used including the new options is as follows:

python trace_visualizer.py -wireshark 4.0.5 -limit 70 -show_timestamp True -custom_packet_filter "exported_pdu" -custom_ip_src "field[@name='exported_pdu.ipv4_src']" -custom_ip_dst "field[@name='exported_pdu.ipv4_dst']" -custom_ip_src_attribute "show" -custom_ip_dst_attribute "show" "C:\<path_to_file>\Trace tests\one_packet.pcapng"

rtommy commented 1 year ago

I think you are trying to do too much at the same time.

I would recommend skip the "exported_pdu" related things completely because there are too many more things to consider. And those new parameters just makes it too complicated I think.

Perhaps just try to fix the regex first to find the multiparts with all the Content-xxx lines properly and then we can take it from there under a new issue.

jkolom commented 1 year ago

You are not the only one having such formats ;) Plus, if I cannot import the pcap, I cannot test it. It was just a partial implementation solving the known problem at the time. The other header csme later. The regex just needs some reworking + some unit tests. The next 3GPP meeting is coming, so I might not have much free time for my hobby project in the next days. The current regex could be quickly (but not very robustly) fixed, I think that for some quick tests for you for the time would be OK Although the parameters look complex, they are actually XPath expressions. They are quite usual to addreds XML elements.

rtommy commented 1 year ago

There is another mistake I believe. Around line 616

                        logging.debug(
                            'Found {1} MIME-multiparts by scanning payload. Boundary: "{0} ({3} bytes)".\n  Parts found: {2}'.format(
                                boundary,
                                len(m_all),
                                ', '.join(multipart_descriptions),
                                len(boundary * 2)))
                        if len(m_all) > 0:
                            boundary_scan = True

Should not be if len(m_all) == 0: instead?

We only want manual scan later if we did not find the multi parts, which means the mime_multipart_payload_regex.finditer() has no result. But in our cases result is there, we have the boundary and multparts found and decoded, hence boundary_scanshould be False.

jkolom commented 1 year ago

Still missing some documentation and some more testing, but it seems to now work for the following three test files: Trace tests.zip

Following script to generate the traces: python trace_visualizer.py -wireshark 4.0.5 -limit 70 -show_timestamp True -custom_packet_filter "exported_pdu" -custom_ip_src "field[@name='exported_pdu.ipv4_src']" -custom_ip_dst "field[@name='exported_pdu.ipv4_dst']" -custom_ip_src_attribute "show" -custom_ip_dst_attribute "show" "<path>\one_packet.pcapng"

one_packet_4 0 5

python trace_visualizer.py -wireshark 4.0.5 -limit 70 -show_timestamp True -custom_packet_filter "exported_pdu" -custom_ip_src "field[@name='exported_pdu.ipv4_src']" -custom_ip_dst "field[@name='exported_pdu.ipv4_dst']" -custom_ip_src_attribute "show" -custom_ip_dst_attribute "show" "<path>\content-transfer-encoding.pcapng"

content-transfer-encoding_4 0 5

python trace_visualizer.py -wireshark 4.0.5 -http2ports "65413,65428,65438,65440,65457,65462,65495,65482,65501,65504,65512,65514,65521,65528,31382,8080,34385" -show_timestamp True "<path>\2020.06.15 Service Request Connected_205.pcap"

2020 06 15 Service Request Connected_205_4 0 5