Pcapng Support

[tiferrei]

Hi there,

I have a Wireshark pcapng file from a very simple QUIC 29 capture that I'd like to convert to qlog for visualisation with qviz. I can see in the README that pcapng are said to be supported, however when running the following command:

node out/main.js -t /Applications/Wireshark.app/Contents/MacOS/tshark -i ~/Documents/UCL/QUIC/quic.pcapng --output ./quic.qlog

I seem to get the following error:

Top level error TypeError: capt.qlog.traces is not iterable

I have also tried exporting the file as pcap, and passing in the original SSL key log file, however I got the same error.

Any further help would be greatly appreciated, thank you! Tiago

[rmarx]

Hello Tiago,

Nice to see UCL joining the qlog herd :)

pcap2qlog does support pcapng (added full support just last week) and so I'm having trouble reproducing your error, as my test files do work... it might have something to do with your environment/NodeJS/Wireshark version (I run on Ubuntu with a very new Wireshark release). The qlog.traces should always be at least an empty array according to the code, so I'm not sure what's happening (I tested with a pcap that contains no QUIC traffic and that also works fine here).

So some things you can try:

1. send me the file that's giving you trouble so I can reproduce
2. run the command with the PCAPDEBUG environment variable set to true (e.g., on linux, PCAPDEBUG=true node out/main.js ... and see if that shows some errors (and then paste them here)
3. try one of my working test files (e.g., https://github.com/triplewy/quic-benchmarks/blob/1603a04ecfea1dd1d59620623682e9f0d0ba6d53/firefox-fb-10mb-loss_1-fail_4-no_tcp-dsb.pcapng?raw=true or https://github.com/triplewy/quic-benchmarks/blob/1603a04ecfea1dd1d59620623682e9f0d0ba6d53/firefox-fb-10mb-loss_1-fail_4-no_tcp-dsb.pcapng?raw=true) and see if you get the same error
4. try to host your pcapng somewhere (e.g., github works or even google drive) and load it directly from URL in qvis (qvis internally runs pcap2qlog so you shouldn't have to run it yourself) (this should also work for the example URLs in nr 3, e.g., https://qvis.edm.uhasselt.be/#/sequence?file=https://github.com/triplewy/quic-benchmarks/blob/1603a04ecfea1dd1d59620623682e9f0d0ba6d53/firefox-fb-10mb-loss_1-fail_4-no_tcp-dsb.pcapng?raw=true).
5. load the pcapng in your Wireshark.app and see if that works correctly (it could be your Wireshark doesn't support -29 yet?)

Please let me know whatever you find, happy to help more if needed :)

[tiferrei]

Hi Robin,

Thanks for the fast reply and warm welcome. 😃

After some further troubleshooting I believe that:

• The issue is likely not with the file. I say this because after trying the command with the working test file I got the same error, and I later uploaded my file to the hosted qvis and the server converted it and visualised the file without any issues. My Wireshark (Version 3.3.0rc0-1429-gf97e20a01114 (v3.3.0rc0-1429-gf97e20a01114) ) was also able to open both files, in fact that's what I used to create my capture file, before I knew about qlog.
• The issue is also likely not directly with the code, since it works well for you, and on the server too.

I would be inclined to believe it's an issue specific to my environment. Specifically:

• It could just be that I'm using macOS and not Linux.
• I could be on an incompatible Node version (v14.4.0)

That being said I found the Docker files from the README and got everything working on a docker container. So I can serve my pcapng files in a docker container in the same network and pass an URL to it to the qvis server. A bit overengineered but I guess it works ¯_(ツ)_/¯

However, let me know if you're keen on getting it working on different OSs, and I'd be happy to help continue debugging!

Thanks, Tiago

[rmarx]

Hello Tiago,

Good to hear you got things working (at least for some definition of 'working').

My guess would be that it's due to the Node version then, as my setup still uses v10.x (though not sure why the code would suddenly bork in v14, I can't replicate the error in the Chrome JS console either...).

Ideally of course I'd like this to work on all platforms and latest versions of things. Practically though, I won't spend much time on that myself, as my main target is deploying in a docker container as part of qvis. So: PRs always welcome, but I'm not sure it's the best use of your time.

I will update the pcap2qlog README though, since now you mention it, it's quite outdated apparently.

Let me know if you have any more issues! Robin

[marty90]

I also had this issue. It seems that the tools has a wrong expectation of what to find in the Tshark json files. It expects a dictionary with a traces key, but It gets a list.

Inserting these lines around line 185 solved for me:

            capt.qlog = {};
            capt.qlog.traces = [];
            capt.qlog.traces.push (JSON.parse(fileContents.toString()));

Edit: NOT WORKING, see below

[rmarx]

Hello @marty90, thanks for letting me know.

I'm however not entirely sure what's going wrong here... which file and which line 185 are you referring to specifically? Do you maybe have an example of a pcap/pcapng file that produces errors that I can look at? Does it work when loading the pcap into qvis (https://qvis.quictools.info/), which is the main deployment of pcap2qlog?

It could be that Wireshark has changed their json output since the last updates here and that I need to fix things, but for that I'd need an example of an updated pcap.

[marty90]

Yep sorry, I am editing directly the file main.json. The original lines are in: https://github.com/quiclog/pcap2qlog/blob/62dbd8be1d3a3a040d1f3afc400bebfaa79658d7/src/main.ts#L215

[marty90]

I tried uploading the PCAPs in https://qvis.quictools.info/ But it tells: Upload currently supports .qlog, .json, and .netlog files. No data is transfered to the server.

[rmarx]

yeah, that's why I was confused, since there you're loading qlog files directly, not parsing them from a pcap... (see the if( fileIsQLOG(capt.capture) ){ on line 211). That would mean you're loading badly formatted qlog files (or potentially newline-delimited qlog files, though those shouldn't parse using JSON.parse...) instead of bad pcap files.

What exactly are you loading into pcap2qlog here? The fileIsQLOG method is very stupid, just looks for .qlog somewhere in the filename. Maybe you're naming files like capture.qlog.pcap or something? But in that case, your code changes shouldn't work... it's all a bit confusing to me :)

For using the pcaps with qvis, you need to upload the files somewhere else on your own webserver and then load by URL (since I don't want to safeguard the server against abuse from uploading huge files)

[marty90]

I am working on PCAP file, providing the keys. My command line is:

node out/main.js --input ../capture.pcap --secrets ../capture.keys --tshark=$(which tshark) --output=../capture.qlog

[marty90]

Oh! I solved! --output=../capture.qlogwas generating the error.

Changing it to --output=../capture.out seems to solve!

Maybe, as you said, the tool was misunderstanding the input format.

[rmarx]

Ooh, I see... I think --output should be used as a directory (so --output=../). If you want to specify a filename, you should use --outputpath.

No idea why that borks on that line of code though, that's very strange... I'm going to reopen this issue to debug it down the line. Good to know you were able to find a workaround that works for you!

[marty90]

Thank you for the prompt support Robin