Qvis - downloadable

[Samy432]

I use https://qvis.edm.uhasselt.be/#/files to visualize qlogs. At times the qlogs are more than 100MB and it seems pretty slow to navigate through. Is there a way to have qvis tool installed on the machine. I also notice that QUICvis is deprecated. Any suggestions please?

[rmarx]

Hey Samy,

Thank you for using qvis!

Currently we don't have an official standalone build, since this would lack support for loading pcap files (as we rely on wireshark to do the conversion, which requires more work). However, @lpardue has recently been tinkering with an electron-based setup that seems to work for the basic usecase of visualizing qlog files directly, see https://github.com/LPardue/qvis/releases/tag/qvis-app-rc1. Maybe you can beta-test that?

However, I'm not sure that will be much faster than loading the files in the browser, since Electron still uses the browser's engines to render the visualizations. While qvis' JavaScript approach is slower than other languages might be, it should still be quite usable even for large files (I've tested files of several 100 MBs myself). Would you be able to somehow share some more details (potentially even an example file?) of what exactly is slow so I can try to reproduce/optimize? Also do let us know if the standalone build is faster for you!

Finally, QUICvis is indeed deprecated, but this was the "old" project on which qvis is based. qvis itself is very much alive and well and planned to continue for quite some time to come. You can just keep using the URL you mention or the code in this repository for that. I hope that's clear.

If you have any more questions, please let us know!

With best regards, Robin

[undvikar]

Hello Robin,

I am experiencing the same issues, however with the standard 31 MB demo file.

Loading it is certainly possible but switching between the different tools, using the zoom function to view packet details and other actions slow down the web browser so much that these actions are delayed by 10 seconds or more. Tools especially affected are the sequence, multiplexing and packetization tools.

Both the multiplexing and packetization tools share a performance issue concerning zoom. The packetization tool also takes about the same time to initially load as the sequence tool. The congestion tool is the quickest to respond of all tools, the zoom works without delay.

I tried all this in Firefox.

In Chromium, delays still happen but are certainly shorter. However, after zooming in and out a few times (and zooming while the zoom action has not yet been performed) in the packetization tool, I manage to make the site crash consistently with the error code SIGTRAP. In Firefox, it simply takes very long for the zooms to happen.

As I am a CS student and currently writing a paper on qlog, qvis and their capabilities, I am interested if a native application, multi-threading or other solutions are planned for better performance with larger qlogs?

I also tried to test the electron application but it could not find a package.json file in the extracted directory and failed. The warning first appeared after running vue add electron-builder.

Thank you and best regards,

Dominik

[LPardue]

if you're building your own electron application, make sure you run vue add electron-builder in the right directory. I think this is at /visualizations

[rmarx]

@toasthorse Thanks a lot for the detailed explanation. Also happy to hear you're writing something on the tools, would love to read that when it's done.

I have only ever used Chromium on Windows when developing and testing the tools, so it's not really a big surprise that things don't work as well in Firefox. I also don't have concrete plans to go in and optimize for other browsers at this point.

I am a bit surprised about the crash in Chromium. Can you confirm the steps to reproduce are: a) load the 31 MB demo file b) open it in the packetization tool c) zoom in/out 5+ times? Could you also say which OS+version you're using (I'd assume linux or OSX?)? Here, the 31MB file is indeed slow in the packetization view, but doesn't crash.

In general, it is expected that the performance of the tools will suffer with (very) large files. This is partly because most tools utilize SVG rendering, drawing each packet/frame as a separate SVG entity, which is slow. The only way to work around that is to use canvas-based rendering, which is what the congestion graph currently does. This however makes it much more difficult to implement picking/hover effects (the congestion graph can be somewhat buggy on that point), which is why we've gone with the SVG approach for the other tools at the moment to get a Proof-of-Concept that works in most cases for now. For the future, the goal is to port the other tools to the canvas-based method as well.

The one exception is the sequence diagram, which uses SVG rendering but doesn't draw the entire trace, but only the currently visible part + a little bit before and after that. As such, I was surprised to hear the sequence diagram is also slow in chromium on large traces for you. It can be slow loading the initial trace, but should be very smooth interacting after that. Could you give a bit more information on what you're seeing there?

However, even with the canvas method, it's still browser-based, which will probably never scale to very large logs (e.g., 500MB+). It's also not possible (afaik) to (easily) make that multi-threaded in things like electron. For this, we have plans to write qlog importers for other native tools (like Windows Performance Analyzer). This is however still a way's off. Porting qvis to native code is also an option, but that would likely have to come from the community (i.e.,: not me ;) )

Finally: have you tried the pre-built electron .exe from the link I provided above instead of building it yourself (assuming you have access to a Windows machine)? What performance does that give you?

Please let us know any other things you find. Feel free to open new issues here. Don't assume this is all highly polished or tested code ;) The goal is to make it better over time, so hearing concrete experiences from users only helps with that.

Best of luck with the paper! Robin

[undvikar]

@rmarx Yes, this is how I produced the crash in Chromium. I am running Arch Linux 5.10.4. However I think that this might be exclusive to Linux, I could not reproduce it on Windows 8 (I currently don't have access to a Windows 10 machine).

Regarding the sequence diagram, I should have been a bit more clear; it only takes a while to load, but it is indeed smooth after loading it :)

I tried the pre-built electron .exe on Windows 8 and I could only load the small demo files, but not the 31 MB one. Only the blue banner appears, but not the green one to confirm that it has been loaded; when clicking on the different tools, it only says "Loading files...". In Chromium, Google Chrome and Firefox on Windows 8 I can load it without a problem.

Thank you for the explanations regarding the implementation of the tools and future plans! I will make sure to let you know if I find anything else.

@LPardue Thank you, running it in the /visualizations directory worked! However it did not compile, there seems to be an error in the implementation:

ERROR in /home/dominik/downloads/qvis/qvis-qvis-app-rc1/visualizations/src/components/sequencediagram/renderer/SequenceDiagramD3Renderer.ts
366:12 Type 'Timeout' is not assignable to type 'number'.
    364 |
    365 |            clearTimeout( scrollTimer );
  > 366 |            scrollTimer = setTimeout( execute, 250 ); // 250 ms
        |            ^
    367 |
    368 |            ++scrollEventCount;
    369 |            if ( scrollEventCount % 5 === 0 ){ // 5 = magic number, from experimental experience in chrome, very scientific

 ERROR  Vue CLI build failed. Please resolve any issues with your build and try again.

[LPardue]

Ah yes, I think this is fixable by a very small tweak. I don't have the exact code available to me right now but see this comment that I think inspired me: https://github.com/microsoft/TypeScript/issues/842#issuecomment-252445883

On Wed, Jan 6, 2021 at 5:08 PM toasthorse notifications@github.com wrote:

@rmarx https://github.com/rmarx Yes, this is how I produced the crash in Chromium. I am running Arch Linux 5.10.4. However I think that this might be exclusive to Linux, I could not reproduce it on Windows 8 (I currently don't have access to a Windows 10 machine).

Regarding the sequence diagram, I should have been a bit more clear; it only takes a while to load, but it is indeed smooth after loading it :)

I tried the pre-built electron .exe on Windows 8 and I could only load the small demo files, but not the 31 MB one. Only the blue banner appears, but not the green one to confirm that it has been loaded; when clicking on the different tools, it only says "Loading files...". In Chromium, Google Chrome and Firefox on Windows 8 I can load it without a problem.

Thank you for the explanations regarding the implementation of the tools and future plans! I will make sure to let you know if I find anything else.

@LPardue https://github.com/LPardue Thank you, running it in the /visualizations directory worked! However it did not compile, there seems to be an error in the implementation:

ERROR in /home/dominik/downloads/qvis/qvis-qvis-app-rc1/visualizations/src/components/sequencediagram/renderer/SequenceDiagramD3Renderer.ts 366:12 Type 'Timeout' is not assignable to type 'number'. 364 | 365 | clearTimeout( scrollTimer );

366 | scrollTimer = setTimeout( execute, 250 ); // 250 ms | ^ 367 | 368 | ++scrollEventCount; 369 | if ( scrollEventCount % 5 === 0 ){ // 5 = magic number, from experimental experience in chrome, very scientific

ERROR Vue CLI build failed. Please resolve any issues with your build and try again.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/quiclog/qvis/issues/38#issuecomment-755432423, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSELNOEUVLHFPV7673XTODSYSKIFANCNFSM4VMGI7CA .

[rmarx]

I've attempted to fix this by making it an "any" instead of "number" in a recent commit (see above). Seems like a later version of TypeScript has different bindings than what I'm using. Please try again?

With regards to the 31MB file in the electron version: @LPardue didn't bundle that in with the .exe, so you should download that separately via https://qvis.edm.uhasselt.be/standalone_data/draft-00/mvfst_large.qlog and then upload it as a local file via Option 2. Sorry for the confusion :)

Hoping you can test on both Arch and Windows 8 and let us know the results. Thanks!

[undvikar]

@rmarx Thank you, I was able to run the built app on Linux and the .exe on Windows by supplying the qlog via option 2, however they both performed about as well as the web tool.

[rmarx]

Hey @toasthorse. Sad to hear using the built app did not significantly improve performance, but also not entirely unexpected since electron is basically chromium bundled with app code...

Please do let us know if you have any more problems/questions and when you finish your text; I'm interested in hearing your findings and opinions :)

I have now opened an issue on the msquic repo that I'll probably use to track some of the efforts I mentioned above for supporting other, more high-performance tooling like Windows Performance Analyzer: https://github.com/microsoft/msquic/issues/1158

[undvikar]

Hello @rmarx, my finished paper has been published here, it mostly describes qlog and qvis but perhaps you will enjoy reading it nonetheless :)

[rmarx]

Hello @toasthorse. Thank you for sharing the paper! I did really enjoy reading it and agree (mostly ;)) with the content and the conclusions :) It's always nice to have your work be of interest to others!