Implement reading & writing of gzip compressed pcap files

[g-v-egidy]

pcap files can get huge quite fast. But with a verbose text protocol like SIP, they have a low information density. So using a lightweight compression algorithm like deflate/gzip will not just result in much smaller files, but also often in faster execution because packing is faster than the additional disk IO. Compression ratios around 90% are common for gzip and SIP pcap files.

gzip is a very common disk format for pcap files. Programs like wireshark directly support gzip compressed pcap files.

Using just the a pipe output and then piping it through gzip would not result in the same functionality as the SIGHUP handler then wouldn't work to reopen a changed file.

So implementing this directly in sngrep yields the best feature set.

---

libpcap doesn't directly support this, so this is implemented using the Linux call fopencookie which rereoutes the read,write,seek,close functions. *BSD seems to have something similar (funopen) which is not implemented in this patch because I'm not familiar enough with BSD.

gzip detection for read is done by first directly opening the given file like before. If this fails, we retry with gzip.

gzip detection for write is done by looking at the filename to write to. If it ends in ".gz" we activate gzip compression.

This currently just works for the commandline option --output because only there you get to set the filename suffix freely. To make this usable in the curses gui, the save dialog would have to be extended to allow setting a .pcap.gz filename extension.

gzip compression must be compiled in to be active. This is done with the new --with-zlib configure option.

[Kaian]

This is awesome! I'll try to test it as soon as I have some spare time :)

Also note any file can be piped through stdin so you can also see gziped files using:

zcat file.gz | sngrep -I -

Regards!

[g-v-egidy]

This is awesome! I'll try to test it as soon as I have some spare time :)

Thanks.

I was not sure if the extra configure option "--with-zlib" is the preferred way or if it would be better to automatically activate the compression feature when zlib is found, as zlib is a very common library. I have the "--with-zlib" configure option in the code right now, but I can easily change it.

Also note any file can be piped through stdin so you can also see gziped files using:

zcat file.gz | sngrep -I -

Yes, I know that for reading this is possible. But I thought when we implement writing, it would be nice to have native reading integrated too. As you see it is not much additional code that is required for read support once the basic framework for zlib, fopencookie etc. is there.

[Kaian]

I was not sure if the extra configure option "--with-zlib" is the preferred way or if it would be better to automatically activate the compression feature when zlib is found, as zlib is a very common library. I have the "--with-zlib" configure option in the code right now, but I can easily change it.

I think --with-zlib flag is the best way to implement it. We can enable some of the configure flags by default (or enable them if we detect they requisites are met), but I right now they're all disabled by default, so keeping it that way is probably the most correct way to implement it.

Regards!

[Kaian]

Hi!

I had some time to test this, everything works perfectly, really good job :)

I'm having a compilation warning when using zlib 1.2.12

capture.c:268:13: warning: initialization of '__ssize_t (*)(void *, char *, size_t)' {aka 'long int (*)(void *, char *, long unsigned int)'} from incompatible pointer type 'ssize_t (*)(void *, const char *, size_t)' {aka 'long int (*)(void *, const char *, long unsigned int)'} [-Wincompatible-pointer-types]
  268 |             gzip_cookie_read, NULL, NULL, gzip_cookie_close
      |       

In this version, the read function signature is:

typedef __ssize_t cookie_read_function_t (void *__cookie, char *__buf, size_t __nbytes);

while gzip_cookie_read uses:

static ssize_t gzip_cookie_read(void *cookie, const char *buf, size_t size)

Removing the const attribute for second argument also removes the warning, but I'm not sure if this is the function signature in previous zlib versions.

Regards!

[g-v-egidy]

In this version, the read function signature is:

typedef __ssize_t cookie_read_function_t (void *__cookie, char *__buf, size_t __nbytes);

while gzip_cookie_read uses:

static ssize_t gzip_cookie_read(void *cookie, const char *buf, size_t size)

This is a copy-n-paste error I made. When you think about it, the read function needs to write the read-out data somewhere, so it can't use a const pointer.

I updated the code. I also removed casting the size_t to unsigned, as keeping everything size_t is more consistent. I did a short test and everything still works.

Thanks for noticing the bug.

[Kaian]

Awesome!

I'll try to make a 1.6.0 release soon with all these contributions.

Thanks!

[g-v-egidy]

Thanks for reviewing & merging my patches.

I'll try to make a 1.6.0 release soon with all these contributions.

Very good. At least I don't have any more patches to merge soon.

What could be added is support for writing gzip compressed pcaps from the curses tui. The F2 dialog for file type selection there would have to be extended to add .pcap.gz as extension. Once the dialog sets the correct filename extension, my new code should automatically activate the compression.

I have never written code for ncurses yet, so I would have to familiarize myself with it before extending the dialog there myself. So this is not something I'll do soon. If you are familiar with curses and have some time you could add it to complete the pcap compression support.