brimdata / super

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.39k stars 64 forks source link

Issue with empty records #1352

Closed henridf closed 4 years ago

henridf commented 4 years ago
21:38 ~/work/brim/zq(master)
$ zq -t test.tzng 
#0:record[tls:record[fingerprint:string,issuerdn:string,ja3:record[],ja3s:record[],notafter:string,notbefore:string,serial:string,sni:string,subject:string,version:string]]
0:[[78:35:38:c6:0f:ad:a5:8e:81:d2:67:da:2b:10:06:36:1c:c7:da:8f;C=US, O=Google Inc, CN=Google Internet Authority G2;[][]2015-05-28T00:00:00;2015-02-27T20:32:20;20:D8:C6:DD:69:6D:FD:45;safebrowsing-cache.google.com;C=US, ST=California, L=Mountain View, O=Google Inc, CN=*.google.com;TLS 1.2;]]
21:38 ~/work/brim/zq(master)
$ zq test.tzng | zq -t -
#0:record[tls:record[fingerprint:string,issuerdn:string,ja3:record[],ja3s:record[],notafter:string,notbefore:string,serial:string,sni:string,subject:string,version:string]]
0:[[78:35:38:c6:0f:ad:a5:8e:81:d2:67:da:2b:10:06:36:1c:c7:da:8f;C=US, O=Google Inc, CN=Google Internet Authority G2;[zng syntax error
philrz commented 4 years ago

Verified in zq commit 85f3a81.

The syntax error is now gone, and turning the TZNG into zq's internal representation and writing it back out again as TZNG gives the same bytes.

$ zq test.tzng | zq -t - | tee output.tzng
#0:record[tls:record[fingerprint:string,issuerdn:string,ja3:record[],ja3s:record[],notafter:string,notbefore:string,serial:string,sni:string,subject:string,version:string]]
0:[[78:35:38:c6:0f:ad:a5:8e:81:d2:67:da:2b:10:06:36:1c:c7:da:8f;C=US, O=Google Inc, CN=Google Internet Authority G2;[][]2015-05-28T00:00:00;2015-02-27T20:32:20;20:D8:C6:DD:69:6D:FD:45;safebrowsing-cache.google.com;C=US, ST=California, L=Mountain View, O=Google Inc, CN=*.google.com;TLS 1.2;]]
$ diff test.tzng output.tzng 
$ echo $?
0

Thanks @henridf!