Closed jameskerr closed 4 years ago
Verified in Brim commit 16a82e1
talking to zqd
commit 51553cf
.
In the attached video, I select three columns in the picker, export the results, then re-import them into a new Space. This all worked correctly relative to how the cut
processor works, though using Export the way I did reminded me of some of behaviors of cut
that we might want to reconsider or add additional options, as users less familiar with cut
might be thrown by the current behavior.
In the video, you can see how I originally had 6 Zeek events on my screen after I'd narrowed down the set of columns, but then when I exported them, only 2 events made it into the exported ZNG. This seems to be due to the cut
behavior that only produces an event if there's values for all the listed field names. Here's a repro of that using the same data in zq
outside the app:
$ zq -t all.zng
#0:record[_path:string,ts:time,ts_delta:duration,peer:bstring,gaps:uint64,acks:uint64,percent_lost:float64]
0:[capture_loss;1585245809.994157;0.487586;zeek;0;2;0;]
#1:record[_path:string,ts:time,peer:bstring,mem:uint64,pkts_proc:uint64,bytes_recv:uint64,pkts_dropped:uint64,pkts_link:uint64,pkt_lag:duration,events_proc:uint64,events_queued:uint64,active_tcp_conns:uint64,active_udp_conns:uint64,active_icmp_conns:uint64,tcp_conns:uint64,udp_conns:uint64,icmp_conns:uint64,timers:uint64,active_timers:uint64,files:uint64,active_files:uint64,dns_requests:uint64,active_dns_requests:uint64,reassem_tcp_size:uint64,reassem_file_size:uint64,reassem_frag_size:uint64,reassem_unknown_size:uint64]
1:[stats;1585245809.994157;zeek;63;9;989;-;-;-;41;40;1;0;0;0;0;0;11;0;1;0;0;0;0;0;0;0;]
#2:record[_path:string,ts:time,fuid:bstring,tx_hosts:set[ip],rx_hosts:set[ip],conn_uids:set[bstring],source:bstring,depth:uint64,analyzers:set[bstring],mime_type:bstring,filename:bstring,duration:duration,local_orig:bool,is_orig:bool,seen_bytes:uint64,total_bytes:uint64,missing_bytes:uint64,overflow_bytes:uint64,timedout:bool,parent_fuid:bstring,md5:bstring,sha1:bstring,sha256:bstring,extracted:bstring,extracted_cutoff:bool,extracted_size:uint64]
2:[files;1585245809.986839;FpIVIaUXxIUYAPdhl;[104.28.19.94;][192.168.5.51;][CfdJY93ULujH9IXQmi;]HTTP;0;[MD5;SHA1;]text/plain;-;0;-;F;15;15;0;0;F;-;25d2d8c3eff2ce996e29c63984d83a8f;b8a709d0f41a3b11e3be1a3195be2580198f561b;-;-;-;-;]
#3:record[_path:string,ts:time,uid:bstring,id:record[orig_h:ip,orig_p:port,resp_h:ip,resp_p:port],trans_depth:uint64,method:bstring,host:bstring,uri:bstring,referrer:bstring,version:bstring,user_agent:bstring,origin:bstring,request_body_len:uint64,response_body_len:uint64,status_code:uint64,status_msg:bstring,info_code:uint64,info_msg:bstring,tags:set[string],username:bstring,password:bstring,proxied:set[bstring],orig_fuids:array[bstring],orig_filenames:array[bstring],orig_mime_types:array[bstring],resp_fuids:array[bstring],resp_filenames:array[bstring],resp_mime_types:array[bstring]]
3:[http;1585245809.519018;CfdJY93ULujH9IXQmi;[192.168.5.51;53262;104.28.19.94;80;]1;GET;ifconfig.co;/;-;1.1;curl/7.64.1;-;0;15;200;OK;-;-;[]-;-;-;-;-;-;[FpIVIaUXxIUYAPdhl;]-;[text/plain;]]
1:[stats;1585245809.506571;zeek;63;1;78;-;-;-;409;12;1;0;0;1;0;0;40;36;0;0;0;0;0;0;0;0;]
#zenum=string
#4:record[_path:string,ts:time,uid:bstring,id:record[orig_h:ip,orig_p:port,resp_h:ip,resp_p:port],proto:zenum,service:bstring,duration:duration,orig_bytes:uint64,resp_bytes:uint64,conn_state:bstring,local_orig:bool,local_resp:bool,missed_bytes:uint64,history:bstring,orig_pkts:uint64,orig_ip_bytes:uint64,resp_pkts:uint64,resp_ip_bytes:uint64,tunnel_parents:set[bstring]]
4:[conn;1585245809.506571;CfdJY93ULujH9IXQmi;[192.168.5.51;53262;104.28.19.94;80;]tcp;http;0.487508;75;404;SF;-;-;0;ShADadFf;6;339;4;576;-;]
$ zq -f table "cut ts,_path" all.zng
TS _PATH
1585245809.994157 capture_loss
1585245809.994157 stats
1585245809.986839 files
1585245809.519018 http
1585245809.506571 stats
1585245809.506571 conn
$ zq -f table "cut ts,_path,uid" all.zng
TS _PATH UID
1585245809.519018 http CfdJY93ULujH9IXQmi
1585245809.506571 conn CfdJY93ULujH9IXQmi
I'll flag this topic for possible discussion at a future discussion with the UX team. In the meantime, what we have here seems good for now.
Thanks @jameskerr!
The loose end described in the previous comment is planned to be addressed via https://github.com/brimsec/zq/issues/852.
Use the cut proc to cut out only the fields that the user has selected in the columns picker.