philrz
commented
4 years ago Verified in Brim commit 16a82e1 talking to zqd commit 51553cf.
In the attached video, I select three columns in the picker, export the results, then re-import them into a new Space. This all worked correctly relative to how the cut processor works, though using Export the way I did reminded me of some of behaviors of cut that we might want to reconsider or add additional options, as users less familiar with cut might be thrown by the current behavior.
In the video, you can see how I originally had 6 Zeek events on my screen after I'd narrowed down the set of columns, but then when I exported them, only 2 events made it into the exported ZNG. This seems to be due to the cut behavior that only produces an event if there's values for all the listed field names. Here's a repro of that using the same data in zq outside the app:
$ zq -t all.zng
#0:record[_path:string,ts:time,ts_delta:duration,peer:bstring,gaps:uint64,acks:uint64,percent_lost:float64]
0:[capture_loss;1585245809.994157;0.487586;zeek;0;2;0;]
#1:record[_path:string,ts:time,peer:bstring,mem:uint64,pkts_proc:uint64,bytes_recv:uint64,pkts_dropped:uint64,pkts_link:uint64,pkt_lag:duration,events_proc:uint64,events_queued:uint64,active_tcp_conns:uint64,active_udp_conns:uint64,active_icmp_conns:uint64,tcp_conns:uint64,udp_conns:uint64,icmp_conns:uint64,timers:uint64,active_timers:uint64,files:uint64,active_files:uint64,dns_requests:uint64,active_dns_requests:uint64,reassem_tcp_size:uint64,reassem_file_size:uint64,reassem_frag_size:uint64,reassem_unknown_size:uint64]
1:[stats;1585245809.994157;zeek;63;9;989;-;-;-;41;40;1;0;0;0;0;0;11;0;1;0;0;0;0;0;0;0;]
#2:record[_path:string,ts:time,fuid:bstring,tx_hosts:set[ip],rx_hosts:set[ip],conn_uids:set[bstring],source:bstring,depth:uint64,analyzers:set[bstring],mime_type:bstring,filename:bstring,duration:duration,local_orig:bool,is_orig:bool,seen_bytes:uint64,total_bytes:uint64,missing_bytes:uint64,overflow_bytes:uint64,timedout:bool,parent_fuid:bstring,md5:bstring,sha1:bstring,sha256:bstring,extracted:bstring,extracted_cutoff:bool,extracted_size:uint64]
2:[files;1585245809.986839;FpIVIaUXxIUYAPdhl;[104.28.19.94;][192.168.5.51;][CfdJY93ULujH9IXQmi;]HTTP;0;[MD5;SHA1;]text/plain;-;0;-;F;15;15;0;0;F;-;25d2d8c3eff2ce996e29c63984d83a8f;b8a709d0f41a3b11e3be1a3195be2580198f561b;-;-;-;-;]
#3:record[_path:string,ts:time,uid:bstring,id:record[orig_h:ip,orig_p:port,resp_h:ip,resp_p:port],trans_depth:uint64,method:bstring,host:bstring,uri:bstring,referrer:bstring,version:bstring,user_agent:bstring,origin:bstring,request_body_len:uint64,response_body_len:uint64,status_code:uint64,status_msg:bstring,info_code:uint64,info_msg:bstring,tags:set[string],username:bstring,password:bstring,proxied:set[bstring],orig_fuids:array[bstring],orig_filenames:array[bstring],orig_mime_types:array[bstring],resp_fuids:array[bstring],resp_filenames:array[bstring],resp_mime_types:array[bstring]]
3:[http;1585245809.519018;CfdJY93ULujH9IXQmi;[192.168.5.51;53262;104.28.19.94;80;]1;GET;ifconfig.co;/;-;1.1;curl/7.64.1;-;0;15;200;OK;-;-;[]-;-;-;-;-;-;[FpIVIaUXxIUYAPdhl;]-;[text/plain;]]
1:[stats;1585245809.506571;zeek;63;1;78;-;-;-;409;12;1;0;0;1;0;0;40;36;0;0;0;0;0;0;0;0;]
#zenum=string
#4:record[_path:string,ts:time,uid:bstring,id:record[orig_h:ip,orig_p:port,resp_h:ip,resp_p:port],proto:zenum,service:bstring,duration:duration,orig_bytes:uint64,resp_bytes:uint64,conn_state:bstring,local_orig:bool,local_resp:bool,missed_bytes:uint64,history:bstring,orig_pkts:uint64,orig_ip_bytes:uint64,resp_pkts:uint64,resp_ip_bytes:uint64,tunnel_parents:set[bstring]]
4:[conn;1585245809.506571;CfdJY93ULujH9IXQmi;[192.168.5.51;53262;104.28.19.94;80;]tcp;http;0.487508;75;404;SF;-;-;0;ShADadFf;6;339;4;576;-;]
$ zq -f table "cut ts,_path" all.zng
TS _PATH
1585245809.994157 capture_loss
1585245809.994157 stats
1585245809.986839 files
1585245809.519018 http
1585245809.506571 stats
1585245809.506571 conn
$ zq -f table "cut ts,_path,uid" all.zng
TS _PATH UID
1585245809.519018 http CfdJY93ULujH9IXQmi
1585245809.506571 conn CfdJY93ULujH9IXQmiI'll flag this topic for possible discussion at a future discussion with the UX team. In the meantime, what we have here seems good for now.
Thanks @jameskerr!
Use the cut proc to cut out only the fields that the user has selected in the columns picker.