search

brimdata / super

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.39k stars 64 forks source link

allow strings as zng field names #1291

Closed mccanne closed 4 years ago

mccanne commented 4 years ago

Relax the syntax for zng columns names. We currently require conformance with JSON identifiers but even javascript and JSON allow object keys to be arbitrary keys. Thus to properly ingest any JSON, we should support arbitrary strings as keys. This means we will need a zql special syntax to support record accesses that have special chars and we might as well follow javascript convention e.g., rec.@foo should be rec["@foo"].

mccanne commented 4 years ago

That said, it seems like you should be able to say @foo=val in zql so there is a little design work to be done here...

philrz commented 4 years ago

Verified in zq commit d9caaa9.

Now special characters like . and @ can appear in field names, and the ZQL language provides the ["..."] notation for escaping these when they need to be referenced, such as below where I successfully match on the event when filtering on the field by name.

$ cat input.tzng 
#0:record[r:record[[.bar@]:int32,x:string]]
0:[[123;hello;]]

$ zq -t 'r[".bar@"]=123' input.tzng 
#0:record[r:record[[.bar@]:int32,x:string]]
0:[[123;hello;]]

Thanks @mccanne!