brimdata / super

An analytics database that puts JSON and relational tables on equal footing
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.4k stars 67 forks source link

Zed literal requires explicit casting or "this" notation #2845

Open philrz opened 3 years ago

philrz commented 3 years ago

Zed commit 389b120 is used in the examples shown below.

While writing the join docs published via #2836, for the multi-value join example, to create the nested records, at first I considered using this syntax to create the fields individually:

$ echo '{"name":"apple","color":"red"}' | zq -z 'put fruitkey.name:=name,fruitkey.color:=color' -

Instead I followed guidance to express it as a ZSON literal record:

$ echo '{"name":"apple","color":"red"}' | zq -z 'put fruitkey:={name:string(name),color:string(color)}' -

This worked, but in comparing the two, it struck me as odd that the first syntax doesn't require explicitly mentioning the string type, but the second does. If I leave out the string() casting wrappers, I get an error:

$ echo '{"name":"apple","color":"red"}' | zq -z 'put fruitkey:={name:name,color:color}' -
rhs of assigment expression: no such type name: "name"

I bounced this one off the Zed dev team, and @nwt commented:

I think what you’re trying to do here ought to be legal Zed, so I think you should create an issue.

philrz commented 3 years ago

@mccanne pointed out that using this can make it work without having to mention the type.

$ zq -version
Version: v0.29.0-422-gb00db736

$ echo '{"name":"apple","color":"red"}' | zq -z 'put fruitkey:={name:this.name,color:this.color}' -
{name:"apple",color:"red",fruitkey:{name:"apple",color:"red"}}
philrz commented 3 years ago

A community user just bumped into this while beta testing. In their own words:

switch event_id (
        3.0 => put process_id := is(process_id, string) ? replace(process_id, ",", "") : process_id
        | put record_number := is(record_number, string) ? replace(record_number, ",", "") : record_number
        | put thread_id := is(thread_id, string) ? replace(thread_id, ",", "") : thread_id
        | put id := {orig_h:src_ip_addr,orig_p:SourcePort}
        | put this := shape(sysmon_type3) | put this := crop(sysmon_type3);

When I added the | put id := {orig_h:src_ip_addr,orig_p:SourcePort} to the script I got an error:
rhs of assigment expression: no such type name: "src_ip_addr"

even though src_ip_addr is one of the field names

I made them aware of this issue and they worked around the problem, but since external parties are starting to hit it, it might be worth making it a priority to handle soon after our next GA release goes out.

mccanne commented 3 years ago

The problem is record types and record values look similar when any of the field types is a type name (you can't tell the difference between a type name and a field name). Right now the parser favors record types over values. I think the way to fix this is to have the parser favor record values and require the use of type(), e.g.,

type({orig_h:src_ip_addr,orig_p:SourcePort})

if you mean "type".