brimdata / super

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.38k stars 64 forks source link

Zed CLI tools won't output ZNG to terminal even if explicitly requested #5245

Open philrz opened 1 month ago

philrz commented 1 month ago

tl;dr

The following still produces ZSON output despite the -f zng.

$ echo '{ts:2024-08-14T19:12:51Z}' | zq -f zng -
{ts:2024-08-14T19:12:51Z}

Details

Repro is with Zed commit b9388c4.

In the absence of an explicit requested format via -f, the CLI tools are currently clever about sending ZSON by default if terminal output is detected and binary ZNG if the output is redirected. However, we offer the -f option if the user is confident they want to force it. In this regard, they can indeed get binary Parquet gobbledygook at the terminal if it's truly what they seek.

$ zq -version
Version: v1.17.0-36-gb9388c4c

$ echo '{ts:2024-08-14T19:12:51Z}' | zq -f parquet - 
PAR1L>j(??>j(?>j(?>j(?>j(??,5schema%tsl?<&?5ts??&>j(?>j(?>j(?>j(??,??
                                                                     parquet-go version 14.0.0?PAR1

Though it's hard to imagine why it would be desirable, it does seem like we should do the same with -f zng.

nwt commented 4 weeks ago

This is by design. Use the -B flag to send ZNG to standard output when it's connected to a terminal.