Closed philrz closed 4 years ago
Slowness is a feature not a bug. It will motivate people to upgrade their zng files from alpha to beta format. Haha.
Seriously, that's pretty slow. Maybe there is a buffer that needs to be inserted. I will take a look.
Ok, I tried some buffering tricks that didn't make any difference and did this measurement:
% time zq -f ndjson new.zng | zq -i ndjson - > /dev/null
real 0m49.561s
...
% time zq -i azng -o /dev/null old.zng
warning: old.zng: converting from alpha zng to beta zng (slow)
warning: update zng by running 'zq -o new.zng old.zng'
real 0m51.164s
...
This says azng is taking about the same time as ndjson to take a trip to string-land and back.
Thanks for confirming @mccanne. I'll close this one and add a comment to #1376 regarding the expected performance delta.
I noticed that with the merge of #1375, for the combination of "newest
zq
reading older alpha ZNG format", the performance is quite poor poor. Since at this point I know the support for the alpha ZNG format is probably only important for as long as it takes for users to migrate their data, it's probably not worth spending a lot of time optimizing performance. However, the drop is significant enough that it has the potential to make migrations take a very long time, so I'm flagging it should we want to check if we made an easy mistake.For the permutations below, I'm using the binary ZNG test data from the zq-sample-data before and after it was updated to the new ZNG format via https://github.com/brimsec/zq-sample-data/pull/16 (which was merged right as #1375 was merged on the zq side).
Here's things performing well with "alpha ZNG & old
zq
":Here's things performing poorly with "alpha ZNG & new
zq
":Here's things performing well again with "new ZNG & new
zq
"):