search

trentm / json

A "json" command for massaging JSON on your Unix command line.
https://trentm.com/json
Other
1.56k stars 122 forks source link

Slow (?) #79

Open systems-rebooter opened 10 years ago

systems-rebooter commented 10 years ago

Hello,

Thanks for the awesome tool! I am about to parse 2,000,000 rows with it in the following way:

$ time echo '{"name":"trent","age":38}' | json name age
trent
38

real    0m0.054s

The dataset I'm using is truncated to be as simple as above ^ The only difference is, that it contains 4 values.

I wrote a script using your tool and execution time bottleneck seems to be json. The oneliner, which is using it executed - 56ms, which is roughly equals same time as above ^ And once json was removed -execution time dropped to 5ms

Could you advice, please, because right now I'm about to assume, that it takes me at least 27 hours in best case scenario to parse simple 2,000,000 dataset :[

>>> 2000000*0.05/60/60
27.77777777777778

Thanks a lot and keep up a good work!

trentm commented 10 years ago

@systems-rebooter You can't extrapolate running time from 1 line to 2 million lines like that. There is basic startup overhead. I'd suggest trying 10k lines, 100k lines to judge full runtime. Also make sure you are on the latest json version.

systems-rebooter commented 10 years ago

Yeah, I will. Thanks @trentm