search

antonmedv / fx

Terminal JSON viewer & processor
https://fx.wtf
MIT License
19.08k stars 438 forks source link

Opening large JSON file takes forever #232

Closed jonashaag closed 1 year ago

jonashaag commented 1 year ago

I'm trying to view a ~ 1 GB JSON file (not JSON-per-line file).

With jless it takes around 2 seconds to open the viewer. With fx it takes longer than 60 seconds (actually I stopped fx after a while).

antonmedv commented 1 year ago

Apparently some bottleneck is there. Will be cool to find it and fix.

antonmedv commented 1 year ago

I did a profiling via:

CPU_PROFILE=cpu.prof ./fx random.json
go tool pprof -http 8080 ./fx cpu.prof

Looks like we can optimize collectSiblings() call to be a lazy loading. It will help with the first load.

Dialga commented 1 year ago

Ditto here, fx also lags in navigating json documents, navigating a large json document with jless is immediate and responsive.

antonmedv commented 1 year ago

Yes, looks like jless got ahead of fx here) I have a few ideas on how to improve fx in the future.

antonmedv commented 1 year ago

I took a look at profile and spotted a few slow places. I will rewrite them to be faster.

antonmedv commented 1 year ago

Wow. It took me the whole week (7*12h) to refactor fx from the ground up. Now fx can open JSON files of any size. I mean HUGE files and will be flying across it. I tested on a 3GB JSON file. I'm so tired. 😮‍💨

antonmedv commented 1 year ago

It took me forever. But I spent the whole week rewriting fx from scratch! Now fx is a rocket!!

GitHub sponsors are appreciated :)