Why is jsonnet so slow ?

mrinal1209 commented 5 years ago

Hello, I have a question regarding the comping speed to jsonnet to json is super slow and why is that so? I am actually working on converting my Grafana Dashboard json to jsonnet conversion . As the lines of jsonnet increase up to 2k, it takes 40 minutes to compile. For lines up to 700 it compiles in less than a second but with more lines added compilation time increases exponentially. In my current situation, I am dividing the lines of code in chunks and later on merging it, is there any better solution to compile big jsonnet(5k lines of file) files?

sbarzowski commented 5 years ago

@mrinal1209 Could you try https://github.com/google/go-jsonnet, or even better go-jsonnet with this PR: https://github.com/google/go-jsonnet/pull/280?

If the code is not confidential or you could anonymize it, we would appreciate getting an additional real world benchmark.

Back to your question. There are two sides of this issue. First, there are a few performance footguns, i.e. ways of writing your code that may seem natural when coming from other languages, but result in recalculating the same stuff over and over again. Usually you can work around that, but anyway we want to fix that soon. The other side is that even when used "correctly" it's fairly slow compared to let's say Python, because of the way it is interpreted (directly from AST) - but this definitely shouldn't lead to the extreme results that you are experiencing.

mrinal1209 commented 5 years ago

@sbarzowski so here's my comparison for both the compiler:-
C ++ variant for jsonnet

C++-Jsonnet

and object caching variant Go-Jsonnet

go-jsonnet

I guess this benchmark shows huge performance benefits.

mrinal1209 commented 5 years ago

@sbarzowski actually this is my GSoC19 project to convert grafana dashboard json files to jsonnet files. I don't find sharing my jsonnet branch for benchmarking purposes. As well I think that even 2mins for 1800 lines will take 30 mins for 6000 lines as the number of lines increases the performance decreases.

sparkprime commented 5 years ago

Yes please share it. The ability to generate different amounts of code and measure the performance of each is very useful.

On Mon, 17 Jun 2019 at 10:20, Mrinal Dutta notifications@github.com wrote:

@sbarzowski https://github.com/sbarzowski actually this is my GSoC19 project to convert grafana dashboard json files to jsonnet files. I don't find sharing my jsonnet branch for benchmarking purposes. As well I think that even 2mins for 1800 lines will take 30 mins for 6000 lines as the number of lines increases the performance decreases.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/google/jsonnet/issues/672?email_source=notifications&email_token=AABJBXX6IABPRXOSQUJLAKTP25JM3A5CNFSM4HYTVXDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX2SFJA#issuecomment-502604452, or mute the thread https://github.com/notifications/unsubscribe-auth/AABJBXWSCWPQ3FJLEJO64IDP25JM3ANCNFSM4HYTVXDA .

mrinal1209 commented 5 years ago

@sbarzowski @sparkprime as aspected as I increased the number of lines by 500 it's now taking an hour to compile here's the link to my branch https://github.com/mrinal1209/grafonnet-lib/tree/network-overview-jsonnet you can find jsonnet file in pmm-templates folder and the root directory is the path of the library to compile with. You can just compare and tell me why the last 500 lines of code are taking to indefinite time to compile.

sparkprime commented 5 years ago

This seems to result in a substantial reduction in run-time:

diff --git a/grafonnet/dashboard.libsonnet b/grafonnet/dashboard.libsonnet index a07010d..7654199 100644 --- a/grafonnet/dashboard.libsonnet +++ b/grafonnet/dashboard.libsonnet @@ -116,7 +116,7 @@ local timepickerlib = import 'timepicker.libsonnet'; _nextPanel: nextPanel + n, panels+::: _panels, },

addPanel(panel, gridPos , style):: self + self.addPanels([panel { gridPos: gridPos , [if style != null then 'style']: style }]),
addPanel(panel, gridPos , style):: self.addPanels([panel { gridPos: gridPos , [if style != null then 'style']: style }]), addRows(rows):: std.foldl(function(d, row) d.addRow(row), rows, self), addLink(link):: self { links+: [link],

sparkprime commented 5 years ago

Remove the self + from that line

sparkprime commented 5 years ago

I'm pretty sure this fixes your specific issue so I'm closing it. Feel free to re-open if you have more performance problems.

google / jsonnet

Why is jsonnet so slow ? #672