Add Qwik to JS Framework Benchmark

[ryansolid]

Repo: https://github.com/krausest/js-framework-benchmark

While this benchmark does not showcase Qwik's main benefit, there is some expectation of being present in the benchmark. It keeps coming up over at the JS Framework Benchmark repo and I've thrown an example together that worked in older versions, but I think it is best if it comes from the team. Or if not I can take care of it but I need a little bit of assistance.

So opening an issue here for more visibility because I feel it belongs here, and that way I can send them back to this issue.

[nnelgxorz]

Hey @ryansolid! Do you have a link to your Qwik code?

[ryansolid]

https://github.com/ryansolid/js-framework-benchmark/tree/qwik/frameworks/keyed/qwik

I'm pretty sure it's the way the client bootstrap works or some top-level config type thing. What I linked was the previous working version. Updating to beta 0.9.0 no longer was working (with no errors) so I just left it as is because to my knowledge @manucorporat already has a working fork of my example he was using for: https://github.com/BuilderIO/qwik/pull/1195

[nnelgxorz]

Thanks @ryansolid! If Manu has a working version of this somewhere it'd probably be best use that. But if not, myself or someone else can contribute a PR using your implementation as a starting point.

[cayolblake]

@ryansolid Can you please elaborate on why this benchmark is not a showcase of Qwik?

[ryansolid]

Qwik's superpower is eliminating the bootup cost of a web page that has previously server-rendered. This benchmark does not test server rendering. Most of the tests are around DOM rendering and updates.

[milahu]

push

While this benchmark does not showcase Qwik's main benefit, there is some expectation of being present in the benchmark.

quik should shine in the "cold" benchmarks

There are some benchmarks that measure hot and some that measure cold performance. https://github.com/krausest/js-framework-benchmark/issues/22#issuecomment-232775908

[wmertens]

@ryansolid looks like the render function iterates over the entire data structure every time?

To make use of the reactivity each row should be a component$ and it should get the data from the store itself (you can pass the store and the index, avoiding useStore in the row component).

Something like

export const Row = component$(({id,label,state})=>{
  return <tr key={id} class={id === state.selected ? "danger": ""}>
            <td class='col-md-1'>{id}</td>
            <td class='col-md-4'><a onClick$={
              () => state.selected = id
            }>{label}</a></td>
            <td class='col-md-1'><a onClick$={
              () => {
                const d = state.data;
                d.splice(
                  d.findIndex((d) => d.id === id),
                  1
                )
              }
            }><span class='glyphicon glyphicon-remove' aria-hidden="true" /></a></td>
            <td class='col-md-6'/>
          </tr>
})

// ...

        { state.data.map(({id, label}) => {
          return <Row key={id} id={id} label={label} state={state} />
        })

[wmertens]

Err, the above is wrong, the loop should not read the data entries, it should iterate over the index and give Row only the index and the state. Then the main render function doesn't subscribe to changes on each row.

[ryansolid]

Right.. separating the Row makes sense. VDOM consideration, otherwise it would cause the whole map to run again on a nested change. I can't remember if I tested it both ways. Separate components can impact creation benchmarks (which there are more of), especially for reactive VDOM hybrids.

I'm pretty sure Manu has a version working already he has shown it in a different issue: https://github.com/BuilderIO/qwik/pull/1195. My challenge was bootstrapping the client-only render. The script Misko gave me way back stopped working.

[wmertens]

UPDATE: Wrong

I changed the creation to

        {Array.from({length: state.data.length}, (_, id) => <Row key={id} id={id} state={state} />)}

and the component to

export const Row = component$((props: {id: number, state: BenchState})=>{
  // It would be better to put the selected prop inside the selected item
  return <tr key={props.id} class={props.id === props.state.selected ? "danger": ""}>
            <td class='col-md-1'>{props.id}</td>
            <td class='col-md-4'><a onClick$={
              () => props.state.selected = props.id
            }>{props.state.data[props.id].label}</a></td>
            <td class='col-md-1'><a onClick$={
              () => {
                const d = props.state.data;
                d.splice(
                  d.findIndex((d) => d.id === props.id),
                  1
                )
              }
            }><span class='glyphicon glyphicon-remove' aria-hidden="true" /></a></td>
            <td class='col-md-6'/>
          </tr>
})

~Is that correct @manucorporat? It doesn't run the main render when updating rows, but when swapping 2 rows it takes a long time for 10k items.~

[wmertens]

Oops I missed id vs i. This seems to be correct and quite performant:

component:

export const Row = component$(({data, state}: {data: BenchState['data'][0], state: BenchState})=>{
  const {id, label} = data
  return <tr class={id === state.selected ? "danger": ""}>
            <td class='col-md-1'>{data.id}</td>
            <td class='col-md-4'><a onClick$={
              () => state.selected = id
            }>{label}</a></td>
            <td class='col-md-1'><a onClick$={
              () => {
                const d = state.data;
                d.splice(
                  d.findIndex((d) => d.id === id),
                  1
                )
              }
            }><span class='glyphicon glyphicon-remove' aria-hidden="true" /></a></td>
            <td class='col-md-6'/>
          </tr>
})

generating table:

      <table class='table table-hover table-striped test-data'><tbody>
        {state.data.map((data) => <Row key={data.id} data={data} state={state} />)}
      </tbody></table>

[wmertens]

And with this change, selecting is also fast:

export const Row = component$(({item, state, isSelected}: {item: BenchState['data'][0], state: BenchState, isSelected: boolean})=>{
  const {id, label} = item
  return <tr class={isSelected ? "danger": ""}>
            <td class='col-md-1'>{item.id}</td>
            <td class='col-md-4'><a onClick$={
              () => {
                state.selected = id
              }
            }>{label}</a></td>
            <td class='col-md-1'><a onClick$={
              () => {
                const d = state.data;
                d.splice(
                  d.findIndex((d) => d.id === id),
                  1
                )
              }
            }><span class='glyphicon glyphicon-remove' aria-hidden="true" />x</a></td>
            <td class='col-md-6'/>
          </tr>
})

      <table class='table table-hover table-striped test-data'><tbody>
        {state.data.map((item) => <Row key={item.id} item={item} state={state} isSelected={item.id===state.selected} />)}
      </tbody></table>

[Dindaleon]

Nice stuff, I am about to start a project with Qwik and would like to have an idea on how it compares to solidjs and svelte.

[aydafield22]

hi all, how about this?

[literalpie]

My challenge was bootstrapping the client-only render. The script Misko gave me way back stopped working.

I believe this was the QwikLoader, which can be imported as a string called QWIK_LOADER from @builder.io/qwik/loader. here's a branch where I have things building and running correctly with the latest versions of Qwik and other dependencies.

[krausest]

I'd love to see a Qwik PR for my benchmark Let me know if can help!

[md-owes]

Any update on this? I am eagerly waiting for Qwik to be part of the benchmark to help me make a decision on my next product. Speed is a decision-making factor here.

[wmertens]

@md-owes loading speed or re-rendering speed?

You won't get faster SSR loading speed than Qwik.

[mhevery]

This is not high on our priority list, but the community could take this on...

A word of caution. The benchmark focuses on already running applications, NOT on startup perf. The principal value of Qwik is startup, so adding it to the benchmark would unlikely show anything unique as it is not what the benchmark is measuring. I expect Qwik to be faster than React but slower than SolidJS.

But if we had a benchmark showing startup perf, I expect Qwik to crush it compared to others, as it is hard to beat doing no work.

[md-owes]

@md-owes loading speed or re-rendering speed?

You won't get faster SSR loading speed than Qwik.

@wmertens, well i would expect both (loading and re-rendering). Anyways i have started my product development on Qwik.

[wmertens]

I used SSG to reliably create index.html, added my changes above and made a PR https://github.com/krausest/js-framework-benchmark/pull/1447

[wmertens]

Closing this because it was added. The benchmark showed that Qwik is the startup king and performs somewhat average for heavy client side DOM manipulation. Good to have a baseline to compare against.

[wenfangdu]

@wmertens Why is Qwik displayed last in "Startup metrics (lighthouse with mobile simulation)", although it has the best scores?

[wmertens]

The frameworks are sorted by the first block scores I believe. Feel free to request better sorting on the benchmark repo :)

[krausest]

The frameworks are sorted by the first block scores I believe. Feel free to request better sorting on the benchmark repo :)

Only by default, because the benchmark's focus is certainly the CPU benchmark. You can already click on "geometric mean" for the memory or startup tables to sort according to their mean. So there's no request needed, just a click on the right place :)

[wmertens]

@krausest wouldn't it make more sense to sort the startup block by startup time?

[wenfangdu]

@krausest wouldn't it make more sense to sort the startup block by startup time?

@krausest I second that, the current behavior is very counter-intuitive, each block should be sorted separately according to their titles.

[krausest]

I got the message, but it doesn't fit in the current concept: I want keyed (and non-keyed as well) results to be behave as one table, such that the scroll position is synchronized for all of the four blocks (Duration, Memory, Startup and Transferred size). Allowing a varying colum order for each of the blocks would break that concept. We recently added sticky left column headers and initially it was without synchronized scrolling in the four blocks and I really hated it. Moreover if I sorted each block by default by each block's average it would be hard to see how good or bad the fastest frameworks do regarding memory or startup (I wouldn't find a good user interaction to sort the startup table by the order or the duration average to achieve that.) So for the time being please click on the row you want to sort by :)