search

pandastrike / jsck

JSON Schema Compiled checK
MIT License
158 stars 14 forks source link

Fixes #72 add is-my-json-valid to benchmarks #73

Closed ebdrup closed 9 years ago

ebdrup commented 9 years ago

Sorry but is-my-json-valid is a lot faster than jsck. Take a look at the Repo. @mafintosh explains his technique.

ebdrup commented 9 years ago

Below are the benchmark results on my Mac Air is-my-json-valid is 10x to 1428x faster than jsck

## Benchmarks for Draft 4

Schema: 'Event - Valid document'.  A simple schema, exercising very few attributes
Sample size: 64
Validations per sample: 1024

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................

  JSCK: validations/millisecond
  median: 102.226    max: 157.417    min: 37.856 

  tv4: validations/millisecond
  median: 52.694    max: 77.623    min: 20.575 

  jayschema: validations/millisecond
  median: 0.76    max: 0.856    min: 0.468 

  is-my-json-valid: validations/millisecond
  median: 1078.462    max: 1875.458    min: 276.085 

  z-schema: validations/millisecond
  median: 56.694    max: 79.503    min: 30.831 

Relative speeds:
is-my-json-valid : 1.000
JSCK : 10.550
z-schema : 19.023
tv4 : 20.467
jayschema : 1419.130

Schema: 'Configuration'.  A moderately complex schema with some nesting and value constraints
Sample size: 64
Validations per sample: 256

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................

  JSCK: validations/millisecond
  median: 62.447    max: 86.662    min: 18.461 

  tv4: validations/millisecond
  median: 24.299    max: 32.821    min: 8.142 

  jayschema: validations/millisecond
  median: 0.4    max: 0.496    min: 0.373 

  is-my-json-valid: validations/millisecond
  median: 840.722    max: 1132.743    min: 398.134 

  z-schema: validations/millisecond
  median: 25.15    max: 36.41    min: 10.922 

Relative speeds:
is-my-json-valid : 1.000
JSCK : 13.463
z-schema : 33.429
tv4 : 34.599
jayschema : 2103.568

Schema: 'Transaction'.  
Sample size: 64
Validations per sample: 64

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................

  JSCK: validations/millisecond
  median: 7.47    max: 11.88    min: 3.207 

  tv4: validations/millisecond
  median: 1.758    max: 2.217    min: 0.958 

  jayschema: validations/millisecond
  median: 0.024    max: 0.026    min: 0.018 

  is-my-json-valid: validations/millisecond
  median: 10666.667    max: 10666.667    min: 9142.857 

  z-schema: validations/millisecond
  median: 2.444    max: 3.132    min: 1.348 

Relative speeds:
is-my-json-valid : 1.000
JSCK : 1428.000
z-schema : 4364.583
tv4 : 6066.583
jayschema : 444760.083
gilesbowkett commented 9 years ago

sorry, while it's true that is-my-json-valid is very fast, it seems to have implementation and reliability issues. both I and @automatthew got is-my-json-valid to declare invalid documents valid, for different reasons.

lots more detail here: https://www.pandastrike.com/posts/20150121-the-fast-and-the-spurious

mafintosh commented 9 years ago

As mentioned in #72 is-my-json-valid 2.0.0 now supports all of JSONSchema draft4 (and passes the JSONSchema draft4 test suite). Could we reopen this PR and add the benchmark?

mafintosh commented 9 years ago

This PR used the error handler wrongly so I opened a new one in #74 - no need to reopen this