You should include the module is-my-json-valid by mafintosh, It's the fastest

ebdrup commented 9 years ago

is-my-json-valid is the fastest JSON-schema validator around. It blows the competition out of the water in speed. There is a blog post about it here: http://cosmicrealms.com/blog/2014/08/29/benchmark-of-node-dot-js-json-validation-modules-part-3/

ebdrup commented 9 years ago

I mean in your benchmarks

automatthew commented 9 years ago

If the validator doesn't support the full spec, it is not a JSON Schema validator. I can write a very fast validator that doesn't support all of JSON Schema:

is_valid = (schema, document) ->
  true

That's a reductio ad absurdum, obviously. But the point should be clear. If a validator doesn't reject documents that are invalid according to the JSON Schema standard, there can be no valid (pun intended) benchmarking of it in comparison to actual JSON Schema implementations.

automatthew commented 9 years ago

I do want to review is-my-json-valid, though. And we should add to the benchmarks a check that the tested libraries can actually detect invalid documents.

ebdrup commented 9 years ago

Actually I think it does the spec.

— Sent from Mailbox

On Sun, Jan 18, 2015 at 7:05 PM, Matthew King notifications@github.com wrote:

I do want to review is-my-json-valid, though. And we should add to the benchmarks a check that the tested libraries can actually detect invalid documents.

Reply to this email directly or view it on GitHub: https://github.com/pandastrike/jsck/issues/72#issuecomment-70418380

mafintosh commented 9 years ago

is-my-json-valid should support all of schema v4 (except for refRemote, ref, which this validator doesn't seem to support either)

ebdrup commented 9 years ago

It will be interesting to see how you handle this. You have a chance to really shine for the open source community.

automatthew commented 9 years ago

is-my-json-valid should support all of schema v4 (except for refRemote, ref, which this validator doesn't seem to support either)

Good deal. No, JSCK doesn't do refRemote yet.

automatthew commented 9 years ago

@mafintosh, I'm seeing some indications that is-my-json-valid does not, in fact, support all of JSON Schema draft 4. Feel free to email me if you want to discuss this in private.

mafintosh commented 9 years ago

@automatthew we can do it in public. which parts are missing?

automatthew commented 9 years ago

I don't see where, for instance, "multipleOf" is implemented.

Following that line of inquiry, I changed the value of that keyword in your test fixture to something that should cause failures for both the valid and invalid documents. All tests still pass.

gilesbowkett commented 9 years ago

sorry, we're going to pass on this. we can give you a shout-out in the readme or something.

is-my-json-valid is very fast but it doesn't seem to properly support the spec.

lots more detail here: https://www.pandastrike.com/posts/20150121-the-fast-and-the-spurious

automatthew commented 9 years ago

It will be interesting to see how you handle this. You have a chance to really shine for the open source community.

I found it very interesting.

mafintosh commented 9 years ago

Didn't know there was a json-schema test-suite. I'll make is-my-json-valid pass that. The orderly thing isn't used for anything internally (or has anything to do with the performance). When I researched json-schema initially i just liked the syntax but since I don't really use it anymore I'll probably remove it at some point.

automatthew commented 9 years ago

You may also find these documents helpful:

http://tools.ietf.org/html/draft-zyp-json-schema-04

http://tools.ietf.org/html/draft-fge-json-schema-validation-00

gilesbowkett commented 9 years ago

I took out most of the stuff about Orderly. Wasn't quite sure how that worked anyway. The good news is every time I tried it, I got [object Object] from toString(), so index.js was probably falling through to the catch every time. So everything I said about Orderly may have been irrelevant. The bad news is I got this result while trying to get is-my-json-valid to tell me that an invalid doc was valid, and that I got that to happen pretty quickly.

mafintosh commented 9 years ago

@gilesbowkett @automatthew is-my-json-valid 2.0.0 now passes the entire JSONSchema draft4 test suite (with exception of remote refs and multibyte unicode string with min/max length)

Running your benchmark yields the following result

## Benchmarks for Draft 4

Schema: 'Event - Valid document'.  A simple schema, exercising very few attributes
Sample size: 64
Validations per sample: 1024

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................

  JSCK: validations/millisecond
  median: 228.955    max: 254.094    min: 143.277

  tv4: validations/millisecond
  median: 125.429    max: 133.629    min: 82.071

  jayschema: validations/millisecond
  median: 1.933    max: 2.08    min: 1.554

  is-my-json-valid: validations/millisecond
  median: 1528.358    max: 2348.624    min: 965.127

  z-schema: validations/millisecond
  median: 97.789    max: 100.53    min: 63.175

Relative speeds:
is-my-json-valid : 1.000
JSCK : 6.675
tv4 : 12.185
z-schema : 15.629
jayschema : 790.847

Schema: 'Configuration'.  A moderately complex schema with some nesting and value constraints
Sample size: 64
Validations per sample: 256

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................

  JSCK: validations/millisecond
  median: 107.113    max: 127.745    min: 52.502

  tv4: validations/millisecond
  median: 36.626    max: 46.486    min: 12.066

  jayschema: validations/millisecond
  median: 1.017    max: 1.127    min: 0.703

  is-my-json-valid: validations/millisecond
  median: 563.877    max: 992.248    min: 503.937

  z-schema: validations/millisecond
  median: 43.866    max: 45.674    min: 27.465

Relative speeds:
is-my-json-valid : 1.000
JSCK : 5.264
z-schema : 12.855
tv4 : 15.395
jayschema : 554.556

Schema: 'Transaction'.
Sample size: 64
Validations per sample: 64

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................

  JSCK: validations/millisecond
  median: 15.885    max: 16.763    min: 14.408

  tv4: validations/millisecond
  median: 3.541    max: 3.632    min: 3.004

  jayschema: validations/millisecond
  median: 0.046    max: 0.05    min: 0.041

  is-my-json-valid: validations/millisecond
  median: 69.945    max: 87.075    min: 38.186

  z-schema: validations/millisecond
  median: 4.068    max: 4.455    min: 2.156

Relative speeds:
is-my-json-valid : 1.000
JSCK : 4.403
z-schema : 17.193
tv4 : 19.756
jayschema : 1519.454

is-my-json-valid is still consistently ~5x faster than JSCK

ebdrup commented 9 years ago

It will be interesting to see how you handle this. You have a chance to really shine for the open source community.

I found it very interesting.

It still is :-) But I'm really glad that you helped mafintosh make is-my-json-valid pass the tests.

gilesbowkett commented 9 years ago

reopened because https://github.com/pandastrike/jsck/pull/74, and well done @mafintosh.

just FYI, the pandastrike.com blog post about this will probably get updated accordingly.

mafintosh commented 9 years ago

@gilesbowkett cool! looking forward to reading it

gilesbowkett commented 9 years ago

here you go: https://www.pandastrike.com/posts/20150123-congrats-imjv

Still got to add an update to the pandastrike.com post about IMJV, but my personal blog post was updated with a link.

gilesbowkett commented 9 years ago

resolved in 5dba8432421c581a5798cc5c81643a8d2ada7286

pandastrike / jsck

You should include the module is-my-json-valid by mafintosh, It's the fastest #72

I do want to review is-my-json-valid, though. And we should add to the benchmarks a check that the tested libraries can actually detect invalid documents.