search

ebdrup / json-schema-benchmark

Benchmarks for Node.js JSON-schema validators
MIT License
385 stars 40 forks source link

Misleading results for @exodus/schemasafe #52

Closed jchook closed 3 years ago

jchook commented 3 years ago

Hi, thanks very much for putting this benchmark together.

I found this repo very helpful in evaluating which JSON schema validator to use in my latest project. However, I found the benchmarks for @exodus/schemasafe misleading.

schemasafe disables error reporting by default, which seems like significantly different (and unexpected) behavior compared to the other validators in this benchmark.

I did see this note in CONTRIBUTING.md:

Also, do not tweak any of the settings or options of a validator. Each validator should be run in it's standard setting. This is to make sure that nothing breaks if the benchmarks changes. But also to reflect the kind of performance a user of the validator would get, if using it without tweaking.

However, in my opinion, to make an "apples to apples" comparison between schemasafe and the other validators, we should apply the includeErrors option to schemasafe.

ChALkeR commented 3 years ago

Yeah. This wasn't done on a purpose, I mentioned this back when I added schemasafe: #46 and proposed to enable includeErrors there.

Glad that this is resolved now ).