jhthorsen / json-validator

:cop: Validate data against a JSON schema
https://metacpan.org/release/JSON-Validator
56 stars 58 forks source link

Adds an option to disable the check that tracks the data structure being validated for recursion #220

Closed jlcooper closed 4 years ago

jlcooper commented 4 years ago

Summary

This pull request adds an attribute that can be used to deactivate the active recursive data protection logic for a JSON::Validator instance.

Motivation

When validating a data structure the recursive data protection has to track the references seen as a way to avoid infinite loops. When validating a large data structure tracking the references can use a significant amount of memory and if the caller is willing to guarantee that their data structure doesn't contain any loops then being able to disable it can reduce their process's memory usage.

jlcooper commented 4 years ago

I've modified my debugging script to use some real life data (about 34MB) and logged the .data segment of the Perl process before validating the data and after validating the data.

With recursive_data_protection active:

                               data (kB)
  Before test data validated    87424
  After test data validated    752724
  Difference                   665300

With recursive_data_protection deactivated:

                               data (kB)
  Before test data validated    87400
  After test data validated    353868 
  Difference                   266468

Difference between active and inactive ~ 398 MB
jlcooper commented 4 years ago

I've added a unit test to test the recursive data protection logic. It's a little more intrusive in the way that it tests the logic than I'd like, but we're having to test how it gets a result rather than the result it gets.

jhthorsen commented 4 years ago

I'm not a fan of this at all, but I'm going to take the PR anyways, since the attribute is marked as EXPERIMENTAL. I have a gut feeling this can be fixed in a better way, but I don't know how.