pboettch
commented
3 years ago Not sure if this is the right place to ask this,
Right place! 😃
but I'm considering using this library within a multi-threaded C++ service to validate JSON per user request and using many different (custom) schemas, with references.
Multiple schemas or one big schema where everything is referenced and you would to validate instances to sub-schemas?
I need to implement schema resource sharing and management (for validator instances) across all of the client threads to reduce memory use and schema loading latency. (We have bursty use cases were a client may need to validate using the same schema 1000's of times in a row.)
OK.
My first question is - is it possible/safe for multiple client threads to concurrently validate different JSON documents against the same validator instance (same schema)?
Yes. The validator is thread-safe for validating (see constness of the validate()-method) - thus can be used from different threads concurrently. Not so for loading: you would need to load the schema in one (main) thread before using.
Next, I see that the schema loader expects the schema JSON to be assigned and returned via a JSON parameter. In my scenario, it would be better if I could return a reference to an already initialized schema validator instance (that would be cached in my schema resource pool). I don't mind making this change, but I wanted to ask if there would be any architectural caveats to this approach before I look into it. The idea is for each client thread to assemble a specific validator from existing loaded and initialized validators to reduce latency and total memory use.
Interesting idea and problem.
I never used huge schemas but had one with references in every sens (recursive, partly nested). It worked well.
While reading your description I immediately thought of making a super-schema which references all of your schemas, so parsing and instantiating would be done once and all validators would only exist once - without changing the library.
Then we would need to add a method to get a (C++) reference to a sub-schema (which actually is a schema) with an URL which can then be used to validate.
Or, we would add an optional parameter to the validate-method which defines the entry-point of the sub-schema to use. I prefer this method - I'm not sure getting sub-schemas as C++-references will be trivial to implement.
Your idea of combining different instances of validators won't work, because of the absolute URLs used to reference sub-schemas. There would probably be duplicated schemas.
If memory usage is a problem, currently the validator could release some memory after finishing loading (and resolving) all references. Out of lazynes I never took care of that.
Another optimization could be to remove duplicated (sub-)schemas.
Could you provide an example
dvstans
Hi, Not sure if this is the right place to ask this, but I'm considering using this library within a multi-threaded C++ service to validate JSON per user request and using many different (custom) schemas, with references. I need to implement schema resource sharing and management (for validator instances) across all of the client threads to reduce memory use and schema loading latency. (We have bursty use cases were a client may need to validate using the same schema 1000's of times in a row.)
My first question is - is it possible/safe for multiple client threads to concurrently validate different JSON documents against the same validator instance (same schema)?
Next, I see that the schema loader expects the schema JSON to be assigned and returned via a JSON parameter. In my scenario, it would be better if I could return a reference to an already initialized schema validator instance (that would be cached in my schema resource pool). I don't mind making this change, but I wanted to ask if there would be any architectural caveats to this approach before I look into it. The idea is for each client thread to assemble a specific validator from existing loaded and initialized validators to reduce latency and total memory use.
Thanks!