Open josvazg opened 7 years ago
iainbeeston
commented
7 years ago Thanks for the detailed bug report. Would it be possible for you to post in a longer stack trace? Stack level too deep errors are usually formed by unterminated recursion, but it's hard to see where that's occurring.
If you don't have the stack trace to hand then I'll try to reproduce using the instructions you gave above
josvazg
commented
7 years ago I was using wtf! and could not figure out how to get more lines, that is the 10 I got. Forgive me, but I am a Ruby newbie and there might be a better way I don't know about.
I found that even as json-schema was correctly discriminating already visited URIs from new ones, BUT it would still do a dup to clone the URI object and that would call the schema checking method in which the "stack level too deep" error is always triggered from. I assume it could be that the validator or reader keeps visiting already processed URIs and thus never finishing the processing.
I still have a branch to try this if you need me to, but I think it will be useful that you try to reproduce on your side with a clean minimal sample. If that does NOT fail for you I'd like to see that code.
iainbeeston
commented
7 years ago Ok no worries, I'll try to reproduce and see if I get the same result for your example.
Could you possibly give me some sample json to test against that schema? (Will make it easier for me to reproduce the problem)
josvazg
commented
7 years ago aha! As I suspected a simple empty json "{}" does the trick as this happens on schema loading, validation of the json data is not even started...
[1] pry(main)> json = '{}'
=> "{}"
[2] pry(main)> require 'json-schema'
=> true
[3] pry(main)> ARM_SCHEMA="https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json"
=> "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json"
[4] pry(main)> JSON::Validator.validate!(ARM_SCHEMA, json)
SystemStackError: stack level too deep
from .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:873:in `=~'
[5] pry(main)> wtf!
Exception: SystemStackError: stack level too deep
--
0: .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:873:in `=~'
1: .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:873:in `!~'
2: .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:873:in `scheme='
3: .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:799:in `block in initialize'
4: .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:2355:in `call'
5: .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:2355:in `defer_validation'
6: .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:796:in `initialize'
7: .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:2228:in `new'
8: .../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:2228:in `dup'
9: .../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:217:in `handle_schema'Ok great, that's helpful, thanks
josvazg
commented
7 years ago A colleague hinted me to use _ex_.backtrace instead, and that worked beatifully:
[8] pry(main)> _ex_.backtrace
=> [".../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:873:in `=~'",
".../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:873:in `!~'",
".../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:873:in `scheme='",
".../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:799:in `block in initialize'",
".../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:2355:in `call'",
".../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:2355:in `defer_validation'",
".../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:796:in `initialize'",
".../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:2228:in `new'",
".../ruby/gems/2.2.0/gems/addressable-2.5.1/lib/addressable/uri.rb:2228:in `dup'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:217:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:193:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:193:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:193:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:203:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:202:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:202:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:179:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:142:in `load_ref_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:151:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:193:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:179:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:179:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:142:in `load_ref_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:151:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:193:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:179:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:179:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:179:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:179:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:142:in `load_ref_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:151:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:193:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:192:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:190:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:179:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:222:in `handle_schema'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:179:in `block (2 levels) in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:178:in `block in build_schemas'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `each'",
".../ruby/gems/2.2.0/gems/json-schema-2.8.0/lib/json-schema/validator.rb:176:in `build_schemas'"
...Let me know if you need more lines of traces.
iainbeeston
commented
7 years ago So, this is a really interesting issue. What's happening here is that deploymentTemplate.json includes several $refs that refer to Microsoft.Sql.json (eg. here). Microsoft.Sql.json in turn has a lot of refs that refer back to deploymentTemplate.json (eg. (here)[https://github.com/Azure/azure-resource-manager-schemas/blob/master/schemas/2014-04-01/Microsoft.Sql.json#L35]). The json-schema gem is naively trying to follow these refs and falls into an infinitely loop.
Normally I'd say this is a bug in the json-schema gem, but strangely enough the json-schema spec explicitly forbids this kind of thing. If you look at the official spec it says that:
A schema MUST NOT be run into an infinite loop against a schema.. Schemas SHOULD NOT make use of infinite recursive nesting... the behavior is undefined.
iainbeeston
commented
7 years ago I'll see if I can raise an issue with Microsoft about this.
josvazg
commented
7 years ago I guess then there are here 2 separate issues:
json_schema should protect itself against cycles and report them, if an URI reference is mentioned again on the same branch, it should then raise an error such as "Invalid Schema tree, can't have reference cycles!" and probably show the cycle or at least mention the repeated reference.
Azure must eventually fix their schemas to be compliant and not have cycles, if they want to use them properly. So yes, an issue raised to them as well seems proper.
What do you think?
iainbeeston
commented
7 years ago Yes that sounds like a good plan. json-schema does already have some protection against cyclic refs but not in this particular case.
This can be reproduced by trying to validate any valid JSON against Azure's ARM JSON Schema at: https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json
The call:
Will end with:
Debugging it a bit I counted 926 different URIs this ARM templates include in their subtree before the failure.
They might include cycles making the validator loop or recurse forever, or it might just be too many, but the fact is that the validator chokes on this input ALWAYS when trying to duplicate yet another re-visited URI.
In my tests this happens after the 926 different URIs seems to be a stable number, so no other new templates are being found yet the validator keeps on running, until it blows up the stack.