JSONPointerException: Recursive $ref

[JonasDaWi]

When I try to generate classes from the FHIR Schema, I get following Exception:

Exception in thread "main" net.pwall.json.pointer.JSONPointerException: Recursive $ref - /definitions/Extension at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:175) at net.pwall.json.schema.parser.Parser.parseRef(Parser.kt:324) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:202) at net.pwall.json.schema.parser.Parser.parseItems(Parser.kt:332) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:223) at net.pwall.json.schema.parser.Parser.parseProperties(Parser.kt:338) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:215) at net.pwall.json.schema.parser.Parser.parseRef(Parser.kt:324) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:202) at net.pwall.json.schema.parser.Parser.parseItems(Parser.kt:332) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:223) at net.pwall.json.schema.parser.Parser.parseProperties(Parser.kt:338) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:215) at net.pwall.json.schema.parser.Parser.parseRef(Parser.kt:324) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:202) at net.pwall.json.schema.parser.Parser.parseProperties(Parser.kt:338) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:215) at net.pwall.json.schema.parser.Parser.parseRef(Parser.kt:324) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:202) at net.pwall.json.schema.parser.Parser.parseCombinationSchema(Parser.kt:309) at net.pwall.json.schema.parser.Parser.parseSchema(Parser.kt:208) at net.pwall.json.schema.parser.Parser.parse(Parser.kt:149) at net.pwall.json.schema.parser.Parser.parse(Parser.kt:120) at net.pwall.json.schema.codegen.CodeGenerator.addTarget(CodeGenerator.kt:482) at net.pwall.json.schema.codegen.CodeGenerator.addTargets(CodeGenerator.kt:374) at net.pwall.json.schema.codegen.CodeGenerator.addTargets$default(CodeGenerator.kt:368) at net.pwall.json.schema.codegen.CodeGenerator.generate(CodeGenerator.kt:358) at net.pwall.json.schema.codegen.CodeGenerator.generate(CodeGenerator.kt:348)

Am I doing something wrong?

[pwall567]

The problem here is exactly what the exception message says – you have a recursive reference (to /definitions/Extension. JSON Schema draft 2019-09 introduced $recursiveRef, and later, draft 2020-12 introduced $dynamicRef, to handle recursive situations. Unfortunately, my implementation does not cover either of these two forms of reference.

There is work under way on a new version, building on this project and including full implementations of both the 2019-09 and the 2020-12 drafts, but that work is still some way off completion.

I'm sorry I'm unable to help with your case.

[aSemy]

I've had a very quick look at this, and when I remove the exception, nothing seems to break.

https://github.com/pwall567/json-kotlin-schema/blob/1eebbb741e169de197b729a22932dffa8686920e/src/main/kotlin/net/pwall/json/schema/parser/Parser.kt#L172-L178

        uri?.let {
            val fragmentURI = uri.resolve(pointer.toURIFragment())
            schemaCache[fragmentURI]?.let {
                return it
//                return if (it !is JSONSchema.False) it else throw JSONPointerException("Recursive \$ref - $pointer")
            }
            schemaCache[fragmentURI] = JSONSchema.False(uri, pointer)
        }

The generated code wasn't fully complete (lots of classes were empty, like open class Schemas), but at least the generator didn't crash.

Maybe there's some other situation where throwing the exception is necessary?

I was generating code using the OpenAPI spec schema https://github.com/OAI/OpenAPI-Specification/blob/aa91a19c43f8a12c02efa42d64794e396473f3b1/schemas/v3.0/schema.yaml

[pwall567]

The parser guards against recursion by storing a dummy entry in its cache (used to resolve $ref references). When parsing of an individual schema is complete, it replaces the dummy entry with the actual schema reference.

The code change above will return the dummy entry (equivalent to a Boolean schema of false) on a $ref lookup, so depending on the order in which the schema definitions are encountered, many references will return an empty object instead of the intended schema.

That may allow the parser to run to completion, but it will not produce correct results.

[volkert-fastned]

The parser guards against recursion by storing a dummy entry in its cache (used to resolve $ref references). When parsing of an individual schema is complete, it replaces the dummy entry with the actual schema reference.

@pwall567 There seems to be something wrong with that mechanism. I'm trying to get it to parse a directory of schemas, and they all end up being written to the same class file, with the name of the first schema it processed.

I opened a new issue for it.

[milux]

@pwall567 I just wanted to mention that recursive $ref usage is fine according to documentation on https://json-schema.org/understanding-json-schema/structuring#recursion.

I managed to fix the JSON parser (https://github.com/pwall567/json-kotlin-schema) accordingly, but fixing this codegen part is even harder. :see_no_evil:

[milux]

I finally managed to implement support for cyclic $refs, see PRs https://github.com/pwall567/json-kotlin-schema-codegen/pull/43 and https://github.com/pwall567/json-kotlin-schema/pull/20. Feedback etc. welcome. :wink: