Big deployment template schema doesn't work

[TyOverby]

JSON Schema file in question: http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json

Running quicktype -s schema deploymentTemplate.json -o deps.ts yields a rather small output file.

You'll notice the resources: any[] field. This is where the bulk of the schema code should be.

Fair warning: once the URL links fully resolved, this file is massive. I wrote a compiler for it once, and it produced tens of thousands of lines of typescript.

[schani]

We don't support external $ref references and allOf yet, unfortunately.

What types would you expect this file to generate?

[TyOverby]

The "expected" output is too big to reason about, but I would expect that the typescript would successfully typecheck all the json that the schema typechecks.

[schani]

I'm curious: if you only care about validation, why wouldn't you use a schema validator like ajv?

[TyOverby]

Oh, it's because I'd like the really nice tooling in Typescript that you get from having the schema translated.

I should add that I'm not really worried about this bug, like I said, I'm currently using a custom json-schema->typescript compiler that I made for this file specifically. I'd like to move off it, and quicktype looks neat, but no rush.

[TyOverby]

Here's the same file, but with all of the $ref references inlined. Warning, it's 93,000 lines.

https://gist.githubusercontent.com/TyOverby/2f4a7036a511684c99accba28128e204/raw/29f25d3067dc7b28a60030e5b256813ea925f5d3/deploymentTemplate.json

[schani]

Thanks!

How do you compile allOf?

[TyOverby]

allOf: [ a, b, c ] becomes a & b & c

[schani]

@TyOverby I implemented all-of in #574. It doesn't completely accept your schema yet, because the schema contains enums with integers and booleans, which I haven't implemented yet. I removed all those here. The types I get look a LOT simpler than the schema, so I'm very skeptical. Could you please provide me with a couple of JSON files that satisfy that schema or, if that's not possible, test it yourself and give me feedback?

[TyOverby]

@schani Yeah, here's a repo filled with them!

https://github.com/Azure/azure-quickstart-templates

Every one of those folders has an azuredeploy.json file that (should) satisfy that schema.

[schani]

@TyOverby I went over all those JSONs with ajv, and most of them do not satisfy the schema. Here's a list of the ones that do.

[TyOverby]

hahaha, good thing I threw that "(should)"!

[schani]

@TyOverby Here's one oddity in your schema: There are quite a few places where you specify a type of boolean, but then for the same type specify an enum containing strings. Example:

{
    "type": "boolean",
    "enum": [
        "true"
    ]
}

This doesn't match any value, which implies that if a value is present in the JSON for that type, the JSON is invalid with respect to the schema.

I'm right now working on making your schema work with quicktype - all the pieces are finally in place - but we won't support "invalid" types for the time being. I'm fixing it by changing the strings to the booleans I'm assuming they should represent.

[schani]

You're doing the same thing with integers.

[TyOverby]

Yeah, honestly this isn't super surprising. This massive schema is built from multiple schamas authored by dozens of teams across Microsoft Azure.

Do you have an easy way of getting a list of the paths for these non-conformant types? I could file bugs internally.

[schani]

@TyOverby No easy way right now. If you just search for "true", "false" and "0" I think should get them all.

[schani]

@TyOverby My latest working branch supports most of the valid JSONs now, with the caveat that we end up with a little too many any types than I'd like - I need to figure out a way to debug where in the schema they come from. The JSONs that don't work are these:

101-redis-cache-azuredeploy.json
101-webappazure-oms-monitoring-azuredeploy.json
201-redis-premium-cluster-diagnostics-azuredeploy.json
201-redis-premium-persistence-azuredeploy.json
201-redis-premium-vnet-cluster-diagnostics-azuredeploy.json

The problem in all of them is the same: A resource with type of either Microsoft.Cache/redis/providers/diagnosticsettings or runbooks, both of which I couldn't find in the Schema, so I figured quicktype might be onto something. Could you look into that, please?

I'll also have to re-run the validation with the Schema with the enums fixed.

[dpwrussell]

@TyOverby Hi. Did you get any further with your efforts to generate objects from the Azure schema? I just tried to do the same thing myself and it would be great to discuss if you were able to make progress in this area?

Were you trying to generate objects to help you build ARM templates imperatively?

[TyOverby]

@dpwrussell: We had success with the translation into typescript for the specific parts of the azure schema that we cared about. You can find an old version of the code here: https://github.com/TyOverby/arm-compiler

Unfortunately we weren't able to use quicktype at the time, so I wrote my own compiler, which you can find pieces of in the project above.