search

cuelang / cue

CUE has moved to https://github.com/cue-lang/cue
https://cuelang.org
Apache License 2.0
3.08k stars 171 forks source link

encoding/jsonschema: various issues importing GitHub schemas #378

Closed myitcv closed 4 years ago

myitcv commented 4 years ago

What version of CUE are you using (cue version)?

$ cue version
cue version +0d4abd7 linux/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

exec curl -s -o github-workflow.json https://raw.githubusercontent.com/SchemaStore/schemastore/master/src/schemas/json/github-workflow.json
exec cue def jsonschema: github-workflow.json -p schema

What did you expect to see?

Successful schema import

What did you see instead?

invalid identifier "working-directory"
myitcv commented 4 years ago

FWIW https://github.com/SchemaStore/schemastore/tree/master/src/schemas/json contains quite a good set of test cases for the JSON schema translation:

cd $(mktemp -d)
git clone https://github.com/SchemaStore/schemastore
cd schemastore/src/schemas/json
for i in $(ls *.json)
do
cue def jsonschema: $i -p schema > $i.cue 2> $i.cue.errors && rm $i.cue.errors || echo "$i has problems"
done

Gives:

azure-iot-edge-deployment-1.0.json has problems
azure-iot-edge-deployment-2.0.json has problems
azure-iot-edge-deployment-template-1.0.json has problems
azure-iot-edge-deployment-template-2.0.json has problems
bootstraprc.json has problems
bukkit-plugin.json has problems
bungee-plugin.json has problems
chrome-manifest.json has problems
circleciconfig.json has problems
cirrus.json has problems
cloudbuild.json has problems
compilerconfig.json has problems
component.json has problems
composer.json has problems
content-security-policy-report-2.json has problems
cryproj.52.schema.json has problems
cryproj.53.schema.json has problems
cryproj.54.schema.json has problems
cryproj.55.schema.json has problems
cryproj.dev.schema.json has problems
cryproj.json has problems
datalogic-scan2deploy-android.json has problems
datalogic-scan2deploy-ce.json has problems
dependabot.json has problems
docfx.json has problems
dss-2.0.0.json has problems
eslintrc.json has problems
function.json has problems
github-workflow.json has problems
grunt-clean-task.json has problems
grunt-copy-task.json has problems
grunt-cssmin-task.json has problems
grunt-jshint-task.json has problems
hemtt-0.6.2.json has problems
host.json has problems
htmlhint.json has problems
install.json has problems
jdt.json has problems
jekyll.json has problems
jsbeautifyrc-nested.json has problems
jsdoc-1.0.0.json has problems
lsdlschema-1.2.json has problems
lsdlschema.json has problems
modernizrrc.json has problems
mtad.json has problems
mta.json has problems
neoload.json has problems
ninjs-1.0.json has problems
ninjs-1.1.json has problems
ninjs-1.2.json has problems
nodehawkrc.json has problems
nodemon.json has problems
ocelot.json has problems
opspec-io-0.1.7.json has problems
project-1.0.0-rc1.json has problems
project-1.0.0-rc2.json has problems
project.json has problems
proxies.json has problems
pubspec.json has problems
renovate.json has problems
sarif-external-property-file-2.1.0-rtm.0.json has problems
sarif-external-property-file-2.1.0-rtm.1.json has problems
sarif-external-property-file-2.1.0-rtm.2.json has problems
sarif-external-property-file-2.1.0-rtm.3.json has problems
sarif-external-property-file-2.1.0-rtm.4.json has problems
sarif-external-property-file-2.1.0-rtm.5.json has problems
sarif-external-property-file.json has problems
schema-draft-v4.json has problems
schema-org-action.json has problems
schema-org-contact-point.json has problems
schema-org-place.json has problems
schema-org-thing.json has problems
sourcemap-v3.json has problems
swagger-2.0.json has problems
template.json has problems
templatesources.json has problems
toolinfo.1.1.0.json has problems
travis.json has problems
tsconfig.json has problems
tslint.json has problems
typings.json has problems
typingsrc.json has problems
ui5-manifest.json has problems
up.json has problems
vega.json has problems
vega-lite.json has problems
vim-addon-info.json has problems
vs-2017.3.host.json has problems
vs-nesting.json has problems
vss-extension.json has problems
webextension.json has problems
web-types.json has problems
mpvl commented 4 years ago

Forgot to label this, but this should have been fixed. It now maps such odd cases to a special map of irregular definitions called #def. So in this case #def: "working-directory": .... I'm assuming these cases should be rare as schema which such oddly named fields make not much sense (considering interop compatibility). But if there is a common class of characters that appears in such fields is identified, we could consider introducing back-quoted identifiers or so (again), like #working-directory, for a limited set of characters.

myitcv commented 4 years ago

Do you want separate issues opened for the other issues thrown up by https://github.com/cuelang/cue/issues/378#issuecomment-624832339?

A rough list of them:

schema expects mapping node, found string:
    ./azure-iot-edge-deployment-1.0.json:36:41
external references (http://json.schemastore.org/azure-iot-edge-deployment-1.0#/properties/modulesContent/properties/$edgeAgent/properties/properties.desired/properties/runtime) not supported:
    ./azure-iot-edge-deployment-template-1.0.json:40:45
invalid JSON for file "/tmp/tmp.CYRyAR5lD4/schemastore/src/schemas/json/bootstraprc.json": invalid character 'ï' looking for beginning of value:
    ./bootstraprc.json:1:1
invalid reference "#/definitions/builtinSteps/documentation/checkout": $ref must be of the form #/definitions/...:
    ./circleciconfig.json:761:19
anchors () not supported:
    ./component.json:134:12
#variables: non-concrete value struct:
    55:128
    99:13

A panic:

panic: interface conversion: ast.Decl is *ast.EmbedDecl, not *ast.Field [recovered]
    panic: interface conversion: ast.Decl is *ast.EmbedDecl, not *ast.Field

goroutine 1 [running]:
cuelang.org/go/cmd/cue/cmd.recoverError(0xc000525ec0)
    /home/myitcv/dev/cuelang/cue/cmd/cue/cmd/root.go:219 +0x95
panic(0xbb92e0, 0xc00061da10)
    /home/myitcv/gos/src/runtime/panic.go:969 +0x175
cuelang.org/go/encoding/jsonschema.glob..func27(0xc00015c380, 0xc0002aa190, 0xc000489b00)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/constraints.go:442 +0x83c
cuelang.org/go/encoding/jsonschema.(*state).schemaState.func1(0xc0000a6871, 0x8, 0xc00015c380, 0xc0002aa190)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:420 +0xc3
cuelang.org/go/encoding/jsonschema.(*state).processMap(0xc000489b00, 0xc00015c380, 0xc000270f50, 0xc000524988)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:473 +0x11a
cuelang.org/go/encoding/jsonschema.(*state).schemaState(0xc0004899e0, 0xc00015c380, 0xc000270f50, 0xdf, 0xc000270f00, 0xc000524a01, 0xc0002d48b0, 0x1)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:409 +0x16c
cuelang.org/go/encoding/jsonschema.(*state).schema(...)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:384
cuelang.org/go/encoding/jsonschema.glob..func33(0xc00015c380, 0xc000270f00, 0xc0004899e0)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/constraints.go:576 +0x150
cuelang.org/go/encoding/jsonschema.(*state).schemaState.func1(0xc0000a66e1, 0x5, 0xc00015c380, 0xc000270f00)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:420 +0xc3
cuelang.org/go/encoding/jsonschema.(*state).processMap(0xc0004899e0, 0xc00015c380, 0xc000270d70, 0xc000524c38)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:473 +0x11a
cuelang.org/go/encoding/jsonschema.(*state).schemaState(0xc000121560, 0xc00015c380, 0xc000270d70, 0xdf, 0xd72f00, 0xc000212070, 0xd74f80, 0xc000212070)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:409 +0x16c
cuelang.org/go/encoding/jsonschema.glob..func26.1(0xc0000a7c51, 0x5, 0xc00015c380, 0xc000270d70)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/constraints.go:405 +0x96
cuelang.org/go/encoding/jsonschema.(*state).processMap(0xc000121560, 0xc00015c380, 0xc000221590, 0xc000524e10)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:473 +0x11a
cuelang.org/go/encoding/jsonschema.glob..func26(0xc00015c380, 0xc000221590, 0xc000121560)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/constraints.go:402 +0xaa
cuelang.org/go/encoding/jsonschema.(*state).schemaState.func1(0xc0000a6661, 0xa, 0xc00015c380, 0xc000221590)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:420 +0xc3
cuelang.org/go/encoding/jsonschema.(*state).processMap(0xc000121560, 0xc00015c380, 0xc000221450, 0xc000524fd8)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:473 +0x11a
cuelang.org/go/encoding/jsonschema.(*state).schemaState(0xc000346fc0, 0xc00015c380, 0xc000221450, 0xdf, 0xd73200, 0xc0002214a0, 0x1, 0x2)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:409 +0x16c
cuelang.org/go/encoding/jsonschema.(*state).schema(...)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:384
cuelang.org/go/encoding/jsonschema.glob..func31.1(0xc0000a78e1, 0xc, 0xc00015c380, 0xc000221450)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/constraints.go:524 +0x2ce
cuelang.org/go/encoding/jsonschema.(*state).processMap(0xc000346fc0, 0xc00015c380, 0xc000295360, 0xc000525200)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:473 +0x11a
cuelang.org/go/encoding/jsonschema.glob..func31(0xc00015c380, 0xc000295360, 0xc000346fc0)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/constraints.go:516 +0xfb
cuelang.org/go/encoding/jsonschema.(*state).schemaState.func1(0xc000038781, 0x11, 0xc00015c380, 0xc000295360)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:420 +0xc3
cuelang.org/go/encoding/jsonschema.(*state).processMap(0xc000346fc0, 0xc00015c380, 0xc000682a50, 0xc0005253d8)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:473 +0x11a
cuelang.org/go/encoding/jsonschema.(*state).schemaState(0xc000346ea0, 0xc00015c380, 0xc000682a50, 0xdf, 0x40f300, 0xc0006191a0, 0x10, 0x10)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:409 +0x16c
cuelang.org/go/encoding/jsonschema.(*decoder).schema(0xc000621340, 0x0, 0x0, 0x0, 0xc00015c380, 0xc000682a50, 0x0, 0x1, 0x203000)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:124 +0x105
cuelang.org/go/encoding/jsonschema.(*decoder).decode(0xc000621340, 0xc00015c380, 0xc000682a50, 0xc22740)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/decode.go:71 +0xbd
cuelang.org/go/encoding/jsonschema.Extract(0xc0001f6090, 0xc000621300, 0x0, 0x0, 0xd5bd40)
    /home/myitcv/dev/cuelang/cue/encoding/jsonschema/jsonschema.go:49 +0x8b
cuelang.org/go/internal/encoding.jsonSchemaFunc.func1(0xc0001f6090, 0xc000360120, 0xc0001f6090, 0x0, 0x0, 0xd73720)
    /home/myitcv/dev/cuelang/cue/internal/encoding/encoding.go:260 +0x10d
cuelang.org/go/internal/encoding.(*Decoder).doInterpret(0xc0001f6000)
    /home/myitcv/dev/cuelang/cue/internal/encoding/encoding.go:99 +0xd0
cuelang.org/go/internal/encoding.(*Decoder).Next(0xc0001f6000)
    /home/myitcv/dev/cuelang/cue/internal/encoding/encoding.go:86 +0xa8
cuelang.org/go/internal/encoding.NewDecoder(0xc000360fc0, 0xc0001f6120, 0x0)
    /home/myitcv/dev/cuelang/cue/internal/encoding/encoding.go:217 +0x6fb
cuelang.org/go/cmd/cue/cmd.parseArgs(0xc000348a40, 0xc000356fc0, 0x2, 0x4, 0xc000357000, 0x14, 0x14, 0x13)
    /home/myitcv/dev/cuelang/cue/cmd/cue/cmd/common.go:426 +0xbe5
cuelang.org/go/cmd/cue/cmd.runDef(0xc000348a40, 0xc000356fc0, 0x2, 0x4, 0x0, 0x0)
    /home/myitcv/dev/cuelang/cue/cmd/cue/cmd/def.go:54 +0x92
cuelang.org/go/cmd/cue/cmd.mkRunE.func1(0xc000230000, 0xc000356fc0, 0x2, 0x4, 0x0, 0x0)
    /home/myitcv/dev/cuelang/cue/cmd/cue/cmd/root.go:46 +0x6c
github.com/spf13/cobra.(*Command).execute(0xc000230000, 0xc000356f80, 0x4, 0x4, 0xc000230000, 0xc000356f80)
    /home/myitcv/gostuff/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:838 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001ef600, 0x0, 0x0, 0x0)
    /home/myitcv/gostuff/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:943 +0x336
github.com/spf13/cobra.(*Command).Execute(...)
    /home/myitcv/gostuff/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:883
cuelang.org/go/cmd/cue/cmd.(*Command).Run(0xc000348a40, 0xd6b840, 0xc000036148, 0x0, 0x0)
    /home/myitcv/dev/cuelang/cue/cmd/cue/cmd/root.go:204 +0x65
cuelang.org/go/cmd/cue/cmd.mainErr(0xd6b840, 0xc000036148, 0xc0000321f0, 0x5, 0x5, 0xcb1ad0, 0xc00033df48)
    /home/myitcv/dev/cuelang/cue/cmd/cue/cmd/root.go:143 +0x8a
cuelang.org/go/cmd/cue/cmd.Main(0xc00007c058)
    /home/myitcv/dev/cuelang/cue/cmd/cue/cmd/root.go:125 +0x9c
main.main()
    /home/myitcv/dev/cuelang/cue/cmd/cue/main.go:24 +0x25
myitcv commented 4 years ago

Oops, forgot to re-open with previous comment in https://github.com/cuelang/cue/issues/378#issuecomment-629027509

mpvl commented 4 years ago

There is a mix of things going on. There are CUE bugs (like the mapping node error) and things CUE does not yet support, like external references or anchors.

But there is some interesting stuff going on in these files. For instance, I was not aware that this was legal JSON schema:

cueckoo commented 3 years ago

This issue has been migrated to https://github.com/cue-lang/cue/issues/378.

For more details about CUE's migration to a new home, please see https://github.com/cue-lang/cue/issues/1078.