Flake: TestClaimSchema

[carolynvs]

Lately I'm seeing TestClaimSchema fail just on CI. When I run it locally, it always passes (and has passed just fine previously).

Our dependency for jsonschema hasn't changed in a long time so I'm not sure what's causing the EOF.

=== RUN   TestClaimSchema
    claim_test.go:224: 
            Error Trace:    claim_test.go:224
            Error:          Received unexpected error:
                            EOF
                            unable to compile schema validator for schema

                            github.com/cnabio/cnab-go/schema.Validate
                                /home/vsts/work/1/s/schema/validation.go:50
                            github.com/cnabio/cnab-go/schema.ValidateClaim
                                /home/vsts/work/1/s/schema/validation.go:30
                            github.com/cnabio/cnab-go/claim.TestClaimSchema
                                /home/vsts/work/1/s/claim/claim_test.go:223
                            testing.tRunner
                                /opt/hostedtoolcache/go/1.17.8/x64/src/testing/testing.go:1259
                            runtime.goexit
                                /opt/hostedtoolcache/go/1.17.8/x64/src/runtime/asm_amd64.s:1581
            Test:           TestClaimSchema
            Messages:       failed to validate claim schema
--- FAIL: TestClaimSchema (0.00s)

[carolynvs]

Okay well it worked when run in the github action so that's a thing. 🤷‍♀️

[epsxy]

👋🏻 I have tried to look at this, I hope that's okay 😃 .

Investigation

Starting from the test function itself. The EOF error means that the schemaData we pass to gojsonschema.NewBytesLoader(schemaData) is empty. I have been able to replicate this error by putting an empty string into that function.

Locally, this test never fails because we read the schema data files from the disk, which are located in ./schema/schema/*.schema.json. But everything seems a bit different in the CI. I have seen that we are relying on another go program to download the latest schema data from cdn.cnab.io (this program is fetch-schemas.go). When the claims are retrieved, in the CI, we erase the repository version with the one retrieved from the web.

At this point my hypothesis was that the request failed and we erased the files with an empty string, because the response of the server was empty.

If we look at the CI, here, we can see:

Retrieving schema https://cdn.cnab.io/schema/cnab-core-1.2.0/bundle.schema.json
unable to fetch bundle schema with version cnab-core-1.2.0: unable to fetch schema from "[https://cdn.cnab.io/schema/cnab-core-1.2.0/bundle.schema.json"](https://cdn.cnab.io/schema/cnab-core-1.2.0/bundle.schema.json%22): Get "[https://cdn.cnab.io/schema/cnab-core-1.2.0/bundle.schema.json"](https://cdn.cnab.io/schema/cnab-core-1.2.0/bundle.schema.json%22): write tcp 10.1.0.146:59996->13.107.219.51:443: write: connection reset by peer
Retrieving schema https://cdn.cnab.io/schema/cnab-claim-1.0.0-DRAFT+b5ed2f3/claim.schema.json

Something happened here, the website didn't respond and returned an empty/invalid response, which made the test TestBundleSchema fails, with the same error you've mentionned with Claims above:

=== RUN   TestBundleSchema
    bundle_test.go:893: 
            Error Trace:    bundle_test.go:893
            Error:          Received unexpected error:
                            EOF
                            unable to compile schema validator for schema

                            github.com/cnabio/cnab-go/schema.Validate
                                /home/vsts/work/1/s/schema/validation.go:50
                            github.com/cnabio/cnab-go/schema.ValidateBundle
                                /home/vsts/work/1/s/schema/validation.go:25
                            github.com/cnabio/cnab-go/bundle.TestBundleSchema
                                /home/vsts/work/1/s/bundle/bundle_test.go:892
                            testing.tRunner
                                /opt/hostedtoolcache/go/1.17.8/x64/src/testing/testing.go:1259
                            runtime.goexit
                                /opt/hostedtoolcache/go/1.17.8/x64/src/runtime/asm_amd64.s:1581
            Test:           TestBundleSchema
            Messages:       failed to validate bundle schema

Conclusion

The error was probably caused because:

• we are fetching schema data files on the web instead of relying on the ones in the repository (probably because we want them to be up-to-date with the spec, I'm not sure about that)
• the website cdn.cnab.io didn't reply
• we are erasing the local schema even if the fetch request fails:

    bytes, err := fetchSchema(schema, version)
    if err != nil {
        fmt.Printf("unable to fetch %s schema with version %s: %s\n", schema, version, err.Error())
    }

    err = writeSchema(schema, bytes)
   //                         ^^^^ if the request fails, bytes is nil!

If we want to keep the current behavior, it would be possible to just avoid erasing the default schema data files at schema/schema/claim.schema.json and schema/schema/bundle.json just by adding a return (EDIT: continue is better as we don't want to skip the full for-loop, only the request which failed), like I did in this draft PR #290. So that if for some reason the request fails, the tests still pass.