search

daveshanley / vacuum

vacuum is the worlds fastest OpenAPI 3, OpenAPI 2 / Swagger linter and quality analysis tool. Built in go, it tears through API specs faster than you can think. vacuum is compatible with Spectral rulesets and generates compatible reports.
https://quobix.com/vacuum
MIT License
490 stars 39 forks source link

Vacuum seems to try to load unrelated files #400

Closed ntotten closed 6 months ago

ntotten commented 6 months ago

When i run this command using version 0.6.1:

vacuum lint -k -s -d -e -b -r packages/runtime/config/policy-ruleset.yaml packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json

I get the error:

{"time":"2023-12-18T13:07:56.051456-05:00","level":"ERROR","msg":"unable to open the rolodex file, check specification references and base path","file":"/Users/ntotten/zuplo/projects/core/test-fixtures/schema-tests/schemas/recordstatus.json","error":"open /Users/ntotten/zuplo/projects/core/test-fixtures/schema-tests/schemas/recordstatus.json: file does not exist"}
{"time":"2023-12-18T13:07:56.053448-05:00","level":"ERROR","msg":"unable to open the rolodex file, check specification references and base path","file":"/Users/ntotten/zuplo/projects/core/packages/core/src/__tests__/routes-source-gen-snapshots/$import(./schemas/order.json)","error":"open /Users/ntotten/zuplo/projects/core/packages/core/src/__tests__/routes-source-gen-snapshots/$import(./schemas/order.json): file does not exist\nopen /Users/ntotten/zuplo/projects/core/packages/core/src/__tests__/routes-source-gen-snapshots/$import(./schemas/order.json): invalid argument"}
{"time":"2023-12-18T13:07:56.054583-05:00","level":"ERROR","msg":"unable to open the rolodex file, check specification references and base path","file":"/Users/ntotten/zuplo/projects/core/packages/core/src/__tests__/routes-source-gen-snapshots/$import(./schemas/response.json)","error":"open /Users/ntotten/zuplo/projects/core/packages/core/src/__tests__/routes-source-gen-snapshots/$import(./schemas/response.json): file does not exist\nopen /Users/ntotten/zuplo/projects/core/packages/core/src/__tests__/routes-source-gen-snapshots/$import(./schemas/response.json): invalid argument"}
panic: runtime error: index out of range [1] with length 1

goroutine 877 [running]:
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14004990a00, 0x1400497e5a0, 0x1400497e320, {0x14001eda440?, 0x1055cb9ee?, 0x2?}, 0xffffffffffffffff?, 0x0?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:185 +0x3464
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14004990a00, 0x1400497e320, 0x1400497de00, {0x14005780920?, 0x1055cb9ee?, 0x2?}, 0xffffffffffffffff?, 0x4?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14004990a00, 0x1400497de00, 0x140036a9180, {0x1400221bee0?, 0x1055cb9ee?, 0x2?}, 0xffffffffffffffff?, 0x0?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14004990a00, 0x140036a9180, 0x140036a9040, {0x1400403fdb8?, 0x12df0ff60?, 0x106b461d8?}, 0x30?, 0x80?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.createNewIndex(0x140036a9040, 0x14004990a00, 0x1)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/spec_index.go:80 +0x158
github.com/pb33f/libopenapi/index.NewSpecIndexWithConfig(0x140036a9040, 0x14000592480)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/spec_index.go:51 +0x2d8
github.com/pb33f/libopenapi/index.(*LocalFile).Index(0x1400375cd20, 0x3a22796669727261?)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex_file_loader.go:191 +0x50
github.com/pb33f/libopenapi/index.(*Rolodex).IndexTheRolodex.func1.1({0x1311df780, 0x1400375cd20}, {0x14000c89c70, 0x65})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex.go:205 +0x124
created by github.com/pb33f/libopenapi/index.(*Rolodex).IndexTheRolodex.func1 in goroutine 16
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex.go:233 +0x27c

The files in the error are in no way referenced or associated with the file I am trying to lint. The file I am linting is a random JSON schema file with no $refs, etc. Its not an OpenAPI file (thus the -k flag).

If i change directories to packages/graphql and run the same command with the paths adjusted:

vacuum lint -k -s -d -e -b -r ../runtime/config/policy-ruleset.yaml ./src/policies/graphql-complexity-limit-inbound/options.json

The operations successfully lints the file.

daveshanley commented 6 months ago

Hmm, this is odd, the second outcome is the desired one.

Is the only difference the working directory you're calling this from? i.e. packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json vs ./src/policies/graphql-complexity-limit-inbound/options.json

Also could you add the -w flag, it will spit out debug messages.

daveshanley commented 6 months ago

Is there any chance I could get access to a trimmed down zip file so I can reproduce the issue locally?

ntotten commented 6 months ago

Running it with the -w flag. Oddly, it is not consistent. I ran it a few times and it worked once and failed two other times, but seemingly in slightly different ways (although i am no expert in go).

vacuum lint -k -s -d -e -b -w -r ./packages/runtime/config/policy-ruleset.yaml ./packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json 
 INFO  Linting file './packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json' against 2 rules: 

panic: runtime error: index out of range [1] with length 1

goroutine 112 [running]:
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x140015faf00, 0x14002647180, 0x14002646f00, {0x14004bbca40?, 0x1011ef9ee?, 0x2?}, 0xffffffffffffffff?, 0x20?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:185 +0x3464
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x140015faf00, 0x14002646f00, 0x14002646dc0, {0x14004bbca40?, 0x1014fab40?, 0x2?}, 0xffffffffffffffff?, 0x65?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x140015faf00, 0x14002646dc0, 0x14001d75900, {0x14004329ea0?, 0x1011ef9ee?, 0x2?}, 0xffffffffffffffff?, 0xfa?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x140015faf00, 0x14001d75900, 0x14001d5e280, {0x14004b8a210?, 0x1011ef9ee?, 0x2?}, 0xffffffffffffffff?, 0x0?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x140015faf00, 0x14001d5e280, 0x14001d5e140, {0x14001f49db8?, 0x14953ec78?, 0x102769d28?}, 0x30?, 0x0?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.createNewIndex(0x14001d5e140, 0x140015faf00, 0x1)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/spec_index.go:80 +0x158
github.com/pb33f/libopenapi/index.NewSpecIndexWithConfig(0x14001d5e140, 0x14000300480)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/spec_index.go:51 +0x2d8
github.com/pb33f/libopenapi/index.(*LocalFile).Index(0x14001fb4500, 0xaebde78d89e5a894?)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex_file_loader.go:191 +0x50
github.com/pb33f/libopenapi/index.(*Rolodex).IndexTheRolodex.func1.1({0x1494fe800, 0x14001fb4500}, {0x1400046b730, 0x66})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex.go:205 +0x124
created by github.com/pb33f/libopenapi/index.(*Rolodex).IndexTheRolodex.func1 in goroutine 25
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex.go:233 +0x27c

core on ξ‚  main [$] via 🐳 desktop-linux via  v18.18.2 on ☁️  (us-east-2) on ☁️  nate@zuplo.com(us-central1) 
❯ vacuum lint -k -s -d -e -b -w -r packages/runtime/config/policy-ruleset.yaml packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json 
 INFO  Linting file 'packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json' against 2 rules: 

panic: runtime error: index out of range [5] with length 5

goroutine 1798 [running]:
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14004323400, 0x14004cfa6e0, 0x14004cd8500, {0x14004b90550?, 0x10321f9ee?, 0x2?}, 0xffffffffffffffff?, 0x0?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:185 +0x3464
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14004323400, 0x14004cd8500, 0x14004cd83c0, {0x14004905db8?, 0x12b53f6c8?, 0x1047993c8?}, 0x30?, 0x0?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.createNewIndex(0x14004cd83c0, 0x14004323400, 0x1)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/spec_index.go:80 +0x158
github.com/pb33f/libopenapi/index.NewSpecIndexWithConfig(0x14004cd83c0, 0x14004368700)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/spec_index.go:51 +0x2d8
github.com/pb33f/libopenapi/index.(*LocalFile).Index(0x1400030e0a0, 0xe6858ce58db8e48b?)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex_file_loader.go:191 +0x50
github.com/pb33f/libopenapi/index.(*Rolodex).IndexTheRolodex.func1.1({0x12b4ba0f8, 0x1400030e0a0}, {0x1400220ca20, 0x51})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex.go:205 +0x124
created by github.com/pb33f/libopenapi/index.(*Rolodex).IndexTheRolodex.func1 in goroutine 20
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex.go:233 +0x27c

core on ξ‚  main [$] via 🐳 desktop-linux via  v18.18.2 on ☁️  (us-east-2) on ☁️  nate@zuplo.com(us-central1) 
❯ vacuum lint -k -s -d -e -b -w -r packages/runtime/config/policy-ruleset.yaml packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json 
 INFO  Linting file 'packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json' against 2 rules: 

          Linting passed, A perfect score! well done!          

core on ξ‚  main [$] via 🐳 desktop-linux via  v18.18.2 on ☁️  (us-east-2) on ☁️  nate@zuplo.com(us-central1) 
❯ vacuum lint -k -s -d -e -b -w -r packages/runtime/config/policy-ruleset.yaml packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json 
 INFO  Linting file 'packages/graphql/src/policies/graphql-complexity-limit-inbound/options.json' against 2 rules: 

panic: runtime error: index out of range [1] with length 1

goroutine 191 [running]:
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14003adc500, 0x14003a879a0, 0x14003a87860, {0x14003af82c0?, 0x1031cab40?, 0x2?}, 0xffffffffffffffff?, 0x0?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:185 +0x3464
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14003adc500, 0x14003a87860, 0x14003a87720, {0x14003a81c00?, 0x1031cab40?, 0x2?}, 0xffffffffffffffff?, 0xe4?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14003adc500, 0x14003a87720, 0x14003a86640, {0x14003ae8830?, 0x102ebf9ee?, 0x2?}, 0xffffffffffffffff?, 0x0?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0x14003adc500, 0x14003a86640, 0x14003a86500, {0x14003675db8?, 0x1047d1460?, 0x104439d28?}, 0x30?, 0x80?, {0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/extract_refs.go:40 +0x238
github.com/pb33f/libopenapi/index.createNewIndex(0x14003a86500, 0x14003adc500, 0x1)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/spec_index.go:80 +0x158
github.com/pb33f/libopenapi/index.NewSpecIndexWithConfig(0x14003a86500, 0x14003ad8080)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/spec_index.go:51 +0x2d8
github.com/pb33f/libopenapi/index.(*LocalFile).Index(0x1400049d180, 0x20200a5d22656772?)
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex_file_loader.go:191 +0x50
github.com/pb33f/libopenapi/index.(*Rolodex).IndexTheRolodex.func1.1({0x12b2d5328, 0x1400049d180}, {0x140000451f0, 0x67})
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex.go:205 +0x124
created by github.com/pb33f/libopenapi/index.(*Rolodex).IndexTheRolodex.func1 in goroutine 19
        /home/runner/go/pkg/mod/github.com/pb33f/libopenapi@v0.14.0/index/rolodex.go:233 +0x27c
ntotten commented 6 months ago

Here is a minimal repro. It does seem to be the folder context is the key. One thing though in my repro I couldn't get it to actually panic - but it does log some errors. It might be that we have A LOT of various OpenAPI schemas in that repo, many of them are used for tests and many of them are invalid for the purposes of testing so if it is loading all the json/openapi files, its probably loading a lot of stuff.

https://github.com/ntotten/vacuum_400_repro

daveshanley commented 6 months ago

This is very helpful, thank you! I am looking into this as a priority.

daveshanley commented 6 months ago

I have found the problem, it's a bug for sure, it's only triggered by the -k flag. Fixes are inbound.

daveshanley commented 6 months ago

Can you try v0.6.2?

ntotten commented 6 months ago

Fixed. THanks!