search

IBM / openapi-validator

Configurable and extensible validator/linter for OpenAPI documents
Apache License 2.0
489 stars 88 forks source link

"Error: Provided ruleset is not an object" after upgrading to ibm-openapi-validator@1.24.0 #687

Closed chadxz closed 1 day ago

chadxz commented 2 days ago

When upgrading from ibm-openapi-validator 1.23.0 to 1.24.0 we started getting this error:

[ERROR] There was a problem with spectral.
[ERROR] Provided ruleset is not an object
[ERROR] Additional error details:
[ERROR] RulesetValidationError: Provided ruleset is not an object
    at assertValidRuleset (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/node_modules/@stoplight/spectral-core/dist/ruleset/validation/assertions.js:12:15)
    at new Ruleset (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/node_modules/@stoplight/spectral-core/dist/ruleset/ruleset.js:35:44)
    at Spectral.setRuleset (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/node_modules/@stoplight/spectral-core/dist/spectral.js:66:73)
    at setup (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/src/spectral/spectral-validator.js:158:12)
    at async runSpectral (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/src/spectral/spectral-validator.js:34:20)
    at async runValidator (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/src/cli-validator/run-validator.js:219:17) {
  code: 'invalid-ruleset-definition',
  path: []
}

.spectral.yml:

extends: "@ibm-cloud/openapi-ruleset"
rules:
  ibm-path-segment-casing-convention: off
  ibm-parameter-casing-convention: off
  ibm-property-casing-convention: off
  ibm-enum-casing-convention: off
  ibm-no-array-responses: off
  no-$ref-siblings: off
overrides:
  - files:
      - "backend/openapi/vendor.json#/paths/~1vendor~1customer-relationship~1%7BcustomerRelationshipId%7D~1custom-attributes/patch"
    rules:
      # this route returns arbitrary key/value pairs, so we can't return an explicit model
      ibm-accept-and-return-models: off

The latest (1.24.1) also has this issue.

CC: @padamstx, looks like this was introduced in https://github.com/IBM/openapi-validator/pull/684

dpopp07 commented 2 days ago

Hmm that's very strange. Nothing about that commit should cause such an error. I'll attempt to reproduce. Thanks for reporting it

dpopp07 commented 2 days ago

I am not able to reproduce. What version of the ruleset do you have installed?

You can run lint-openapi --version and if your Spectral config file is read, it will tell you both what version you're using through that and what version would be used by default with your validator installation.

chadxz commented 2 days ago 
❯ lint-openapi --version                     
validator: 1.24.1; ruleset: 1.23.1
dpopp07 commented 2 days ago

Unfortunately, I'm still not able to reproduce - even with the same versions, set up, ruleset, etc.

Is it possible anything else has changed about your setup since upgrading? If you return to v1.23.0 without changing anything else, does the error go away?

Will you also try turning on debug logging? Add -l debug to your validation command.

chadxz commented 2 days ago

If you return to v1.23.0 without changing anything else, does the error go away?

yep, it goes away

Will you also try turning on debug logging?

❯ lint-openapi -e backend/openapi/vendor.json -l debug
[DEBUG] Using validator configuration:
{
  "colorizeOutput": true,
  "errorsOnly": true,
  "files": [
    "backend/openapi/vendor.json"
  ],
  "limits": {
    "warnings": -1
  },
  "ignoreFiles": [],
  "logLevels": {
    "root": "debug"
  },
  "outputFormat": "text",
  "ruleset": null,
  "summaryOnly": false
}
IBM OpenAPI Validator (validator: 1.24.1), @Copyright IBM Corporation 2017, 2024.

Validation Results for backend/openapi/vendor.json:

[DEBUG] Using Spectral ruleset file: /Users/chad/src/smartrr/.spectral.yml
[DEBUG] Custom ruleset file found in: /Users/chad/src/smartrr
[DEBUG] Found IBM ruleset package file at: /Users/chad/src/smartrr/node_modules/@ibm-cloud/openapi-ruleset/package.json
[ERROR] There was a problem with spectral.
[ERROR] Provided ruleset is not an object
[ERROR] Additional error details:
[ERROR] RulesetValidationError: Provided ruleset is not an object
    at assertValidRuleset (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/node_modules/@stoplight/spectral-core/dist/ruleset/validation/assertions.js:12:15)
    at new Ruleset (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/node_modules/@stoplight/spectral-core/dist/ruleset/ruleset.js:35:44)
    at Spectral.setRuleset (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/node_modules/@stoplight/spectral-core/dist/spectral.js:66:73)
    at setup (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/src/spectral/spectral-validator.js:158:12)
    at async runSpectral (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/src/spectral/spectral-validator.js:34:20)
    at async runValidator (/Users/chad/src/smartrr/node_modules/ibm-openapi-validator/src/cli-validator/run-validator.js:219:17) {
  code: 'invalid-ruleset-definition',
  path: []
}
dpopp07 commented 1 day ago

Okay, I'm perplexed 🙂

I can keep asking questions to try and dig into why this might be happening but we've covered most angles... The nuclear way to see exactly what's happening here would be to open /Users/chad/src/smartrr/node_modules/ibm-openapi-validator/node_modules/@stoplight/spectral-core/dist/ruleset/validation/assertions.js:12 and throw a console.log in there to see what the ruleset variable looks like. Then we'd know exactly what "non-object" is being given to Spectral and causing the error.

chadxz commented 1 day ago

The resulting log file is 128M of text.

here's the beginning of it.

❯ head -n100 lint-openapi.log
[DEBUG] Using validator configuration:
{
  "colorizeOutput": true,
  "errorsOnly": true,
  "files": [
    "backend/openapi/vendor.json"
  ],
  "limits": {
    "warnings": -1
  },
  "ignoreFiles": [],
  "logLevels": {
    "root": "debug"
  },
  "outputFormat": "text",
  "ruleset": null,
  "summaryOnly": false
}
IBM OpenAPI Validator (validator: 1.24.1), @Copyright IBM Corporation 2017, 2024.

Validation Results for backend/openapi/vendor.json:

[DEBUG] Using Spectral ruleset file: /Users/chad/src/smartrr/.spectral.yml
[DEBUG] Custom ruleset file found in: /Users/chad/src/smartrr
[DEBUG] Found IBM ruleset package file at: /Users/chad/src/smartrr/node_modules/@ibm-cloud/openapi-ruleset/package.json
Ruleset {
  maybeDefinition: {
    extends: {
      extends: {
        documentationUrl: 'https://meta.stoplight.io/docs/spectral/docs/reference/openapi-rules.md',
        formats: [
          [Function: oas2] { displayName: 'OpenAPI 2.0 (Swagger)' },
          [Function: isOas3] { displayName: 'OpenAPI 3.x' },
          [Function: oas3_0] { displayName: 'OpenAPI 3.0.x' },
          [Function: oas3_1] { displayName: 'OpenAPI 3.1.x' }
        ],
        aliases: {
          PathItem: [ '$.paths[*]' ],
          OperationObject: [ '#PathItem[get,put,post,delete,options,head,patch,trace]' ],
          SecurityRequirementObject: [ '$.security[*]', '#OperationObject.security[*]' ],
          ResponseObject: {
            targets: [
              {
                formats: [
                  [Function: oas2] {
                    displayName: 'OpenAPI 2.0 (Swagger)'
                  }
                ],
                given: [ '#OperationObject.responses[*]', '$.responses[*]' ]
              },
              {
                formats: [ [Function: isOas3] { displayName: 'OpenAPI 3.x' } ],
                given: [
                  '#OperationObject.responses[*]',
                  '$.components.responses[*]'
                ]
              }
            ]
          },
          LinkObject: {
            targets: [
              {
                formats: [ [Function: isOas3] { displayName: 'OpenAPI 3.x' } ],
                given: [ '$.components.links[*]', '#ResponseObject.links[*]' ]
              }
            ]
          },
          ArrayProperties: {
            targets: [
              {
                formats: [
                  [Function: oas2] {
                    displayName: 'OpenAPI 2.0 (Swagger)'
                  },
                  [Function: oas3_0] { displayName: 'OpenAPI 3.0.x' }
                ],
                given: [ '$..[?(@ && @.type=="array")]' ]
              },
              {
                formats: [ [Function: oas3_1] { displayName: 'OpenAPI 3.1.x' } ],
                given: [
                  '$..[?(@ && @.type=="array")]',
                  '$..[?(@ && @.type && @.type.constructor.name === "Array" && @.type.includes("array"))]'
                ]
              }
            ]
          }
        },
        rules: {
          'operation-success-response': {
            description: 'Operation must have at least one "2xx" or "3xx" response.',
            recommended: true,
            given: '#OperationObject',
            then: {
              field: 'responses',
              function: [Function: oasOpSuccessResponse] {
                validator: [Function (anonymous)]
              }
            }
          },
chadxz commented 1 day ago

Perhaps one of these deps upgrades is the culprit

Screenshot 2024-10-01 at 2 58 10 PM
dpopp07 commented 1 day ago

Okay - it seems your code is getting an instance of the Ruleset class, rather than a plain object, which it seems to expect. When I use the same log, I see

...
{
  extends: [],
  rules: {
    'ibm-path-segment-casing-convention': 'off',
    'ibm-parameter-casing-convention': 'off',
    'ibm-property-casing-convention': 'off',
    'ibm-enum-casing-convention': 'off',
    'ibm-no-array-responses': 'off',
    'no-$ref-siblings': 'off',
    'ibm-string-attributes': 'off'
  },
  overrides: [ { files: [Array], rules: [Object] } ]
}
{
  extends: [],
  documentationUrl: 'https://github.com/IBM/openapi-validator/blob/main/docs/ibm-cloud-rules.md',
  formats: [ [Function: isOas3] { displayName: 'OpenAPI 3.x' } ],
  rules: {
    'operation-success-response': 'off',
    'oas2-operation-formData-consume-check': true,
    'operation-operationId-unique': true,
...

Perhaps one of these deps upgrades is the culprit

Yep, I was about to say that this makes me think the underlying Spectral behavior has changed in some subtle way.

Will you run:

npm list @stoplight/spectral-core
npm list @stoplight/spectral-cli

and see what versions you have installed?

chadxz commented 1 day ago 
❯ yarn why @stoplight/spectral-core 
├─ @stoplight/spectral-cli@npm:6.13.1
│  └─ @stoplight/spectral-core@npm:1.18.3 (via npm:^1.18.3)
│
├─ @stoplight/spectral-formats@npm:1.6.0
│  └─ @stoplight/spectral-core@npm:1.18.3 (via npm:^1.8.0)
│
├─ @stoplight/spectral-formats@npm:1.7.0
│  └─ @stoplight/spectral-core@npm:1.18.3 (via npm:^1.8.0)
│
├─ @stoplight/spectral-formatters@npm:1.3.0
│  └─ @stoplight/spectral-core@npm:1.18.3 (via npm:^1.15.1)
│
├─ @stoplight/spectral-functions@npm:1.8.0
│  └─ @stoplight/spectral-core@npm:1.18.3 (via npm:^1.7.0)
│
├─ @stoplight/spectral-functions@npm:1.9.0
│  └─ @stoplight/spectral-core@npm:1.18.3 (via npm:^1.7.0)
│
├─ @stoplight/spectral-ruleset-bundler@npm:1.6.0
│  └─ @stoplight/spectral-core@npm:1.18.3 (via npm:>=1)
│
├─ @stoplight/spectral-rulesets@npm:1.19.1
│  └─ @stoplight/spectral-core@npm:1.18.3 (via npm:^1.8.1)
│
├─ @stoplight/spectral-rulesets@npm:1.20.2
│  └─ @stoplight/spectral-core@npm:1.18.3 (via npm:^1.8.1)
│
└─ ibm-openapi-validator@npm:1.24.1
   └─ @stoplight/spectral-core@npm:1.19.1 (via npm:^1.19.1)

❯ yarn why @stoplight/spectral-cli
└─ ibm-openapi-validator@npm:1.24.1
   └─ @stoplight/spectral-cli@npm:6.13.1 (via npm:^6.13.1)
chadxz commented 1 day ago

if I pin spectral-core using "@stoplight/spectral-core": "1.18.3" in resolutions, things work.

dpopp07 commented 1 day ago

I'm not familiar with yarn. Do the dependencies get de-duped? Or is it possible you have 1.18.3 and 1.19.1 of spectral-core both in use?

I have only 1.19.1 in use in my working sandbox.

dpopp07 commented 1 day ago

if I pin spectral-core using "@stoplight/spectral-core": "1.18.3" in resolutions, things work.

I imagine the same thing might work with 1.19.1. Perhaps the issue is caused by a discrepancy somewhere.

chadxz commented 1 day ago

yeah pinning to 1.19.1 works. Here's the resulting diff

diff --git a/.yarn/cache/@stoplight-spectral-core-npm-1.18.3-34e2ddeefd-18edf248da.zip b/.yarn/cache/@stoplight-spectral-core-npm-1.18.3-34e2ddeefd-18edf248da.zip
deleted file mode 100644
index 4b16ce2442..0000000000
Binary files a/.yarn/cache/@stoplight-spectral-core-npm-1.18.3-34e2ddeefd-18edf248da.zip and /dev/null differ
diff --git a/package.json b/package.json
index 1206fe6a11..1f34ae4c1c 100644
--- a/package.json
+++ b/package.json
@@ -72,7 +72,8 @@
     "print-config:production": "scripts/print-config.sh production"
   },
   "resolutions": {
-    "typeorm": "0.2.40"
+    "typeorm": "0.2.40",
+    "@stoplight/spectral-core": "1.19.1"
   },
   "dependencies": {
     "@types/terminal-kit": "^2.5.6",
diff --git a/yarn.lock b/yarn.lock
index 710d88d439..f42b933eae 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -11203,36 +11203,7 @@ __metadata:
   languageName: node
   linkType: hard

-"@stoplight/spectral-core@npm:>=1, @stoplight/spectral-core@npm:^1.15.1, @stoplight/spectral-core@npm:^1.18.3, @stoplight/spectral-core@npm:^1.7.0, @stoplight/spectral-core@npm:^1.8.0, @stoplight/spectral-core@npm:^1.8.1":
-  version: 1.18.3
-  resolution: "@stoplight/spectral-core@npm:1.18.3"
-  dependencies:
-    "@stoplight/better-ajv-errors": "npm:1.0.3"
-    "@stoplight/json": "npm:~3.21.0"
-    "@stoplight/path": "npm:1.3.2"
-    "@stoplight/spectral-parsers": "npm:^1.0.0"
-    "@stoplight/spectral-ref-resolver": "npm:^1.0.0"
-    "@stoplight/spectral-runtime": "npm:^1.0.0"
-    "@stoplight/types": "npm:~13.6.0"
-    "@types/es-aggregate-error": "npm:^1.0.2"
-    "@types/json-schema": "npm:^7.0.11"
-    ajv: "npm:^8.6.0"
-    ajv-errors: "npm:~3.0.0"
-    ajv-formats: "npm:~2.1.0"
-    es-aggregate-error: "npm:^1.0.7"
-    jsonpath-plus: "npm:7.1.0"
-    lodash: "npm:~4.17.21"
-    lodash.topath: "npm:^4.5.2"
-    minimatch: "npm:3.1.2"
-    nimma: "npm:0.2.2"
-    pony-cause: "npm:^1.0.0"
-    simple-eval: "npm:1.0.0"
-    tslib: "npm:^2.3.0"
-  checksum: 10/18edf248dabae0627d20a8bafe57c69a05417891f7000b172fbceca089279e84ebbd05052b57c610fdf6de9ce02883bf45e55168e474d8b39635d7656159d261
-  languageName: node
-  linkType: hard
-
-"@stoplight/spectral-core@npm:^1.19.1":
+"@stoplight/spectral-core@npm:1.19.1":
   version: 1.19.1
   resolution: "@stoplight/spectral-core@npm:1.19.1"
   dependencies:
@@ -11354,7 +11325,7 @@ __metadata:
   languageName: node
   linkType: hard

-"@stoplight/spectral-ref-resolver@npm:^1.0.0, @stoplight/spectral-ref-resolver@npm:^1.0.4":
+"@stoplight/spectral-ref-resolver@npm:^1.0.4":
   version: 1.0.4
   resolution: "@stoplight/spectral-ref-resolver@npm:1.0.4"
   dependencies:
@@ -14532,7 +14503,7 @@ __metadata:
   languageName: node
   linkType: hard

-"ajv@npm:^8.12.0, ajv@npm:^8.6.0, ajv@npm:^8.6.3, ajv@npm:^8.9.0":
+"ajv@npm:^8.12.0, ajv@npm:^8.6.3, ajv@npm:^8.9.0":
   version: 8.16.0
   resolution: "ajv@npm:8.16.0"
   dependencies:
chadxz commented 1 day ago

I'll use the npm commands you suggested.

Here's the list output before pinning:

❯ npm list @stoplight/spectral-core
smartrr@ /Users/chad/src/smartrr
└─┬ ibm-openapi-validator@1.24.1
  ├─┬ @ibm-cloud/openapi-ruleset@1.23.1
  │ ├─┬ @stoplight/spectral-formats@1.7.0
  │ │ └── @stoplight/spectral-core@1.18.3 deduped
  │ ├─┬ @stoplight/spectral-functions@1.9.0
  │ │ └── @stoplight/spectral-core@1.18.3 deduped
  │ └─┬ @stoplight/spectral-rulesets@1.20.2
  │   ├── @stoplight/spectral-core@1.18.3 deduped
  │   └─┬ @stoplight/spectral-functions@1.8.0
  │     ├── @stoplight/spectral-core@1.18.3 deduped
  │     └─┬ @stoplight/spectral-formats@1.6.0
  │       └── @stoplight/spectral-core@1.18.3 deduped
  ├─┬ @stoplight/spectral-cli@6.13.1
  │ ├── @stoplight/spectral-core@1.18.3
  │ ├─┬ @stoplight/spectral-formatters@1.3.0
  │ │ └── @stoplight/spectral-core@1.18.3 deduped
  │ ├─┬ @stoplight/spectral-ruleset-bundler@1.6.0
  │ │ ├── @stoplight/spectral-core@1.18.3 deduped
  │ │ └─┬ @stoplight/spectral-functions@1.8.0
  │ │   ├── @stoplight/spectral-core@1.18.3 deduped
  │ │   └─┬ @stoplight/spectral-formats@1.6.0
  │ │     └── @stoplight/spectral-core@1.18.3 deduped
  │ └─┬ @stoplight/spectral-rulesets@1.19.1
  │   ├── @stoplight/spectral-core@1.18.3 deduped
  │   └─┬ @stoplight/spectral-formats@1.6.0
  │     └── @stoplight/spectral-core@1.18.3 deduped
  └── @stoplight/spectral-core@1.19.1

and after pinning:

❯ npm list @stoplight/spectral-core
smartrr@ /Users/chad/src/smartrr
└─┬ ibm-openapi-validator@1.24.1
  ├─┬ @ibm-cloud/openapi-ruleset@1.23.1
  │ ├─┬ @stoplight/spectral-formats@1.7.0
  │ │ └── @stoplight/spectral-core@1.19.1 deduped
  │ ├─┬ @stoplight/spectral-functions@1.9.0
  │ │ └── @stoplight/spectral-core@1.19.1 deduped
  │ └─┬ @stoplight/spectral-rulesets@1.20.2
  │   ├── @stoplight/spectral-core@1.19.1 deduped
  │   └─┬ @stoplight/spectral-functions@1.8.0
  │     ├── @stoplight/spectral-core@1.19.1 deduped
  │     └─┬ @stoplight/spectral-formats@1.6.0
  │       └── @stoplight/spectral-core@1.19.1 deduped
  ├─┬ @stoplight/spectral-cli@6.13.1
  │ ├── @stoplight/spectral-core@1.19.1 deduped
  │ ├─┬ @stoplight/spectral-formatters@1.3.0
  │ │ └── @stoplight/spectral-core@1.19.1 deduped
  │ ├─┬ @stoplight/spectral-ruleset-bundler@1.6.0
  │ │ ├── @stoplight/spectral-core@1.19.1 deduped
  │ │ └─┬ @stoplight/spectral-functions@1.8.0
  │ │   ├── @stoplight/spectral-core@1.19.1 deduped
  │ │   └─┬ @stoplight/spectral-formats@1.6.0
  │ │     └── @stoplight/spectral-core@1.19.1 deduped
  │ └─┬ @stoplight/spectral-rulesets@1.19.1
  │   ├── @stoplight/spectral-core@1.19.1 deduped
  │   └─┬ @stoplight/spectral-formats@1.6.0
  │     └── @stoplight/spectral-core@1.19.1 deduped
  └── @stoplight/spectral-core@1.19.1
chadxz commented 1 day ago

Completely removing the package and then adding it back results in the packages all having the same spectral-core version. I guess this may be some idiosyncrasy with how yarn manages its lockfile.

You can close if you're over it :) Thanks for your help!

dpopp07 commented 1 day ago

I guess this may be some idiosyncrasy with how yarn manages its lockfile.

Seems like that may be the case - how strange! Good to know about, glad you've got it sorted now.

You can close if you're over it :) Thanks for your help!

Will do, no problem!