"Invalid ref" error when generating server with cross-file reference when "--keep-spec-order" specified

[mdelvecchio-wasabi]

Problem statement

When I generate a server from a complete YAML file, I am able to validate the document and generate the server.

When I move an object definition to a separate YAML file in the same directory, and use a cross-file reference to the definition, validation passes. But server generation fails when the "--keep-spec-order" option is specified. It succeeds when it is not.

Please remove the sections that don't apply

Swagger specification

First, the complete document without external references, which works:

swagger: '2.0'
info:
  title: Test API
  description: Test
  version: v1
host: host
schemes:
- https
- http
basePath: /path
produces:
- application/json

paths:
  /obj:
    get:
      summary: Get an object
      operationId: getObject
      responses:
        200:
          description: The object
          schema:
            $ref: '#/definitions/MyObject'

definitions:
  MyObject:
    title: My object
    type: object
    properties:
      string1:
        type: string
        title: A string
      bool1:
        type: boolean
        title: A boolean
    default:
      string1: "first"
      bool1:   false

Next, the document split into two, starting with the main document with an external reference:

swagger: '2.0'
info:
  title: Test API
  description: Test
  version: v1
host: host
schemes:
- https
- http
basePath: /path
produces:
- application/json

paths:
  /obj:
    get:
      summary: Get an object
      operationId: getObject
      responses:
        200:
          description: The object
          schema:
            $ref: './test-definitions.yaml#/definitions/MyObject'

And next, the test-definitions.yaml file, which is located in the same directory:

definitions:
  MyObject:
    title: My object
    type: object
    properties:
      string1:
        type: string
        title: A string
      bool1:
        type: boolean
        title: A boolean
    default:
      string1: "first"
      bool1:   false

Steps to reproduce

Validation succeeds in both cases:

$ ./go-swagger validate test.yaml
2020/02/06 08:43:13 
The swagger spec at "test.yaml" is valid against swagger specification 2.0

And basic server generation also succeeds in both cases:

$ ./go-swagger generate server -f test.yaml -t ./testout
...
2020/02/06 08:47:11 Generation completed!

But when I add the --keep-spec-order flag, it fails on the external reference:

$ ./go-swagger generate server -f test.yaml -t ./testout --keep-spec-order
2020/02/06 08:48:21 validating spec /tmp/test.yaml940091833
The swagger spec at "/tmp/test.yaml940091833" is invalid against swagger specification 2.0. see errors :
- invalid ref "./test-definitions.yaml#/definitions/MyObject"

The error message says the swagger spec is in the /tmp directory, so I assume that the relative reference is not being resolved properly to the current directory. If I copy test-definitions.yaml to /tmp, generation succeeds.

Environment

swagger version: v0.21.0 go version: go1.11.5 linux/amd64 OS: Ubuntu Linux 16

[mdelvecchio-wasabi]

The problem seems to be in WithAutoXOrder() in generator/support.go. It specifies "" as the directory parameter to ioutil.TempFile():

    tmpFile, err := ioutil.TempFile("", filepath.Base(specPath))

This places the temporary file in /tmp, but the rest of the code doesn't know that relative references should not refer to /tmp, but instead to the directory where the original file is located.

[mdelvecchio-wasabi]

Just ran into this again on another project. This is very frustrating, as it prevents me from using Go's db.Scan() to scan a SQL row into a generated struct, because the property order matters for db.Scan().

This also makes it impossible to share object definitions across multiple swaggers, which is what I am trying to do now.

[fredbi]

The --keep-spec-order hack looks like it has been hastily implemented to solve a very narrow use-case. As an aside, for SQL, I am using github.com/jpmoiron/sqlx.StructScan, with named columns. Much less error-prone.

[mdelvecchio-wasabi]

I have abandoned --keep-spec-order.

I now use reflection to determine the property order of the generated model, then select the columns in that order. That works for me.

I think it's too late for us to switch to a different DB scanning technology; we have 4+ years built on Go's library. But it would be nice to find something faster.

Thanks.

[mattalexander-pantheon]

Looks like this problem is relatively well known. I suppose I'll leave my recreation steps, since it's in my clipboard already:

cd "$(mktemp -d)"
go mod init swagtest
wget https://raw.githubusercontent.com/openservicebrokerapi/servicebroker/v2.16/swagger.yaml
swagger generate model --keep-spec-order

Output:

go: creating new go.mod: module swagtest
--2021-07-09 01:11:27--  https://raw.githubusercontent.com/openservicebrokerapi/servicebroker/v2.16/swagger.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22231 (22K) [text/plain]
Saving to: ‘swagger.yaml’

swagger.yaml                                                100%[===========================================================================================================================================>]  21.71K  --.-KB/s    in 0.002s

2021-07-09 01:11:28 (8.56 MB/s) - ‘swagger.yaml’ saved [22231/22231]

2021/07/09 01:11:28 validating spec /var/folders/59/qq76hmt55wb642f39q6nbxqr0000gp/T/swagger.yaml946778639
The swagger spec at "/var/folders/59/qq76hmt55wb642f39q6nbxqr0000gp/T/swagger.yaml946778639" is invalid against swagger specification 2.0. see errors :
- some references could not be resolved in spec. First found: invalid character 's' looking for beginning of value

[fredbi]

Fixed by #2621