Support JSON references to other files?

[kindaro]

see for yourself

Apply the following diff:

diff --git a/specifications/petstore.yaml b/specifications/petstore.yaml
index c896cf7..99081d5 100644
--- a/specifications/petstore.yaml
+++ b/specifications/petstore.yaml
@@ -588,16 +588,6 @@ components:
           description: User Status
       xml:
         name: User
-    Category:
-      type: object
-      properties:
-        id:
-          type: integer
-          format: int64
-        name:
-          type: string
-      xml:
-        name: Category
     Tag:
       type: object
       properties:
@@ -628,7 +618,7 @@ components:
           type: integer
           format: int64
         category:
-          "$ref": "#/components/schemas/Category"
+          "$ref": "petstore-Category.yaml"
         name:
           type: string
           example: doggie

Create the following file:

specifications/petstore-Category.yaml

type: object
properties:
  id:
    type: integer
    format: int64
  name:
    type: string
xml:
  name: Category

Run openapi3-code-generator-exe petstore.yaml.

what I see

INFO  (paths./pet.PUT): Generating operation with name 'updatePet'
INFO  (paths./pet.POST): Generating operation with name 'addPet'
INFO  (paths./pet/findByStatus.GET): Generating operation with name 'findPetsByStatus'
INFO  (paths./pet/findByStatus.GET.parameters.[0].schema.items): Define as enum named 'FindPetsByStatusParametersStatus'
INFO  (paths./pet/findByTags.GET): Generating operation with name 'findPetsByTags'
INFO  (paths./pet/{petId}.GET): Generating operation with name 'getPetById'
INFO  (paths./pet/{petId}.POST): Generating operation with name 'updatePetWithForm'
INFO  (paths./pet/{petId}.POST.requestBody.content.application/x-www-form-urlencoded.schema): Define as record named 'UpdatePetWithFormRequestBody'
INFO  (paths./pet/{petId}.DELETE): Generating operation with name 'deletePet'
WARN  (paths./pet/{petId}.DELETE.parameters): Parameters are only supported in query and path (skipping parameters in cookie and header).
INFO  (paths./pet/{petId}/uploadImage.POST): Generating operation with name 'uploadFile'
WARN  (paths./pet/{petId}/uploadImage.POST.requestBody.content): Only content type application/json and application/x-www-form-urlencoded is supported
INFO  (paths./store/inventory.GET): Generating operation with name 'getInventory'
INFO  (paths./store/order.POST): Generating operation with name 'placeOrder'
INFO  (paths./store/order/{orderId}.GET): Generating operation with name 'getOrderById'
INFO  (paths./store/order/{orderId}.DELETE): Generating operation with name 'deleteOrder'
INFO  (paths./user.POST): Generating operation with name 'createUser'
INFO  (paths./user/createWithArray.POST): Generating operation with name 'createUsersWithArrayInput'
INFO  (paths./user/createWithList.POST): Generating operation with name 'createUsersWithListInput'
INFO  (paths./user/login.GET): Generating operation with name 'loginUser'
INFO  (paths./user/login.GET.parameters): Define as record named 'LoginUserParameters'
INFO  (paths./user/logout.GET): Generating operation with name 'logoutUser'
INFO  (paths./user/{username}.GET): Generating operation with name 'getUserByName'
INFO  (paths./user/{username}.PUT): Generating operation with name 'updateUser'
INFO  (paths./user/{username}.DELETE): Generating operation with name 'deleteUser'
INFO  (components.schemas.ApiResponse): Define as record named 'ApiResponse'
INFO  (components.schemas.Order): Define as record named 'Order'
INFO  (components.schemas.Order.properties.status): Define as enum named 'OrderStatus'
INFO  (components.schemas.Pet): Define as record named 'Pet'
INFO  (components.schemas.Pet.properties.status): Define as enum named 'PetStatus'
INFO  (components.schemas.Tag): Define as record named 'Tag'
INFO  (components.schemas.User): Define as record named 'User'
WARN  (components.securitySchemes.petstore_auth): The security scheme 'petstore_auth' is not supported (currently only http-basic and http-bearer are supported).
Output directory will be created
Remove old output directory
Output directory removed, create missing directories
Directories created
Write file to path: out/src/OpenAPI.hs
Write file to path: out/src/OpenAPI/Operations/UpdatePet.hs
Write file to path: out/src/OpenAPI/Operations/AddPet.hs
Write file to path: out/src/OpenAPI/Operations/FindPetsByStatus.hs
Write file to path: out/src/OpenAPI/Operations/FindPetsByTags.hs
Write file to path: out/src/OpenAPI/Operations/GetPetById.hs
Write file to path: out/src/OpenAPI/Operations/UpdatePetWithForm.hs
Write file to path: out/src/OpenAPI/Operations/DeletePet.hs
Write file to path: out/src/OpenAPI/Operations/UploadFile.hs
Write file to path: out/src/OpenAPI/Operations/GetInventory.hs
Write file to path: out/src/OpenAPI/Operations/PlaceOrder.hs
Write file to path: out/src/OpenAPI/Operations/GetOrderById.hs
Write file to path: out/src/OpenAPI/Operations/DeleteOrder.hs
Write file to path: out/src/OpenAPI/Operations/CreateUser.hs
Write file to path: out/src/OpenAPI/Operations/CreateUsersWithArrayInput.hs
Write file to path: out/src/OpenAPI/Operations/CreateUsersWithListInput.hs
Write file to path: out/src/OpenAPI/Operations/LoginUser.hs
Write file to path: out/src/OpenAPI/Operations/LogoutUser.hs
Write file to path: out/src/OpenAPI/Operations/GetUserByName.hs
Write file to path: out/src/OpenAPI/Operations/UpdateUser.hs
Write file to path: out/src/OpenAPI/Operations/DeleteUser.hs
Write file to path: out/src/OpenAPI/Types.hs
Write file to path: out/src/OpenAPI/TypeAlias.hs
Write file to path: out/src/OpenAPI/Types/ApiResponse.hs
Write file to path: out/src/OpenAPI/Types/Order.hs
Write file to path: out/src/OpenAPI/Types/Pet.hs
Write file to path: out/src/OpenAPI/Types/Tag.hs
Write file to path: out/src/OpenAPI/Types/User.hs
Write file to path: out/src/OpenAPI/Configuration.hs
Write file to path: out/src/OpenAPI/SecuritySchemes.hs
Write file to path: out/src/OpenAPI/Common.hs
Write file to path: out/src/OpenAPI/Types/ApiResponse.hs-boot
Write file to path: out/src/OpenAPI/Types/Order.hs-boot
Write file to path: out/src/OpenAPI/Types/Pet.hs-boot
Write file to path: out/src/OpenAPI/Types/Tag.hs-boot
Write file to path: out/src/OpenAPI/Types/User.hs-boot
Write file to path: out/openapi.cabal
Write file to path: out/stack.yaml
finished

As you see, out/src/OpenAPI/Types/Category.hs is never generated. Consequently, the generated package does not build.

what I want to see

Everything builds.

why I think it should

This can be done. openapi-generator generate --input-spec petstore.yaml --generator-name 'haskell-http-client' successfully generates a Haskell package that builds and includes a data type PetstoreCategory according to specification. It seems we only need to tweak the parser of references and teach it to go get the file being referred.

what I am going to do

I am going to try to fix this issue, but I cannot promise anything. If you do not hear from me within a week, I must have decided to try something else instead.

[kindaro]

Actually, it looks like this change will entail a wide refactoring. There are some places where local references are hard coded, like so:

-- | Maps the subtypes of components to the entries of the 'ReferenceMap' and filters references (the lookup table should only contain concrete values).
buildReferencesForComponentType ::
  Text ->
  (a -> ComponentReference) ->
  Map.Map Text (OAT.Referencable a) ->
  [(Text, ComponentReference)]
buildReferencesForComponentType componentName constructor =
  fmap (BF.first (("#/components/" <> componentName <> "/") <>))
    . Maybe.mapMaybe (convertReferencableToReference constructor)
    . Map.toList

It seems best to make a new sum type for references.

[kindaro]

Alright, I know how to solve this.

I am going to write a JSON parser that follows references, either within our outside the given file. It shall have type ByteString → IO Value. Then I am going to wire it into the program instead of decodeFileEither here:

-- | Decodes an OpenAPI File
decodeOpenApi :: Text -> IO OAT.OpenApiSpecification
decodeOpenApi fileName = do
  res <- decodeFileEither $ T.unpack fileName
  case res of
    Left exc -> die $ "Could not parse OpenAPI specification '" <> T.unpack fileName <> "': " <> show exc
    Right o -> pure o

This code is already in IO, so we can read all the referenced files and even follow HTTP links right here.

This will also make the code that currently handles references needless, so I shall remove that code.

[NorfairKing]

@kindaro Before you spend too much time going down this rabit hole: is the example that you showed even a valid OpenAPI spec?

[kindaro]

Yes. Here is the relevant piece of OpenAPI specification.

By the way, I just noticed: OpenAPI specification says that references are only allowed in specific places of an OpenAPI document. But I think it will be easier for us to unfold references everywhere in JSON, than to sift through the whole specification and make sure we throw an error when a reference is found where it is not allowed. If we are somewhat more lenient than specified, this is even better, right?

[kindaro]

Turns out we cannot simply unfold all references, because cyclic references are allowed by the specification and handy in practice, but trying to unfold them will not end. See https://github.com/OAI/OpenAPI-Specification/issues/822 for discussion and examples of cyclic references.

Instead, it seems I should need to build some kind of a «map» that accounts for all files connected by references, and all references that show up in those files. It is not super clear how to do this yet. One problem is that references can be relative. Textually identical references that show up in different files can refer to different stuff, and textually different references that show up in different files can refer to the same thing. If we want to have a map with references being keys and pieces of Data.Aeson.Value being values, it seems we should «normalize» references so that textually different references refer to different stuff, and textually identical references refer to the same thing, before building our map. It is not super clear how to perform such a normalization.

[joel-bach]

I am going to try to fix this issue, but I cannot promise anything. If you do not hear from me within a week, I must have decided to try something else instead.

Based on this I assume that the plan to implement this here has been abandoned, so I'll close this issue for the moment. I think it would be neat to have this in the generator but I do not have the time currently to implement it myself. If someone has a PR implementing this, I'll definitely have a look.