Breaking up large swagger.yaml files

[patrickw276]

Is there any way to break up large swagger.yaml files into smaller pieces? I've tried using external references but it doesn't seem like Connexion supports these. I've also tried using jinja features to break the swagger files up but this doesn't work (and its ugly).

Am I missing a feature that would let me do this? If not, is relative referencing a feature Connexion would be interested in? I know that the jsonschema library used internally supports relative referencing so it might not be too difficult to implement. If you guys are interested, I can take a crack at adding the feature.

[tahmidefaz]

@tomghyselinck Anything you guys ended up doing to work around it? Currently, dealing with the same issue.

[HRogge]

I decided not to look at OpenAPI again until they got to version 3.1 I lost too much time on the inconsistencies between JSON-Schema and (OpenAPI3)-JSON-Schema.

[tomghyselinck]

Hi @tahmidefaz, unfortunately I haven't been able to work around it.

We haven't split up the (meanwhile very large) OpenAPI yaml file.

I have been playing around with the connexion and openapi-spec-validator code but haven't been able to make some mature changes.

[tahmidefaz]

@tomghyselinck thank you for the reply!

[RonnyPfannschmidt]

i took over the maintenance of prance and started to work towards supporting recursive refs and openapi 3.1

[Glutexo]

Recursive references are going to be supported very shortly. There is already an alternative parser ready that handles this correctly.

Update 1: A pull request here – https://github.com/RonnyPfannschmidt/prance/pull/101 Update 2: Merged!

[aniketbhatnagar]

Thank you for fixing this. Any clue when this will be released?

[aniketbhatnagar]

Thank you for fixing this. Any ETA on when to expect recursive references to work?

[advance512]

@Glutexo is this already part of the latest released version?

[RonnyPfannschmidt]

@advance512 recursive references are part of the latest release, but it needs a correct parser configuration atm

[alexandr-san4ez]

One more way to use $ref with separate files:

import connexion
from connexion.json_schema import default_handlers as json_schema_handlers

app = connexion.App(__name__, specification_dir='swagger/')

json_schema_handlers[''] = lambda uri: json_schema_handlers['file'](str(app.specification_dir / uri))
app.add_api('spec_v1.yaml')

I tested it on connexion==2.13.1. Not sure how it will work with other versions.

Perhaps someone will be useful.

[wackykid]

i really do wish this feature is implemented too... i am involved in a large scale project where there are multiple developers contributing to the development so having a single YAML file is too huge and difficult to manage... so we split it into multiple files by different modules each of them being a microservice... but collectively they are all considered as part of the same system...

the way we have to deal with this is to have a script to generate flask server for each YAML file and having each of them run independently with its own flask instance using different ports... ideally we would want to be able to just run a single flask server so that we don't have to deal with a different port for each module/microservice...

[erans]

The above solution (https://github.com/spec-first/connexion/issues/254#issuecomment-1133843523) help load multiple yaml files referenced from a main one, you also need this:

def my_resolve_refs(spec, store=None, handlers=None):
    """
    Resolve JSON references like {"$ref": <some URI>} in a spec.
    Optionally takes a store, which is a mapping from reference URLs to a
    dereferenced objects. Prepopulating the store can avoid network calls.
    """
    from connexion.json_schema import default_handlers

    spec = deepcopy(spec)
    store = store or {}
    handlers = handlers or default_handlers
    resolver = RefResolver("", spec, store, handlers=handlers)

    def _do_resolve(node):
        if isinstance(node, Mapping) and "$ref" in node:
            path = node["$ref"][2:].split("/")
            try:
                # resolve known references
                node.update(deep_get(spec, path))
                del node["$ref"]
                return node
            except KeyError:
                # resolve external references
                with resolver.resolving(node["$ref"]) as resolved:
                    # if not fixed:
                    #     return resolved
                    # else:
                    return _do_resolve(resolved)
                    # endfix
        elif isinstance(node, Mapping):
            for k, v in node.items():
                node[k] = _do_resolve(v)
        elif isinstance(node, (list, tuple)):
            for i, _ in enumerate(node):
                node[i] = _do_resolve(node[i])
        return node

    res = _do_resolve(spec)
    return res

A slightly modified $ref resolver that is recursive, so if you see somewhere inside the files $ref: fileb.yml/MyObject it will know how to resolve it (even if its not on the main file.

Once you have this function in the code you can patch it by applying:

spec.resolve_refs = my_resolve_refs

Prior to calling add_api.

This is basically a combination of a few things I've found online that helped us manage it without forking connexion.

[zdanial]

tried a bunch of the solutions here but couldn't get any to work. opted for the quick and dirty approach as follows:

app = connexion.App(__name__, specification_dir='swagger/')
with open('./myapi/swagger/myapi_merged.yml', 'w') as out, \
    open('./myapi/swagger/myapi.yml', 'r') as main, \
    open('./myapi/swagger/myapi_schemas.yml', 'r') as schemas:

    main_yml = main.read()
    merged_yml = main_yml.replace('{{schemas}}', schemas.read())

    out.write(merged_yml)

app.add_api('myapi_merged.yml')

so the last part of main.yml is:

components:
    ...
    schemas:
    {{schemas}}

don't need to use {{schemas}} but figured jinja syntax would be most readable but any unique flag here works.

[phasath]

Is referencing not working yet? The issue seems a bit confusing due to some answers in which it was supported and merged, but not working?