Support for Open API Spec in YAML in addition to JSON

[bombaywalla]

I was attempting to use martian for the Amazon Ads API (specifically Sponsored Products) but that seems to have a Open API Spec in YAML., which does not seem to be supported by martian. I converted that YAML spec to EDN using io.forward/yaml, and then fed that to (martian/bootstrap-openapi amznapi-base-uri oas-edn martian-http/default-opts)) but that gives me a NPE (during printing).

Execution error (NullPointerException) at martian.openapi/process-responses$iter$fn (openapi.cljc:98).
Cannot invoke "clojure.lang.Named.getName()" because "x" is null

I'm new to OpenAPI and martian. Happy to dig deeper and work on a PR with some guidance.

[oliyh]

Hello,

Thanks for reporting this. Could you attach the edn and yaml files please?

Thanks

[bombaywalla]

Had to change the extensions of the files to .txt to get github to allow the upload. The internal oas-edn was created by (yaml/parse-string (slurp "sponsored-products-yaml.txt")). The edn file was created by (spit "sponsored-products-edn.txt" oas-edn).

sponsored-products-yaml.txt sponsored-products-edn.txt

[bombaywalla]

On taking a closer look, it seems that the EDN that is generated from the YAML may be faulty. For example, the following YAML snippet

  /v2/sp/campaigns:
    post:
      tags:
        - Campaigns
      operationId: createCampaigns
      summary: Creates one or more campaigns.
      parameters:
        - $ref: '#/components/parameters/clientHeader'
        - $ref: '#/components/parameters/profileHeader'
      requestBody:
        description: An array of campaigns.
        content:
          application/json:
            schema:
              type: array
              minItems: 0
              maxItems: 100
              items:
                $ref: '#/components/schemas/CreateCampaign'
      responses:
        207:
          description: Success.
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/CampaignResponse'
        401:
          $ref: '#/components/responses/Unauthorized'
    put:
      tags:
        - Campaigns
      operationId: updateCampaigns
      summary: Updates one or more campaigns.
      parameters:
        - $ref: '#/components/parameters/clientHeader'
        - $ref: '#/components/parameters/profileHeader'
      requestBody:
        description: An array of campaigns with updated values.
        content:
          application/json:
            schema:
              type: array
              minItems: 0
              maxItems: 100
              items:
                $ref: '#/components/schemas/UpdateCampaign'
      responses:
        207:
          description: Multi-status. An array of campaign response objects reflecting the same order as the request.
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/CampaignResponse'
        401:
          $ref: '#/components/responses/Unauthorized'
    get:
      tags:
        - Campaigns
      operationId: listCampaigns
      summary: Gets an array of campaigns.
      parameters:
        - $ref: '#/components/parameters/clientHeader'
        - $ref: '#/components/parameters/profileHeader'
        - $ref: '#/components/parameters/startIndex'
        - $ref: '#/components/parameters/count'
        - $ref: '#/components/parameters/stateFilter'
        - $ref: '#/components/parameters/name'
        - $ref: '#/components/parameters/portfolioIdFilter'
        - $ref: '#/components/parameters/campaignIdFilter'
      responses:
        200:
          description: Success.
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Campaign'
        401:
          $ref: '#/components/responses/Unauthorized'
        404:
          $ref: '#/components/responses/NotFound'

results in the following EDN snippet

[:/v2/sp/campaigns
 {:post
  {:tags ["Campaigns"],
   :operationId "createCampaigns",
   :summary "Creates one or more campaigns.",
   :parameters
   [{:$ref "#/components/parameters/clientHeader"}
    {:$ref "#/components/parameters/profileHeader"}],
   :requestBody
   {:description "An array of campaigns.",
    :content
    #:application{:json
                  {:schema
                   {:type "array",
                    :minItems 0,
                    :maxItems 100,
                    :items
                    {:$ref
                     "#/components/schemas/CreateCampaign"}}}}},
   :responses {nil {:$ref "#/components/responses/Unauthorized"}}},
  :put
  {:tags ["Campaigns"],
   :operationId "updateCampaigns",
   :summary "Updates one or more campaigns.",
   :parameters
   [{:$ref "#/components/parameters/clientHeader"}
    {:$ref "#/components/parameters/profileHeader"}],
   :requestBody
   {:description "An array of campaigns with updated values.",
    :content
    #:application{:json
                  {:schema
                   {:type "array",
                    :minItems 0,
                    :maxItems 100,
                    :items
                    {:$ref
                     "#/components/schemas/UpdateCampaign"}}}}},
   :responses {nil {:$ref "#/components/responses/Unauthorized"}}},
  :get
  {:tags ["Campaigns"],
   :operationId "listCampaigns",
   :summary "Gets an array of campaigns.",
   :parameters
   [{:$ref "#/components/parameters/clientHeader"}
    {:$ref "#/components/parameters/profileHeader"}
    {:$ref "#/components/parameters/startIndex"}
    {:$ref "#/components/parameters/count"}
    {:$ref "#/components/parameters/stateFilter"}
    {:$ref "#/components/parameters/name"}
    {:$ref "#/components/parameters/portfolioIdFilter"}
    {:$ref "#/components/parameters/campaignIdFilter"}],
   :responses {nil {:$ref "#/components/responses/NotFound"}}}}]

Note that the response status codes are missing.

[oliyh]

Yes, those nil status codes correspond to the exception that's being thrown.

[bombaywalla]

Thanks. That addresses the NPE.

[bombaywalla]

I was able to generate EDN from the YAML that was acceptable to martian. To help anyone wanting to add built-in YAML support for martian, here is what I needed to do.

  (require '[clj-yaml.core :as yaml])
  (require '[martian.core :as martian])
  (require '[martian.hato :as martian-http])
  (require '[martian.openapi :as openapi])
  (require '[clojure.walk :as w])

  (defn cleanup
    "Clean up the EDN returned by clj-commons/clj-yaml
    to be compatible with what martian expects."
    [edn]
    (w/postwalk (fn [x]
                  (cond
                    ;; replace all LazySeqs with Vectors
                    ;; See https://github.com/clj-commons/clj-yaml/pull/18
                    (and (seq? x) (not (vector? x)))
                    (into [] x)
                    ;; make sure all the keys of maps are keywords, including keys that were numbers
                    ;; See https://github.com/clj-commons/clj-yaml/blob/master/src/clojure/clj_yaml/core.clj#L128-L129
                    (map? x)
                    (into {} (map (fn [[k v]] [(or (keyword k) (if (number? k) (keyword (str k)) k)) v]) x))
                    ;; otherwise do nothing
                    :else
                    x))
                edn))

  (defn yaml->edn
    "Convert a YAML OpenAPI Spec to EDN compatible with martian."
    [url]
    (-> url
        (slurp)
        (yaml/parse-string)
        (cleanup)))

  (let [sp-url "https://d3a0d0y2hgofx6.cloudfront.net/openapi/en-us/sponsored-products/2-0/openapi.yaml"
        edn (yaml->edn sp-url)
        base-url (openapi/base-url edn)
        m (martian/bootstrap-openapi base-url edn martian-http/default-opts)]
    (martian/explore m))

[bombaywalla]

If you'd like a PR that handles YAMP OpenAPI specs for just CLJ clients, I'm happy to make one based on the above comment.

[oliyh]

Hi @bombaywalla ,

Thanks for your work on this. I'd be happy to accept a PR, you could make a new namespace called martian.yaml in the core/src directory.

Thanks

[bombaywalla]

I'm presuming you want a martian.yaml cljc file but one that will (currently) only work for CLJ. If something different, let me know.

[oliyh]

Hi,

You might as well make it a clj file if it won't have any public cljs fns, makes it clearer how to use it.

Cheers

[bombaywalla]

Okay.

[awb99]

Very keen to see yaml working. It seems many apis only publish yaml. Xero accounting api for example https://raw.githubusercontent.com/XeroAPI/Xero-OpenAPI/master/xero_accounting.yaml

Converting such a huge api to edn seems like a lot of room for translation errors.

[oliyh]

Added in #132 and available in 0.1.21-SNAPSHOT