Support REST API doc generation

[ktoso]

Issue by guersam Saturday Dec 20, 2014 at 23:05 GMT Originally opened as https://github.com/akka/akka/issues/16591

---

Related ML thread: https://groups.google.com/forum/#!topic/akka-user/d18_smRGeoE

Hi,

Although this topic has been discussed several times in spray-user list [1], I'd like to refresh this issue because I think it's an important feature for wider adoption as well as the mostly asked ones like WebSocket.

Spray/akka-http's routing DSL is the best among the existing framework/libraries IMO, however, it's cascaded and mixed routing structure makes automated documentation and code generation somewhat difficult. Although there's already good looking spray-swagger, it's annotation based approach seems too verbose sometimes.

A random thoughts for alternatives are:

1. Macro based introspection which extracts routing structure from the DSL
2. Integration with the testkit (either way of doc <-> test code generation) instead of with routing code directly

I know it isn't urgent now, just hoping to gather some wisdom to bring it to the next step for the long term goal.

Regards, Jisoo

[1] https://groups.google.com/forum/#!searchin/spray-user/swagger [2] https://github.com/gettyimages/spray-swagger [3] http://www.scalatra.org/2.3/guides/swagger.html

[ktoso]

Comment by ktoso Monday Dec 22, 2014 at 10:23 GMT

---

Hi there, I agree this would be a really really good feature. Thanks for making it an issue so we don't forget about it!

Question being if it isn't better to be taken up as an community effort – as we currently do not have enough man-power to tackle this one.

// cc @jrudolph @sirthias

[ktoso]

Comment by drewhk Monday Dec 22, 2014 at 10:29 GMT

---

I put the label on it, and see what happens ;)

[ktoso]

Comment by hepin1989 Monday Dec 22, 2014 at 11:12 GMT

---

https://github.com/gettyimages/spray-swagger like this one?

[ktoso]

Comment by guersam Friday Dec 26, 2014 at 09:36 GMT

---

@hepin1989 As mentioned in the first post, spray-swagger looks good but IMHO, it sometimes hurts the conciseness of the routing DSL with verbose annotations. This thread is to discuss various approaches from scratch in a bigger community.

[ktoso]

Comment by hepin1989 Friday Dec 26, 2014 at 11:02 GMT

---

@guersam wow,that should be very nice,in fact I don't think add the annotation should be elegant too.and the best way is provide a sbt plugin,which could generate the api doc.

[ktoso]

Comment by sirthias Wednesday Feb 11, 2015 at 14:30 GMT

---

As already discussed several times on the spray ML automatically producing documentation for the current server-side routing DSL is close to impossible to do robustly. The DSL is built upon nested functions which are fully opaque to the outside. One would have to use macros to inspect them at compile time. Also, because the DSL allows any kind of Scala construct on all levels any kind of route structure analysis tool will be very easy to trip up.

As a simply example, think about how you would want to automatically document this snippet:

path("order" / IntNumber) { orderId =>
  if (orderId < 10000) // legacy orders only support GET
    get {
      parameters('foo, 'bar) { (foo, bar) =>
        // ...
      }
    }
  else
    get {
      parameters('baz) { baz =>
        // ...
      }
    } ~
    post {
      // ...
    }
}

In its current form the routing DSL is simply too powerful to allow for robust automatic inspection. We will have to evolve it into something less powerful, that additionally separates route structure construction from route structure exection (2 phases instead of one, as we have it know), in order to have a chance to to this properly.

[ktoso]

Comment by rkuhn Thursday Feb 12, 2015 at 15:22 GMT

---

Possible idea: instead of trying to analyze and interpret the route definition, why not observe its execution while running the test suite (that every app surely must have)? The added benefit is that only those parts are documented that are also exercised in the tests. Another component here would be to provide test fixtures for certain RESTful resource patterns to make this task easier.

[ktoso]

Comment by kodemaniak Friday Feb 27, 2015 at 16:04 GMT

---

+1

Using the tests to produce documentation for a REST API sounds like a good idea. For Spring MVC there is

https://github.com/spring-projects/spring-restdocs

I specifically like the integration with asciidoctor.

[ktoso]

Comment by drewhk Friday Feb 27, 2015 at 16:08 GMT

---

interesting idea. If this is ever implemented then it would be nice to have an intermediate format emitted so that it can be used in various other ways.

[ktoso]

Comment by oseval Friday Feb 27, 2015 at 20:17 GMT

---

One more idea: instead of generating documentation by API it possible would be better generate an interface of API from other format like swagger.

[ktoso]

Comment by hhimanshu Saturday Jul 11, 2015 at 05:25 GMT

---

Any progress or recommendation on this?

[ktoso]

Comment by ktoso Saturday Jul 11, 2015 at 13:16 GMT

---

The akka team does not have enough people/time to tackle this problem (shipping stable 1.0 releases takes priority over this extra feature).

External contributions are more than welcome!

[ktoso]

Comment by kodemaniak Sunday Jul 12, 2015 at 17:18 GMT

---

A couple of weeks ago I've done a proof-of-concept, that tried to provide something similar to Spring Restdocs (https://github.com/spring-projects/spring-restdocs) for akka-http. The project can be found here:

https://github.com/kodemaniak/akka-http-restdoc

(I hope the README is accurate and does not only work on my computer)

That POC is far from being complete. But currently I use a spray port of it to document our main microservice at work. My conclusion after a couple of weeks is that following such an approach (i.e. using asciidoctor for the bulk of documentation and extracting request/reply information from the tests) feels like the right way to go. It fits very well together. It is much easier to keep the docs in sync with the actual behaviour, at least the request/response headers and bodies are generated from tested code, and you save some work since you don't have to write down JSON bodies or HTTP requests by hand.

I thought I throw that POC into this dicsussion to get some more feedback. Our spray port is a bit more advanced already, but adding the missing bits and pieces also to the akka-http version would be rather simple.

[ktoso]

Comment by analytically Sunday Jul 12, 2015 at 22:25 GMT

---

@kodemaniak awesome approach, would love to see this developed further!

[ktoso]

Comment by rkuhn Monday Jul 13, 2015 at 08:07 GMT

---

Thanks @kodemaniak for pushing this, looks like a good starting point!

[ktoso]

Comment by phani1kumar Thursday Aug 27, 2015 at 05:22 GMT

---

I have recently started to know about akka-http and scala and started off to build my first microservice in this technology. I humbly accept that I am not qualified to make points in this space, but would like to seek guidance on the following points:

1. We are pretty much used to follow javadoc/scaladoc standards while writing code in which we already provide the documentation of the methods, and parameters. Can we somehow plug this information into the swagger API? through macros / some scala magic! ==> If there is some specific documentation only to be known to the programmers / maintainers, we may introduce exclusion syntax in the existing javadoc/scaladoc comments!
2. Scala as per my lame understanding has a very strong type support, hence the optional/ mandatory part of the types may be made available to swagger transparently again through macros / some scala magic

By following the above two points, we may avoid large portions of the annotations and the annotations we will be left with would be the routes! As mentioned earlier by @sirthias automating this portion of annotations might be difficult, and if we can avoid rest of the clutter may be we can live with this one!

I know, I might be sounding stupid, but I just want to get your expert opinion on my lame thoughts.

[ktoso]

Comment by michallepicki Monday Aug 31, 2015 at 09:48 GMT

---

My not-so-pretty attempt at allowing to write separate endpoints that are automatically documented in swagger.

This is ugly and very likely has errors, e.g. I have a problem with merging parameter directives with (my own) SwaggerPathMatcher. I can't compile

path("user" / IntMatcher("userNumber")) & parameter('x.as[Int]) when I'm using the original parameter Directive so parameters can't be auto-documented yet.

But the general idea is to give an interface for writing simple auto-documented API endpoints that generate akka-http directives and to allow extending and wrapping them with normal akka-http directives, what do you think?

[ktoso]

Comment by ktoso Thursday Jan 07, 2016 at 11:55 GMT

---

Note for interested parties, I currently believe to be more efficient and powerful: generating the structure from tests, similar to: http://docs.spring.io/spring-restdocs/docs/1.0.x/reference/html5/#getting-started-documentation-snippets-invoking-the-service

[ktoso]

Comment by pshirshov Thursday Jan 07, 2016 at 13:34 GMT

---

Guys-guys-guys.

Automatic docs generators is the only one part of problem.

What about me - I'm strictly need API schemas. It's impossible to generate schema during the test.

Akka-http is definitely needs directives tree introspection. And it's not so hard to implement this. I may try to start this project, but I need to be sure that it will be accepted to upstream.

[ktoso]

Comment by ktoso Thursday Jan 07, 2016 at 13:35 GMT

---

I suggest starting out as separate project and if it makes sense to be pulled into Akka HTTP we can do so later on. I agree it's a really awesome thing to have, but I also think it's best if we start incubating it outside the project and pull in once mature and you're willing to contribute (and there actually is a need to be pulled in – it could be a separate lib and do it's job well :-))

[ktoso]

Comment by pshirshov Thursday Jan 07, 2016 at 13:45 GMT

---

Ok, I have to possible ways to implement this:

1. External library, which will duplicate all the features of scaladsl and be able to build tree with introspection data.
2. Break into directives and built introspection abilities in.

I prefer second option. I already have partial implementation of (1) and it requires a lot of work until completion.

[ktoso]

Comment by ktoso Thursday Jan 07, 2016 at 13:47 GMT

---

trait IntrospectableDirectives which replicates API and delegates to the Akka Directives and you're good to go.

Keep method parity and it's a drop-in replacement for pure Akka HTTP then. I don't see why it has to be "break into Akka's directives" right away. It can be a library, exposing same methods. If it works out, everyone is happy, and if you'd like to contribute then we have a place to start from.

[ktoso]

Comment by pshirshov Thursday Jan 07, 2016 at 13:51 GMT

---

Ok, got it. Will try to implement own directives hierarchy keeping same interfaces.

[ktoso]

Comment by pjfanning Wednesday Feb 03, 2016 at 13:24 GMT

---

Hi, I have forked the swagger-spray project (https://github.com/swagger-akka-http/swagger-akka-http) and I have published it to maven central. The code requires the swagger.io annotations and JAX-RS annotations but at least, gives support for creating Swagger API models based on request and response case class definitions. It isn't ideal to duplicate the routing informational in Swagger annotations but it may be useful as an interim solution. The swagger definition allows for a lot of description about the API that can't be inferred from the Akka-Http Routing DSL so any solution that does generate swagger docs from Akka-Http routes would ideally allow informational items like license details and API descriptions to be provided.

[ktoso]

Comment by bfil Monday Apr 04, 2016 at 14:42 GMT

---

@FeiWongReed have you made any progress around the IntrospectableDirectives? I'm trying to look into using a similar approach to generate basic API docs. I'm hitting huge road blocks though around some of the types, for example the PathMatcher obfuscates everything about the matched requests paths, and needs to be rebuilt in a compatible/introspect-able way. How far have you gone with this?

[ktoso]

Comment by pshirshov Thursday Apr 07, 2016 at 18:48 GMT

---

Unfortunately I still have not enough time to finish it. I have my own subset, working for my services, but its dirty and can't be used in arbitrary project.

[ktoso]

Comment by pshirshov Thursday Apr 07, 2016 at 18:51 GMT

---

for example the PathMatcher obfuscates everything about the matched requests paths, and needs to be rebuilt in a compatible/introspect-able way. How far have you gone with this?

You may use implicit macros and construct not only pathmatchers but also additional introspection data. Anyway its hard to do that in general way. Proper solution is to make scaladsl introspectable.

[chadselph]

It's safe to say with the current DSL this will never be possible in an arbitrary way.

I think the most reasonable approach is to have some kind of library that's much more opinionated about your route structure, build an intro-spectable view of what it's created, and then spits out something that conforms to akka.http.scaldsl.server.Route.

I have something similar to @pshirshov, but it's just a proof of concept and requires some hairy shapeless magic for deriving JSON encoders/validators. Here's what I have for PathMatchers specifically:

trait ApiPath[L] { left =>

  val pathSegmentDescriptions: Seq[String]
  val matcher: PathMatcher[L]

  def /[R](other: ApiPath[R])(implicit join: Join[L, R]): ApiPath[join.Out] = {
    new ApiPath[join.Out] {
      override val pathSegmentDescriptions: Seq[String] =
        left.pathSegmentDescriptions ++ other.pathSegmentDescriptions
      override val matcher: PathMatcher[join.Out] = left.matcher / other.matcher
    }
  }
  def /[R : DocumentedUrlParameter](other: PathMatcher1[R])(implicit join: AppendOne[L, R]): ApiPath[join.Out] = {
    new ApiPath[join.Out] {
      override val pathSegmentDescriptions: Seq[String] =
        left.pathSegmentDescriptions ++ Seq("<" ++ implicitly[DocumentedUrlParameter[R]].name ++ ">")
      override val matcher: PathMatcher[join.Out] = left.matcher / other
    }
  }
}

trait DocumentedUrlParameter[T] {
  def name: String
}

object DocumentedUrlParameter {
 // implicit impls here
}

object ApiPath {

  implicit def stringToApiPath(s: String): ApiPath[Unit] = new ApiPath[Unit] {
    override val pathSegmentDescriptions: Seq[String] = Seq(s)
    override val matcher: PathMatcher[Unit] = s : PathMatcher0
  }

  def /(s: String): ApiPath[Unit] = stringToApiPath(s)

}

can then be generated with

case class ApiFormats[BasePathType, InstPathType](basePath: ApiPath[BasePathType], instancePath: ApiPath[InstPathType])

  val apiFormats = new ApiFormats(
    ApiPath / "v1" / accountIdMatcher / "foos",
    ApiPath / "v1" / accountIdMatcher / "foos" / otherMatcher
  )

println(apiFormats.basePath) // "/v1/<account>/foos"

[pshirshov]

I have something similar to @pshirshov, but it's just a proof of concept and requires some hairy shapeless magic for deriving JSON encoders/validators.

Yup-yup. Unfortunately, all the wrappers around current scaladsl are "just a proof of concept" because, as @ktoso said scaladsl is too powerful and/or we don't have enough time to make a small revolution here.

scaladsl definitely needs to evolve into something simpler and better introspectable. Or we may try to implement a "simplescaladsl" which would be based on scaladsl and provide a functional subset enough for real world purposes. And yes, it should compose routing tree statically and expose it in a traversable form.

I really wish to do that, but I have no time unfortunately.

An yup, I agree that the initial idea with adding some kind of metainformation into the each directive would not work because of dynamic nature of scaladsl.

[jtownson]

I have created a library that provides a workable solution. There is a post about it on Akka-Users at https://groups.google.com/forum/#!searchin/akka-user/townson%7Csort:relevance/akka-user/Pet2WaavbJA/AW66zyEGAQAJ

At the moment, it's on bitbucket at https://bitbucket.org/jtownson/swakka.

The approach I took is to provide a set of case classes that can be instantiated to represent an OpenApi (e.g. the petstore example). The swakka library then generates akka routes and swagger.json.

If you're interested to read more, the project does have a fairly extensive readme given that it's an early release.

I've been returning periodically to this ticket for what seems like an eternity, so I'd really like to try and get swakka fully functional in the next few months -- so this ticket might finally be closed. If you care about swagger support for akka-http, I'd really appreciate your feedback.