Some architectural improvements

[anatoliykmetyuk]

There are architectural problems that would prevent me from using this project should I decide to write a real-world web app. It is possible this project is just fine-tuned for SPAs or that my approach is not what people use in practice. However, here are my thoughts.

Were I writing a web app, I would have seen MetaRouter as a convenient and functional way to specify the routes. I would also have had some backend that is supposed to work with these routes (I know how to work with Java Servlets, Finch, Scalatra and Play, so I'd probably chose one of these. Not the first one for sure :) ).

The lifecycle of a route you present a user with goes as follows:

1. Create a route
2. Create a mapping of a route to a case class
3. Create a parse table with all your routes
4. Invoke parse method on that table to generate case classes

Examples in the context of other frameworks

Consider we want to describe a path /details/{int}.

Java Servlets

Everything is fine there. You can define a handler for any request that comes your way, extract the request URL, parse it via the parse table and match on the case class. Another thing is that no Scala programmers really use that.

Finch

The endpoints are specified individually for each URL, you can't specify one endpoint for all. For /details/{int} you will write "details" :: int. And you do not want to parse that to a case class - I would have found it a clatter on Finch. What we want to do instead is handle the request in place. We had experience with Finch and MetaRouter, so we know how it went. Precisely, we ended up folding the HList with path fragments.

Scalatra

Endpoints are specified in some controller, and for our example you would write get("/details/:int") {/* handling code */}. In in Finch we were folding the path to its own format, here we need to fold it to a String, with the framework-specific formatting. Again, I do not see any need for case classes here, I want to handle the request in place.

Play

Even more interesting here, it requires you to have a separate file with all your routes mapped to the controllers. Very heavy approach, if you ask me. Were I using MetaRouter here, I would expect to take the entire List of routes I have and fold it into that file, on compile time. Naturally I do not want the mapping to case classes here either - what I want instead is the controllers with the handling code to be generated for me.

Conclusion

The approach MetaRouter presets to users has an implication that the web framework the users are using has a handler method that will handle all the requests, then the control is given up to MetaRouter.

However, this is a scenario valid for the low-level frameworks like Servlets - most people will use something higher-level, and they may have their own ways of specifying handlers.

Also, mapping requests to case classes is heavy, I personally would not want to define extra classes just for the requests, as it clutters the code.

Architectural improvements

Let's see how the lifecycle above could be adapted to make a seamless integration with these frameworks possible:

1. Create a route

The routes in their current form are good: all the elements are reified into a HList, so that they can be easily accessible.

2. Create a mapping of a route to a case class

More often than not we want another operation to be performed instead - not a mapping to a case class, but a handling function execution. Case class is just a special case of handling an incoming request - (id: Int) :: HNil => Details(id). So why have Router.route generate a MappedRoute infrastructure, if we can have an arbitrary function there, resulting in a HandledRoute?

3. Create a parse table with all your routes

This is good, because all the routes end up in the same data structure and are easily accessible from the outside world.

4. Invoke parse method on that table to generate case classes

parse method will hardly ever be invoked in high-level frameworks, since the framework takes care of parsing the URL. Instead, we want to collapse this parse table into a set of instructions for the framework on how to handle the requests.

Theory

What I am talking about here is a more "free" approach to things:

1. Define your routes in a structure, describing them perfectly (as in HLists and case classes, where all the members and types are visible), but not enforcing you to do things in a particular way (as in mapping stuff to case classes and then parsing URLs via the parse table).
2. Define capabilities to collapse these data structures to something concrete, like fold in the current implementation - catamorphisms - see this paper. This "something concrete" is a set of instructions for the target framework on how to do things.

Practice

Let's see how to do best our /details/{int} example, so that it is usable from the frameworks described above.

Step 1 from Theory:

The route becomes Route(GET, "details" :: Arg[Int] :: HNil), as it is now.

Moreover, we also want to encapsulate the handling code here - instead of Router.route[CaseClass](ourRoute), we want the actual handling logic encapsulated here:

val route = MappedRoute(Route(GET, "details" :: Arg[Int] :: HNil), (id: Int) :: HNil => FlatMapped(GetModelFromDB(id), model => ReturnTemplate(model))

val table = RouteTable(route :: HNil)

(Free monad goes after =>, probably also doable with an effectful monad. No problem if we don't need the handlers here and actually need just a case class as in the current implementation - just use Route, see Java Servlet example below).

Note how we have managed to describe both the route and the handling logic without depending on any framework. As well as to unify all our routes (one) into a table, which is a wrapper over HList.

Naturally in the real world people usually stick to a single framework (maybe). But for me, this framework independency is for reducing the mental load of learning a new framework foremost: you first define what you want to do, and then take your time to learn how to do it with the framework in question.

Step 2 from Theory

Now let's see how the above works out with the four frameworks from the previous section. For every framework, we need to collapse table (apply a catamorphism) to something the framework will understand.

Java Servlets

The current implementation works fine, let's see how to mimic it.

trait Mapping
case class Details(id: Int) extends Mapping

How the catamorphism will map things:

• Route --> String => Option[Mapping] - a route optionally can match a string, and if the match succeeds, the case class is returned (more type safety needed here, Mapping should be Details)
• MappedRoute(route, handler) --> (String => Option[Mapping], Mapping => Unit) - if you have already associated a handler with the mapping.
• RouteTable(mappedRoutes) --> String => Unit - optionally handle the mapped routes
• RouteTable(nonMappedRoutes) --> String => Option[Mapping] - if you have not specified the handlers, but need the case classes instead.

Finch

• Route(GET, "details" :: Arg[Int]) --> get("details" :: int): Endpoint
• MappedRoute(route, handler) --> collapseRoute(route).apply(handler): Endpoint
• RouteTable(routes <: HList) --> routes.map(collapseRoute) match { case r1 :: r2 :: ... :: HNil => r1 :+: r2 :+: ... }: Endpoint

Scalatra

• Route(GET, "details" :: Arg[Int]) => "details/:int": String
• MappedRoute(route, handler) => get(collapseRoute(route)) { handler } (get apparently is effectful in Scalatra; and translate that GET into get() somehow, I did not think that far, but should not be hard)
• RouteTable(routes) => routes.foreach(collapse) - effectfulness is evil, but that's what the framework wants.

Play

• Route(GET, "details" :: Arg[Int]) => "GET /details/:int Handlers.firstHandler" - the entry of the routes file.
• MappedRoute(route, handler) => inject the def firstHandler = {handler} method in the synthetic Handlers class - probably from a macro. Play is heavy.
• RouteTable(routes) => routes.map(collapseRoute).mkString("\n") - and then this string becomes the routes file.

[tindzk]

Thanks for the thorough analysis! I like your idea of considering the case class mapping just as a special use case, and find an abstraction for it.

The main advantages for choosing case class were:

• They enforce named arguments
• If we have a set of routes, we can match them easily
• We can map them back to an URL while enforcing correct types
• IDE support

One thing we need to explore more is the case where we map from a route back to an URL. I would like to retain the possibility to do this for case classes with your proposed architecture. Let's consider this example:

val details = Route("details" :: Arg[Int] :: HNil)  // Or shorter: Root / "details" / Arg[Int]

While we can instantiate an URL via Router.url(details, 42 :: HNil), we do not know the names of the arguments.

I would argue against including the HTTP method in the route. This library may be used for other protocols like FTP which do not have the notion of GET/POST requests.

Furthermore, I would like to propose the following syntactic changes:

val details  = Router.route(Root / "details" / Arg[Int], (i: Int) => i.toString)
val userInfo = Router.route(Root / "user" / Arg[String], (u: String) => u)

val table    = Router.table(details :: userInfo :: HNil)
val run1     = Router.run(details, "/details/42") // "42"
val run2     = Router.run(table,   "/user/test")  // "test"

It is important that the compiler is able to infer the correct types, e.g. it should be possible to know beforehand that run1 and run2 are strings.

[anatoliykmetyuk]

val run1 = Router.run(details, "/details/42") // "42"

How would you turn that into a Finch/Scalatra/Play handler?

One thing we need to explore more is the case where we map from a route back to an URL. I would like to retain the possibility to do this for case classes with your proposed architecture. Let's consider this example:

We'll just need to have two functions: Route => (String => CaseClass) and Route => (CaseClass => String).

For me, there are two real problems:

• Inability to integrate MetaRouter with other web frameworks
• Case classes are heavy

Personally, for me the fact that you need to do at least three operations to create a new route (the route itself, the case class and the mapping between the two) overrides the benefits you named. It may seem like nothing, but every time you want to define a new route, you need to remember all these operations, and where the code for each should go. This creates mental overhead for such a purely declarative task. If a framework distracts me from my primary task, this is a big disadvantage. IMO case classes should be an option, not an enforced style.

[tindzk]

How would you turn that into a Finch/Scalatra/Play handler?

The definition of Route would stay the same, so the user could write a regular route and then call fold in place:

val r = Root / "details" / Arg[String] / Arg[Int]
val f = ...        // Rewrite route for Scalatra
Router.fold(r, f)  // "/details/:string/:int"

We'll just need to have two functions: Route => (String => CaseClass) and Route => (CaseClass => String).

What do you think about changing MappedRoute to:

case class MappedRoute[ROUTE <: HList, Args <: HList, Result](route: Route[ROUTE], f: Args => Result)

Then we could write:

val r = Root / "details" / Arg[String] / Arg[Int]  // R = Route[String :: Arg[String] :: Arg[Int] :: HNil]
Router.url(r, "test" :: 42 :: HNil)                // "/details/test/42"

case class Details(s: String, i: Int)
val m = Router.route(r, Details.apply)  // MappedRoute[R, String :: Int :: HNil, Details]

Now we could introduce the route product a little differently:

case class RouteProduct[Results <: HList, Routes <: HList](results: Results, routes: Routes)

This maps each result type to its corresponding route. An alternative would be to use an HMap here.

val p = Router.product(m :: HNil)  // RouteProduct[Details :: HNil, R :: HNil]

Finally, we can overload url with RouteProduct:

Router.url(p, Details("test", 42)) // "/details/test/42"

While doing my recent refactoring, I was wondering whether we should eliminate RouteData and the fill operation altogether. What is your take on this?

[anatoliykmetyuk]

What do you think about changing MappedRoute to:

Looks good.

If we have fold for a simple Route, we need a similar function for a MappedRoute. In case of Scalatra, we must first do fold on the Route to get "/details/:int", then apply another function to register a handler: get("details/:int") { handler }. The question is, do we need a framework support for this pattern (handler function composed with the fold catamorphism), or will it be convenient for the user to do perform the pattern by hand?

Finally, we can overload url with RouteProduct:

This def url considers case classes as a special case. How about something like this:

case class IsoMappedRoute[ROUTE <: HList, Args <: HList, Result](route: Route[ROUTE], f: Args => Result, g: Result => Args)

def Route.url(route: IsoMappedRoute, r: route.Result): String = {
  val args = route.g(r)
  Route.url(route.route, args)
}

def Route.url(route: Route, args: Args): String = ???

val route: IsoMappedRoute = Router.route(r, Details.apply, Details.unapply)
val path = Route.url(route, Details(42))

While doing my recent refactoring, I was wondering whether we should eliminate RouteData and the fill operation altogether. What is your take on this?

Probably; if one does need such a functionality, it is just a MappedRoute[Route, Args, FilledRoute](originalRoute, args => substituteArgsIntoRoute(originalRoute, args)).

By the way, all these Route, MappedRoute, and IsoMappedRoute should have the same parent and IMO these classes should be hidden from the user. Route can become RawRoute:

trait Route[A]
case class RawRoute[Path <: HList, Args <: HList] extends Route[Args]  // Raw route's ultimate goal is to extract arguments
case class MappedRoute[A, B](route: Route[A], f: A => B) extends Route[B]
case class IsoMappedRoute[A, B](route: Route[A], f: A => B, g: B => A) extends Route[B]

implicit val f1: Functor[Route] = ???
implicit val f2: Invariant[Route] = ??? // https://static.javadoc.io/org.typelevel/cats-core_2.11/0.9.0/index.html#cats.functor.Invariant

val route = Root / "details" / Arg[Int]

// Warning: Functor <: Invariant, collision will happen here
val mapped = route.imap(route, Details.apply, Details.unapply)

val fromDetails: String = Route.url(mapped, Details(42))
val toDetails: Details = Route.parse(mapped, "/details/42")

val handler: FinchHandler = Route.toHandler(mapped, folder, handler: Details => Unit)

Why invariant: one, two.

[tindzk]

Instead of encoding the operations in MappedRoute, we could also express the routing table as an HList and provide a parse method that can be used as follows:

val details = Root / "details" / Arg[String] / Arg[Int]
val routes  = details :: HNil
Router.parse(routes, uri).map {
  case RouteData(details, s :: i :: HNil) => renderDetails(s, i)
}

This way, we can retain the current functionality of mapping routes to case classes. Would that be a satisfying solution?

[anatoliykmetyuk]

Instead of encoding the operations in MappedRoute, we could also express the routing table as an HList

That's a nice idea: no need to define a class if you can do with HList.

This way, we can retain the current functionality of mapping routes to case classes. Would that be a satisfying solution?

Looks good.