What are endpoints for?

[icamys]

I'm starting to implement a few microservices using this wonderful package and I found myself spending a lot of time converting data structures from service layer to endpoint layer, from endpoint layer to transport layer. It's pretty annoying especially when we have a lot of different data structures passed through these layers.

As far as I can see all of these conversions can be made on the transport layer.

In official faq there is very few info about the purpose of the endpoint layer.

So I came here with a question: what are endpoints for? Are there any use cases when this layer can be useful?

Thank you.

[peterbourgon]

Endpoints are the way that Go kit connects specific transports like HTTP to your service, on a method-by-method basis. This gives a couple of advantages. One is that certain types of middleware, like rate limiters, are agnostic to the actual content of the request/response, and so can be implemented "generically" at the endpoint layer. Another is that the same service and set of endpoints can be attached to multiple transports without any additional glue code or specialization in the transport layers, so the same business logic can be served over e.g. HTTP and gRPC simultaneously. But, mostly it exists because weaknesses in Go type system don't allow me to usefully abstract an arbitrary user service to each of the transports.

As far as I can see all of these conversions can be made on the transport layer.

They can be, if you don't care about any of the advantages I listed above, and don't care about more tightly coupling your service to a particular transport.

—

One additional note.

I'm starting to implement a few microservices using this . . . package

I'm not sure your exact motivations or context, but it's worth mentioning that Go kit is not designed for first-draft or prototype-stage microservices. We require you to perform a relatively large amount of "ceremony" up-front, under several assumptions: that you understand your business domain pretty well, that your bounded contexts are well-defined, that all of the services are going to be long-lived and maintained by multiple people, etc. Basically, it's a toolkit for relatively mature organizations, teams, and services. If that doesn't describe your situation, it may make a lot more sense to simply write your application in the Go kit "style" and opt-in to actual Go kit stuff as the need arises.

—

Does this help?

[icamys]

One is that certain types of middleware, like rate limiters, are agnostic to the actual content of the request/response, and so can be implemented "generically" at the endpoint layer.

Agreed. However such a content-agnostic logic could be implemented in some service middleware (perhaps if we treat the endpoint layer as a service middleware it makes sense). For instance in the examples/stringsvc3/main.go the proxyingMiddleware is a service middleware. Or maybe I misunderstand something?

Another is that the same service and set of endpoints can be attached to multiple transports without any additional glue code or specialization in the transport layers, so the same business logic can be served over e.g. HTTP and gRPC simultaneously.

This is also true but only in cases when the transport layer uses endpoint request and response objects and fill them out. Then whatever transport will be added only the endpoint request and response objects will be used to transform incoming requests and responses.

But there's another case when http req and resp objects differ from endpoint objects for some reason (in my case I have a specific http response and also I have to add swagger comments to this response structure in order to generate API docs, I think that all http stuff should stay within transport/http). In such a case I have to convert http response/request into endpoint response/request. After that I take endpoint request object and pass its fields to service func. And at this moment I ask myself a dangerous and insidious question, why can not I just call services from the transport layer? (pretty tempting) I feel that its wrong however only feeling is not enough.

Also I've got some simple questions about endpoints. If you will find some time to kindly answer them I will be very grateful. Frankly speaking it's my first experience with onion-like architecture so perhaps some of these questions may sound naive.

1. Should endpoint layer know about transport layer? (I suppose no, the dependency direction should be from outside to inside)
2. Can an endpoint operate with several service functions? For example: call two service functions and combine the results. Or the combination logic is a part of business logic so it should be hidden inside the service?
3. Can the transport layer know about service models? For example we have a model Pony and we want to return a list of Ponys on a http request. Can we just take the service.Pony and json marshal it for http response?
4. Suppose we have a model that maps to a record in database. We want to return a list of these records via http but with another property names. Obviously at some point there should be a model conversion from service to transport layer. Were exactly should such a conversion occur? Is the endpoint layer responsible for such things or endpoint can simply pass the service model to transport layer and each transport implementation should be responsible for converting service models to its own response?

Thank you for your time.

[peterbourgon]

However such a content-agnostic logic could be implemented in some service middleware

Sure, but the point is with Endpoints and endpoint middleware, the same literal middleware can be used for every method and service. With Services and service middleware, you have to customize middleware for each service interface you implement.

This is also true but only in cases when the transport layer uses endpoint request and response objects and fill them out. Then whatever transport will be added only the endpoint request and response objects will be used to transform incoming requests and responses.

But there's another case when http req and resp objects differ from endpoint objects for some reason (in my case I have a specific http response and also I have to add swagger comments to this response structure in order to generate API docs, I think that all http stuff should stay within transport/http).

Endpoint request and response types should be the same across all transports, and should probably capture the input and output parameters of your service methods, purely in your service's domain language, agnostic to transport concerns like Swagger comments.

In such a case I have to convert http response/request into endpoint response/request. After that I take endpoint request object and pass its fields to service func. And at this moment I ask myself a dangerous and insidious question, why can not I just call services from the transport layer? (pretty tempting) I feel that its wrong however only feeling is not enough.

If you don't care about the value that the endpoint layer provides, you can indeed couple your transport directly to your service in this way.

Should endpoint layer know about transport layer? (I suppose no, the dependency direction should be from outside to inside)

No, dependencies point inward. Endpoint should not know about transport. Service should not know about endpoint or transport.

Can an endpoint operate with several service functions? For example: call two service functions and combine the results. Or the combination logic is a part of business logic so it should be hidden inside the service?

Probably a bad idea. The endpoint layer is meant to be very thin, just translation and simple middleware. If your service interface has a method which is conceptually the union of two other functions, that's business logic and should be implemented in the service layer.

Can the transport layer know about service models? For example we have a model Pony and we want to return a list of Ponys on a http request. Can we just take the service.Pony and json marshal it for http response?

The transport layer is free to import service types, so this is basically fine, though arguably a layer violation. How strict you want to be in separating transport-layer data transfer objects from service-layer domain objects is up to you.

Suppose we have a model that maps to a record in database. We want to return a list of these records via http but with another property names. Obviously at some point there should be a model conversion from service to transport layer. Were exactly should such a conversion occur? Is the endpoint layer responsible for such things or endpoint can simply pass the service model to transport layer and each transport implementation should be responsible for converting service models to its own response?

The endpoint layer is the wrong place.

The question is more conceptual: should your service interface return types that closely mirror the database structure, or should it return types that are closer to user expectation? My instinct is that database details shouldn't leak outside of the service boundary, and prefer the second, but without seeing an example it's hard to say more.

With that said, the transport layer is always going to have to do some interpretation and translation of the responses it receives before sending them to the client (and vice versa!). Some errors should map to e.g. 404 Not Found, others to e.g. 201 Accepted, etc. Because of the inward-facing dependency rule, anything you have to do related to e.g. HTTP belongs in the HTTP transport layer.

That's the rule, but we sometimes cheat. For example, the service layer may define its own error types that have a StatusCode() int method, which the transport layer can type-assert against and use to send a response code, rather than maintaining its own e.g. error-to-code mapping table. This is technically a layer violation but it might be a good solution for a given situation.