Evaluate using a application framework for backend services

[bk3c]

Based on feedback from @pbn4 and my own previous experience, I think it's worth looking to see if we want to use a framework on top of Express of Koa to structure the back-end services, much like we use NextJS on the front-end. This wouldn't be required for every service we build (e.g. it doesn't seem to make sense if for take node-oidc-provider off the shelf), but it might provide some useful benefits for services like Listings and Applications that are heavy on business logic.

Since a choice here could have significant implications on the ORM choice, it's probably best we consider this in parallel with #227. Much like the ORM choice, key considerations would be:

• Offloading of plumbing code while still making it easy for us to write and structure business logic.
• Minimal boilerplate code to get straight-forward things done
• Testing and documentation leverage to make it easy for us to keep up on those things
• Good Typescript support

Potential contenders include:

• NestJS -- very robust and widely used, but also relatively heavyweight from my POV
• Ts.ED -- lighter option, but much smaller community and feature set

There are also a bunch of other smaller options, any of which we could entertain if they're compelling.

[pbn4]

Ts.ED

A lightweight and easy to learn typescript framework on top of Express. Integrates nicely with well known express middlewares, provides support for most popular orm - TypeORM.

My demo branch can be found here.

Features:

• TypeORM support
• built-in socket.io support
• Support for dependency injection, both for app lifecycle and request-scoped services.
• framework supports async/await syntax. Controllers support many return types out of the box e.g. Observables, ReadStreams, Buffers etc.
• out of the box middlewares for oauth2, I am not 100% sure but this probably will play nicely with OIDC with little modifications. More info on Passport.js integration can be found here
• tests, documentation explain how to test with dependency injection and how to bootstrap a testing server
• Project structure By default the framework assumes the following project structure:

  ├── src
  │   ├── controllers
  │   ├── services
  │   ├── middlewares
  │   ├── index.ts
  │   └── Server.ts
  └── package.json

which in my opinion is a good thing across multiple projects. Once everyone gets used to it navigation is easier.

• Api versioning: Nice built-in solution for api versioning:

@ServerSettings({
  mount: {
    "/rest/v0": "./controllers/v0/**/*.ts",
    "/rest/v1": "./controllers/v1/**/*.ts"
  }
})

• models validation with ajv, validation error result is customizable and easy to define. This is structured solution for form errors reporting.

• documentation generation with swagger (when Ts.ED models are in use), there could potentially be a problem with that since this is decorator based so either we duplicate models knowledge and add property documentation somewhere else e.g. services/listings and leave the shared domain models clean or we have them coupled in shared models.

Example /listings route:

@Controller("/listings")
export class ListingCtrl {
  @Get()
  @ReturnsArray(ListingModel)
  async findAll(@QueryParams() params: QueryParamsModel): Promise<ListingModel[]> {
    ...
  }
}

requires such model definition to properly generate swagger:

class ListingModel implements Listing {
  @Property()
  acceptingApplicationsAtLeasingAgent: boolean
  ...
  @PropertyType(Address)
  applicationAddress: Address
  ...
}

where Address is also a decorated model definition. We can already see that keeping the domain clean with this approach is problematic because of the duplication.

Notes

• I could not find any way to inline QueryParams validation in the controller route. It leads to creating separate classes for query params if one wants validation:

export class QueryParamsModel {
  @Required()
  jsonpath: string;
}
...
@Get()
async findAll(@QueryParams() params: QueryParamsModel): Promise<any> {

by the way: jsonpath is not a required parameter in our case, it's just a test case scenario.

• I could not find an easy way to derive serializers from models definitions as compared to e.g. NestJS and that will lead to creating our own implementations of those derivation metaclasses or it will introduce a lot of duplication.
• nest-js has a plugin for easier i18n which I could not find for Ts.ED, but this is not a big issue since there are orthogonal packages doing that

[pbn4]

If you have any questions, feel free to ask them. I think that conclusion should be made based on comparison between frameworks not on a single framework alone.

[pbn4]

Also I quickly glanced over loopback.io which is designed for REST api development, it's heavy in terms of conventions used and the learning curve seems pretty steep. Nevertheless It seems to use many established patterns for server side applications and might be worth a deeper investigation. It's also typescript friendly. :)

[software-project]

I did a run with nestjs. I pushed a working example that includes typeorm to: https://github.com/bloom-housing/bloom/tree/nestjs-with-typeorm There is plenty of examples out there, I really like that it's structured, it has cors, helmet and CSRF protection. I had a brief look at TsED and they look quite similar. Both based on express. I like the module part of config a little more in NestJS, plus bigger community, more plugins so I would pick that one. @bencpeters @bk3c @jaredcwhite what do you think?

[jaredcwhite]

@pbn4 @software-project Taking a look at your examples/demos, thanks for pulling that all together.

My initial comment is I would certainly advocate for as simple a coupling as possible between the controller and model layers…for example in the NestJS demo, having a service object sit between the controller and the model seems like too much ceremony for too little gain. For the Te.ED demo, I couldn't quite tell what the controller was doing there…maybe calling our old service? Anyway, this is mainly what I'd be on the lookout for… pseudo-code that looks somewhat like this:

ListingsController {
  index() {
    return Listing.findAll(relations…).map(listing => {
      // do some more stuff
    }
  }
  get(params) {
    return Listing.findOne(id: params[:id], relations…)
  }
}

[jaredcwhite]

Looking through the NestJS docs, I must say I'm not very impressed by the code smell in a lot of the examples. Feels like some hardcore Angular devs discovered the backend and went bananas. 😆 There is no way I'm ever going to get excited writing code that looks like this:

😬

Of the two, Ts.ED seems much more straightforward and streamlined. Still not sure it's worth it vs. just using Express/Koa + TypeORM or whatever.

[software-project]

Having looked at Ts.ED example that @pbn4 prepared I still prefer NestJS. It has much bigger community, I like the structure and it looks cleaner to me. Unless we really want sth simple then maybe we can consider TSOA. It has bigger community then Ts.ED and it looks alright as well.

[jaredcwhite]

The two examples do seem pretty similar, at least at this stage. I like the controller directly calling the DB and utilizing the query parameters for things like JSON Path in the Ts.ED example. @software-project do you think you could work on a variant of the NestJS demo that moves the service object logic into the controller and uses query params? Then I think we'll be in a good place to make an informed decision between the two frameworks.

[software-project]

@jaredcwhite sure, I'll give it a try. Using services should be somewhat optional, but to be fair I'm a fan of thin controllers in Rails. So I actually like this separation. That being said I don't have much experience with them on js side.

[jaredcwhite]

@software-project it's a matter of preference to be sure. I tend to advocate for focusing more on modeling higher-level domain logic (using POROs or in this case POJOs) than using service objects. In this case though it feels pretty legit in the controller context just to load the listings from the DB, run the transformation, and return the output—but if we wanted to encapsulate more of that in the model layer under a single function call, I'm down for that too. (I guess we'd have to decide if that goes in the Listing class or the ListingRepository class. My vote would be in Listing as part of the ActiveRecord pattern.)

[pbn4]

@software-project @jaredcwhite I do not have strong opinion on thin controllers. Seems that common operations might be reused when using thin controllers and delegation to services, but how often we are going to need that? It's also easy to introduce at later stage.

On the other hand I find it useful in a case where we would like to have different protocols like e.g. HTTP, gRPC or some websockets access the same "queries/command" (CQRS). Then the application layer should expose Commands and Queries and thin controllers are a must to keep things DRY.

[bk3c]

Following up after a discussion at today's standup, we're going to move forward with NestJS.