Version 2.0 (June-August)

[LoicPoullain]

Version 2

Updated on June 19, 2020.

This issue provides an overview of the next version of Foal TS. It will be released between June and August 2020.

The purpose of this release is not to add many new features, which are regularly added in minor versions, but to fix issues and hacks that require some break changes. It will also remove unused and deprecated components and make parts of the framework simpler and more intuitive.

Changes & New Features

:white_check_mark: = Implemented

1. Developper experience (CLI) :white_check_mark:

Migrations and shell scripts are not developper friendly when developping (#494). There are too many CLI commands and it is not intuitive to know which one to use and in which order.

In version 2, the files tsconfig.migrations.json and tsconfig.scripts.json will be removed and the CLI commands will be reduced and simplified.

More details here: #684.

Examples

Build, make and run migrations

# Version 1
npm run build:app
npm run migration:generate -- -n my_migration
npm run build:migrations
npm run migration:run

# Version 2
npm run makemigrations
npm run migrations

Build migrations, scripts and the app

# Version 1
npm run build:app
npm run build:scripts
npm run build:migrations

# Version 2
npm run build

2. Authentication with sessions :white_check_mark:

Authentication with sessions is complicated and requires a lot of code. Current implementation also prevents the framework from adding new features (#525, #521, #510).

In version 2, the authentication system with sessions will be greatly simplified.

See #799.

3. Schema references in validation hooks :white_check_mark:

It is not possible to use references (especially OpenAPI references) in validation hooks.

In version 2, this will be fixed.

More details here: #552.

Example

const productSchema = {
  // ...
}

@ApiDefineSchema('product', productSchema)
@ValidateBody({
  $ref: '#/components/schemas/product'
})

4. Service initialization :white_check_mark:

Services can be initialized using their boot method. For this to work, we currently need to call ServicesManager.boot() in src/index.ts.

In version 2, this feature will be enabled by default. No need to call ServicesManager.boot() anymore.

5. Accessing file metadata during uploading :white_check_mark:

When uploading a file, we have only access to its content (#673).

In version 2, we will also have access to its file size, mime type and original file name.

6. Safer configuration :white_check_mark:

Current Config.get function has several faults (#496).

In version 2, it will be replaced with the new Config.get2 added in v1.7.

// Version 1 
const port = Config.get<number>('settings.port', 3000);
const port = Config.get<number>('settings.port');

// Version 2
const port = Config.get('settings.port', 'number', 3000);
const port = Config.getOrThrow('settings.port', 'number')

7. Improve the naming of JWT settings :white_check_mark:

Version 1: settings.jwt.secretOrPublicKey

Version 2: settings.jwt.secret, settings.jwt.publicKey

8. Remove support of Mongoose :white_check_mark:

Mongoose brought many problems in the past while maintaining the framework. It takes time to maintain it and it is also not typed and does not seem to be used by FoalTS users: https://www.npmjs.com/package/@foal/mongoose.

FoalTS will no longer provide Mongoose tools starting from v2 (i.e. CLI code generation and the fetchUser function). Users will still be able to use it "by hand" if they wish.

9. Improve the configuration system :white_check_mark:

See #805, #497.

10. Simplify the management of custom errors thrown in controllers and hooks :white_check_mark:

See https://github.com/FoalTS/foal/issues/638#issuecomment-673915802.

Cleanup :white_check_mark:

• Remove the outdated and useless security headers.
• Remove the PermissionDenied and ObjectDoesNotExist which are unused and do not appear in the documentation.
• Remove the object ValidationError and the function validate which are unused and do not bring value to the framework.
• Remove the deprecated function createHttpResponseFile and use Disk.createHttpResponseFile instead. Once done, remove the package mime from @foal/core dependencies.
• Remove the package @foal/formidable and use the @ValidateMultipartFormDataBody hook instead.
• Remove the deprecated legacy option of the functions hashPassword and verifyPassword. This option was used in very old versions of Foal when no one was using the framework.
• Remove the deprecated package @foal/ejs and the possibility to create FoalTS template engines. Only Express templates engines will be allowed (Twig, EJS, pug, etc).
• Drop the support of Node 8 and run tests on Node 12 and 14. Version 8 of Node is not maintained anymore. Uninstall pump in @foal/core, @foal/storage and @foal/aws-s3 and use the pipeline function from the standard library (added in Node 10).
• Remove the unused command foal g sub-app.
• Check if password hashing needs an update (OWASP).

• Make createApp return a promise and remove createAndInitApp. The function createAndInitApp was introduced because we needed to return a promise.

  // Before
  const app = createApp();
  const app = await createAndInitApp();
  
  // After
  const app = await createApp();

• Only allow one way to pass a custom express instance to createApp.

  // Before
  const expressInstance = express();
  const app = createApp(expressInstance);
  // OR
  const app = createApp({ expressInstance });
  
  // After
  const expressInstance = express();
  const app = createApp({ expressInstance });

Upgrading Dependencies

• @foal/graphql
  • graphql@15.0

[anonimusprogramus]

Some minor things to consider,

• If a developer using class-validator in his code, he'll have to convert validation into ajv schema when he needs to build a script. An option for class-validator in TS framework sounds right.

• Built-in Hooks have built-in internal response messages. If the app is multi-language type, the 'hacky' solution is with a translate post-hooks. What if FoalTS provides a way that developer can feed a string enums of error/warning responses?

That's all so far.

[anonimusprogramus]

Just come into my mind, it would be nice to have FoalTS shown in realword.

[FredericLatour]

Maybe considering migrating from typeorm to mikro-orm (but any other orm would be a better choice than typeorm if you ask me).

The documentation of typeorm is lacking which is problematic because issues are not answered anymore. If you ask for some clarification, it will mostly never be answered.

As an example, I had to dig hard to discover that @CreateDateColumn and @UpdateDateColumn were relying on Mysql CURRENT_TIMESTAMP to get their initial values but on the other hand it does not rely on MySql to update @UpdateDateColumn value. Not only I'm not sure this is a sensible choice but the documentation is completely unhelpful (it says that it deals with those columns but it's not mentioned that it only works if the schema was generated from scratch or if you update your database definition) and of course you can't get any answer and are on your own to find out.

I also just got bitten by BeforeInsert that is not always firing (https://github.com/typeorm/typeorm/issues/5493).

And there are many other problems/annoyance like this that were reported for months (even with a PR available) and basically you can't expect to get them fixed.

The situation is well summarized by this post:

• Future of TypeORM

Even though I can understand the frustration of the developer, I believe that his approach will make people move away (and this is already the case).

mirko-orm is another typescript based orm. Nothing can prevent that one day the same problem occurs. However at this time, the developer is very responsive. He answers issues. He can be reached on slack and he is actively developing the library.

[anonimusprogramus]

Thanks for the tip, @FredericLatour

[kingdun3284]

After 2.0 release, I suggest to have a basic CRUD controller and service for each model which can be easily established by simply extend a controller and service with a base class. I can provide some of my code.

[FredericLatour]

Additional suggestions:

Regarding the Config approach. I don't think Foalts should rely directly on the "Config" service provided:

1. That's really too bad that using typescript, one don't get intellisense and typesafe for application configuration.
2. It makes it difficult to replace (or remove) the existing Config service you are providing with something else.

I would instead pass an IFoaltsConfig object when creating and initializing the application. And by the way, nothing would prevent anybody to have a Config tree that mimics IFoaltsConfig.

const foaltsConfig = Config.get<IFoaltsConfg>('settings')
const app = await createAndInitApp(AppController, foaltsConfig)

I had a look at the config service and was wondering why you were not consolidating/merging the various config sources (except for process.env) instead of your current approach in the Get method. You could have some strategy where objects are merged instead of being replaced.

That being said, I tend to give up JSON configuration and I use Typescript instead. In most cases, we will want our configuration to be committed so I can't really see any advantage of using Json. Json or .env files make sense for information that we don't want to commit (passwords, etc ..).

[EmilCataranciuc]

Maybe considering migrating from typeorm to mikro-orm (but any other orm would be a better choice than typeorm if you ask me).

The documentation of typeorm is lacking which is problematic because issues are not answered anymore. If you ask for some clarification, it will mostly never be answered.

As an example, I had to dig hard to discover that @CreateDateColumn and @UpdateDateColumn were relying on Mysql CURRENT_TIMESTAMP to get their initial values but on the other hand it does not rely on MySql to update @UpdateDateColumn value. Not only I'm not sure this is a sensible choice but the documentation is completely unhelpful (it says that it deals with those columns but it's not mentioned that it only works if the schema was generated from scratch or if you update your database definition) and of course you can't get any answer and are on your own to find out.

I also just got bitten by BeforeInsert that is not always firing (typeorm/typeorm#5493).

And there are many other problems/annoyance like this that were reported for months (even with a PR available) and basically you can't expect to get them fixed.

The situation is well summarized by this post:

• Future of TypeORM

Even though I can understand the frustration of the developer, I believe that his approach will make people move away (and this is already the case).

mirko-orm is another typescript based orm. Nothing can prevent that one day the same problem occurs. However at this time, the developer is very responsive. He answers issues. He can be reached on slack and he is actively developing the library.

I don't think moving away from TypeORM is a good idea. If it lacks something then it should be improved. That requires involvement. Moving away to another, less popular, ORM is like spiraling in a vicious circle. No mature solution comes out of it. If you find TypeORM lacking support consider to donate for at least of 1 hour of the developer's time. He has at least one mouth to feed.

[FredericLatour]

@EmilCataranciuc

1. TypeORM development is paused and has basically no support. That's a fact.
2. People have donated and participated through Pull Requests that are not well handled anymore. That does not inspire confidence for anybody else to invest time and money in the project.
3. Weither I donate or not will not change anything because it won't be enough for the guy to make a living and the ORM/DB nodejs space is currently too crowdy and fast moving for any project to expect significant incomes.
4. I'm not questioning the guy position. I can understand its concerns but it does not change anything to the current situation.
5. It was anyway just a suggestion.

[LoicPoullain]

Thank you for sharing your thoughts on version 2.

The issue description has been updated.

@anonimusprogramus

• If a developer using class-validator in his code, he'll have to convert validation into ajv schema when he needs to build a script. An option for class-validator in TS framework sounds right.

Yes, having to deal with AJV schemas in shell scripts is not the best when we use class-validator in the app. But an option for class-validator in the framework would make the library as part of the framework which I do not want to go to. We had issues with it in the past (for example with a compiler version changing between two minor releases). Adding a line in the shell scripts may be sufficient for now. Maybe the framework could have in the future its own class-based validation following JSON schema spec.

• Built-in Hooks have built-in internal response messages. If the app is multi-language type, the 'hacky' solution is with a translate post-hooks. What if FoalTS provides a way that developer can feed a string enums of error/warning responses?

The framework cannot implement easily this feature because the message is generated by Ajv, not Foal. I'm not sure this feature would be used by many users of the framework so I'd rather not to add it.

[LoicPoullain]

After 2.0 release, I suggest to have a basic CRUD controller and service for each model which can be easily established by simply extend a controller and service with a base class. I can provide some of my code.

@kingdun3284 could you open an issue for this and provide your code so that we can work on? I would have more in mind to add this code in the documentation or in a generator rather than in the core of the framework. CRUD components can become complex and limiting as the framework or the application grow, which would go against Foal's philosophy: simple and progressive. But having an initial code to provide (via the doc or a generator) to users to create reusable components for building a API REST seems to be a good idea to me 👍 .

[LoicPoullain]

Additional suggestions:

Regarding the Config approach. I don't think Foalts should rely directly on the "Config" service provided:

1. That's really too bad that using typescript, one don't get intellisense and typesafe for application configuration.
2. It makes it difficult to replace (or remove) the existing Config service you are providing with something else.

@FredericLatour could you open an issue on this? Could you also add some examples of code/configuration where you find the Config is limiting/not appropriate? Eventually, can you provide an example where the Config service prevents you from using another configuration class?

I had a look at the config service and was wondering why you were not consolidating/merging the various config sources (except for process.env) instead of your current approach in the Get method. You could have some strategy where objects are merged instead of being replaced.

Hmm, it's been a while. I think that at that time, it was for performance reasons. But I didn't make tests on this so I'm sure it really performs better that way.

That being said, I tend to give up JSON configuration and I use Typescript instead. In most cases, we will want our configuration to be committed so I can't really see any advantage of using Json. Json or .env files make sense for information that we don't want to commit (passwords, etc ..).

So would you like to have a committed TS file in config/ for example? When I created the Config class, here what I had in mind:

• JSON and YAML files in config/ are committed and do not have sensitive information (ex: logging and debug settings)
• .env file contains sensitive information (passwords, etc) and is not committed (related issue: #497)

Let's continue this discussion on a separate issue.

[LoicPoullain]

@EmilCataranciuc @FredericLatour

Thank you for your thoughts on TypeORM.

I don't think this is the right place to talk about everyone's donation policy or TypeORM's business plan. Please let's keep us on focusing on the improvements of FoalTS.

TypeORM will be kept in v2 as default ORM for these reasons:

• All existing documentation, ORM components and generators are based on TypeORM.
• Current FoalTS application running in production use TypeORM. Changing to another ORM would require "a lot of" work for each of them.
• mikro-orm is pretty new with a small community so the maintainer probably has time to respond to each request, issue and PR. It is not sure it would continue the same if the product becomes more popular. To give an example, it took me several hours to respond to all the comments in this issue. We may find ourselves in the same situation as TypeORM in the end.
• TypeORM keeps publishing new versions (https://www.npmjs.com/package/typeorm) so it doesn't look like the project is dead to me. Important and relevant issues and PRs may just be prioritized.

However choosing TypeORM as default ORM does not prevent the framework from being used with another one such as mikro-orm (for example the hooks @TokenRequired and @JWTRequired work with any ORM).

[anonimusprogramus]

@LoicPoullain thank you again, you're good with code and words.

[rizalashidiq97]

Is there any roadmap for built-in support microservice that will be provided in the future ? (like redis,mqtt,queue system)

[anonimusprogramus]

How about translating FoalTS docs, is it possible for v2+?

If there's gonna be guidance, I'm in. Gonna take my part translating into Bahasa Indonesia.

[LoicPoullain]

@rizalashidiq97 Currently there is no such map. But if you see some features that are missing in the framework, feel free to open an issue with some use cases.

[LoicPoullain]

How about translating FoalTS docs, is it possible for v2+?

If there's gonna be guidance, I'm in. Gonna take my part translating into Bahasa Indonesia.

@anonimusprogramus could you open a specific issue for this?

[muco-rolle]

When would we expect Version 2.0?

[LoicPoullain]

@muco-rolle

I’m running late on version 2.

The reason behind this is that some parts of the code took me more time that I was expected. The same goes writing the upgrading guide and updating the documentation.

Where we are now:

• The code is almost ready (need some extra checks on sessions).
• The migration guide is written for the most part (the session section is missing).
• The work on documentation is 30%/50% done I would say.
• Some acceptance test have to be written/updated.

Once done:

• Version 2 will be published in beta.
• The migration guide will be tested.

If everything is ok, v2 will be published in production.

I can’t give a precise date but I would say that the final version will be released in October-November.

[anonimusprogramus]

It is said that

When using MongoDB, there are some features that are not available: - the foal g rest-api command, - and the Groups & Permissions system."

Will v2 still have those limitations?

[LoicPoullain]

When using MongoDB, there are some features that are not available: - the foal g rest-api command, - and the Groups & Permissions system."

Will v2 still have those limitations?

@anonimusprogramus yes, it will. The reason behind this it that groups and permissions logic is based on many relations between the groups, the permissions and the users. This can be managed with relational databases (i.e. SQL databases) but hardly with db such as MongoDB.

The foal g rest-api is hard to maintain in Foal but is kept in place because it seems to be used by the community. Adding the support with MongoDB would add extra work of maintenance, which I prefer to keep on other features. However, I guess that if one is using MongoDB with TypeORM, it would require not much work to make the generated code work with g rest-api with MongoDB. I haven't dug into though.

[anonimusprogramus]

I see, it makes sense.

Thank you.

[LoicPoullain]

Point of progress:

• [x] The code is ready (unless bugs are found during further testing, see below).
• [x] The migration guide is finished.
• [x] The migration guide has been tested with samples.
• [x] Beta version has been published.
• [x] A part of the documentation still needs to be updated and tested (including the tutorials).
• [x] Some more end-to-end tests have to be written/updated.

[LoicPoullain]

Version 2.0.0 currently published as next on npm

[LoicPoullain]

Version 2 is officially released 🎉