Add Web Workers in Typeorm

[mohammadalikassem]

Feature Description

When dealing with huge entities with lots of one to many and many to many joins, the typeorm hydration process takes a painful amount of time. This has been highlighted in multiple issues . The problem in the nodeJs world is that the whole event loop, async operations is that for processing intensive tasks (not waiting for io like DB), any code executed blocks the main thread until the code is done. This is critical in a server application and may affect all the APIs of said server.

The Solution

Use Web Workers to transform entities, Either in orm config or somewhere else, devs can define the resources for worker thread pools if they want to enable it. The pools can be defined and bound globally or maybe even directly bound to a connection and maybe giving the option to have multiple pools for different operations. In my application, the main bottleneck seems to be transform method in RawSqlResultsToEntityTransformer and offloading this task to another worker unblocks the main event loop and may have a performance benefit on high loads.

Considered Alternatives

My main issue is for reads to as a solution, I considered writing a rust package to aggregate the raw results from the raw entities using neon bindings. I also considered having a separate server just to do the mapping and return the results but I would assume that inter process communication should be a lot cheaper (resources wise) than going thru the whole networking and load balancing hoop.

Another Alternative I had in mind is to aggregate entities in the database. I use postgres so writing queries to aggregate data should be possible. The problem with this approach is that my query takes a lot of parameters and filters that are optional so the query is not really static. but still it is still an option that I could consider. And for my specific usecase, this solves the issues I have. Also it may be possible to compile the query builder query to a sql query that aggregates the results at the database level and returns jsonb results as entities.

It's clear that the main problem I am having is in read operations so adding this option even if only in read operations (separate ger many, get many and count ) functions helps.

Additional Context

My use case is especially caused by having a big database with huge tables (column counts) with multiple layers of many many relations.

Relevant Database Driver(s)

• [X] aurora-mysql
• [X] aurora-postgres
• [X] better-sqlite3
• [X] cockroachdb
• [X] cordova
• [X] expo
• [X] mongodb
• [X] mysql
• [X] nativescript
• [X] oracle
• [X] postgres
• [X] react-native
• [X] sap
• [X] spanner
• [X] sqlite
• [X] sqlite-abstract
• [X] sqljs
• [X] sqlserver

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, but I don't know how to start. I would need guidance.

[richardliebmann]

+1

[ertl]

+1