node-apps: TypeORM not comparable with full WebPack

[xmlking]

I am exploring new node-apps builder with NestJS framework. so far I have success with ng test api, ng test api-e2e ng lint api etc. but facing an issue with ng serve api due to TypeORM runtime expects some entity models available outside webpack bundle.

TypeORM runtime expects entities source available loosely at a preconfigured location e.g., ./**/**.entity{.ts,.js} When we run ng serve api , with TYPEORM_ENTITIES=./dist/**/**.entity.js env variable, where I pre-generated JS entities with npm run api:prestart:prod, I am getting following error:

[ExceptionHandler] No repository for "Notification" was found. Looks like this entity is not registered in current "default" connection? 

My ask is, either we have to use ts-node for ng serve instead of webpack, or allowing users to exclude some files form including in webpack bundle, and compile them loosely in /dist

sample repo to reproduce: https://github.com/xmlking/ngx-starter-kit

[FrozenPandaz]

Sorry, I tried my best to understand the environment you are working in so I might not be correct.

I was able to start a typeorm connection with mysql after reconfiguring it with my environment.

The error indicates that there was a problem with locating the entities. You might want to make sure that the path for the entities in the environment variable is an absolute path. Also, you might want to change /**/**.entity.js to /**/*.entity.js.

Let me know if this helps you?

Those entity files are generated and available in the dist. If you're using relative paths.. you should be able to use a value of './**/*.entity.js'. The cwd of the process will be the outputPath of the build.. which defaults to dist/apps/appName.

[xmlking]

I tried TYPEORM_ENTITIES=/full_path_to/ngx-starter-kit/dist/apps/api/**/*.entity.js but still facing same error, will you be able to share sample setup project? or can you see any issues with my repo?

query: COMMIT
[Nest] 55876   - 10/9/2018, 5:29:59 PM   [InstanceLoader] TypeOrmCoreModule dependencies initialized +509ms
[Nest] 55876   - 10/9/2018, 5:29:59 PM   [ExceptionHandler] No repository for "Notification" was found. Looks like this entity is not registered in current "default" connection? +1ms
RepositoryNotFoundError: No repository for "Notification" was found. Looks like this entity is not registered in current "default" connection?

[xmlking]

some one saying glob pattern will not work for typeorm when used with webpack! https://github.com/nestjs/nest/issues/755#issuecomment-394073763

[xmlking]

workaround: declare entities directly instead of glob pattern ['.//.entity{.ts,.js}']

        entities: [Notification, User],

Ref: https://github.com/xmlking/ngx-starter-kit/blob/develop/apps/api/src/core/core.module.ts

[tomastrajan]

There is a possible workaround using webpack context API.

[creadicted]

workaround: declare entities directly instead of glob pattern ['.//.entity{.ts,.js}']

        entities: [Notification, User],

Ref: https://github.com/xmlking/ngx-starter-kit/blob/develop/apps/api/src/core/core.module.ts

This will not work if you plan to use migrations with typeorm.

[ghost]

Workaround for migrations: Changed tsconfig module to commonjs. Generation of migrations works only with commonjs or umd. Creating of ormConfig.json inside the root directory

Import

[creadicted]

@deimosOmegaChan that works. The commonjs was the problem here. Thank you! This works great with migration files that are based on sql. Do you have experience with seedings that reference the entities?

export class SeedDefaultGroups1524199022084 implements MigrationInterface {
  public async up(queryRunner: QueryRunner): Promise<any> {
    const gUser = await queryRunner.manager.getRepository<Group>(Group).save(
      plainToClass(Group, {
        name: 'user',
        title: 'User'
      })
    );
}}

One method that comes to mind is exporting the whole project in to .js files:

"build:ball-files": "rimraf dist && tsc -p ./apps/api/tsconfig.app.json"

then convert the tsconfig paths to relative paths

tscpaths -p tsconfig.json -s ./ -o ./dist/

and build the applicatin via

ng serve api

to overwrite the main.js file. Now the entities and migration files are there and could be read on application startup.

I am shure this is not good pipeline to apply seeds to the application....

[ghost]

No, I don't have any experience with seedings that reference the entities.

[ghost]

Solution by using webpackConfig Options from builder: @nrwl/builders:node-build With this solution: TypeOrmModule.forRoot() works Need to use two ormconfig files: One for cli, the other one for production ormconfig.json

{
  "type": "sqlite",
  "synchronize": false,
  "logging": true,
  "migrationsRun": true,
  "database": "database/db.sqlite",
  "entities": ["dist/apps/backend/app/database/entity/*.js"],
  "migrations": ["dist/apps/backend/app/database/migration/*.js"],
  "subscribers": ["dist/apps/backend/app/database/subscriber/*.js"]
}

ormconfigdev.json

{
  "type": "sqlite",
  "synchronize": false,
  "logging": true,
  "migrationsRun": true,
  "database": "database/db.sqlite",
  "entities": ["apps/backend/src/app/database/entity/*.ts"],
  "migrations": ["apps/backend/src/app/database/migration/*.ts"],
  "subscribers": ["apps/backend/src/app/database/subscriber/*.ts"],
  "cli": {
    "entitiesDir": "apps/backend/src/app/database/entity",
    "migrationsDir": "apps/backend/src/app/database/migration",
    "subscribersDir": "apps/backend/src/app/database/subscriber"
  }
}

webpackConfig

const glob = require('glob');
const path = require('path');

const ormConfig = require('./ormconfigdev.json')

module.exports = function (webpackConfig, context) {
  const parentDirectory = path.dirname(context.options.sourceRoot).split(path.sep).pop()
  // clean base directory for dist
  const baseDirectory = path.resolve(__dirname, 'dist/' + parentDirectory);
  // generate additionalEntries
  const additionalEntries = {
    ...getAdditionalEntries(ormConfig.entities[0]),
    ...getAdditionalEntries(ormConfig.migrations[0]),
    ...getAdditionalEntries(ormConfig.subscribers[0])
  }

  // merge entries
  webpackConfig.entry = {
    ...webpackConfig.entry,
    ...additionalEntries
  }

  //output
  webpackConfig.output = {
    path: baseDirectory,
    filename: '[name].js',
    libraryTarget: 'commonjs',
  }

  return webpackConfig
}

function getAdditionalEntries(globPattern) {
  const relativePaths = glob.sync(globPattern, {
    absolute: false
  });
  const additionalEntries = {};
  for (let index = 0; index < relativePaths.length; index++) {
    const relativePath = relativePaths[index]
    const absolutePath = glob.sync(relativePath, {
      absolute: true
    });
    const key = relativePath.split('/src/')[1].split('.').slice(0, -1).join('.');
    additionalEntries[key] = absolutePath;
  }
  return additionalEntries;
}

How is this solution?

[gperdomor]

same problem with nestjs-config... https://github.com/nrwl/nx/issues/1430

[sasos90]

@deimosOmegaChan it properly transpiles the files, and it works, until I want to use the entity in a module, where I have to import the entities like TypeOrmModule.forFeature([..... So after using forFeature, I am gettting this "repository not found for..." error. Any ideas on that? Anyone else has this problem too?

[dotdevio]

+1

[ghost]

I did today some tests to find out how to solve this problem. At my solution I combined with the build command and nodemon to run the main.js file. I published it to the repo deimosOmegaChan/nrwl-nest-typeorm-example

[uRTLy]

the real problem is nx when using ng serve puts entire nestjs project into single main.js file(look at dist/apps/app_name) and thats why typeorm cant find entities even the glob you've provided is correct.

questions is how do you use ng/nx scripts to build nest during dev and production without bundling it to a single file ?

@deimosOmegaChan even though it may be working thats not correct solution It is supposed to work out of the box. If I wanted to rewrite Nestjs's already working scripts and mimic poorly their cli / starter project I would just go with lerna

[johannesschobel]

Dear everyone in this thread,

i managed to solve the migrations issue in my nrwl/nx application. This is, what i did: 1) (optional!) install scripty (https://github.com/testdouble/scripty), which lets you execute shell-scripts from npm; npm install --save-dev scripty. While this step is optional, it makes your life easier

2) add a few npm scripts to your package.json file to easily trigger the commands:

{
  // ...
  scripts : {
    // ...
    "migrate:create": "scripty",
    "migrate:run": "scripty"
  }
}

Note that those npm scripts just call the previous mentioned scripty library.

3) Create the proper scripty shell scripts: in your root folder (next to the package.json file) create a scripts folder. This folder is "crawled" by scripty when invoking a npm script. For example, if your command is called migrate:create (see step 2), you need to create a file in scripts/migrate/create.sh.

scripts/migrate/create.sh

#!/usr/bin/env sh

cd ./apps/"$1"/src
npx typeorm migration:create -n "$2"

scripts/migrate/run.sh

#!/usr/bin/env sh

cd ./apps/"$1"/src
npx tsc database/migrations/src/*.ts --outDir database/migrations/generated
npx typeorm migration:run

4) add an ormconfig.js file into the root directory of your application that works with a database. As your mono-repository may contain multiple apps that interact with databases, you may want to have different ormconfig.js files.

apps/api/src/ormconfig.js

module.exports = {
  type: 'postgres',
  host: 'localhost',
  port: 5432,
  username: 'databaseuser',
  password: 'databasepassword',
  database: 'databasename',
  migrationsTableName: 'migrations',
  migrations: ['database/migrations/generated/*.js'],
  cli: {
    migrationsDir: 'database/migrations/src',
  },
  synchronize: false,
};

5) To create a new migration, call

# npm run migrate:create PROJECTNAME MIGRATIONNAME
npm run migrate:create api CreateUsersTable

This will create a new file apps/api/src/database/migrations/src/123456789-CreateUsersTable.ts. You can now fill this migration with life!

6) To run (i.e., execute) all migrations and replicate them to the database, you can call

# npm run migrate:run PROJECTNAME
npm run migrate:run api

This command will first transpile all existing apps/api/src/database/migrations/src/*.ts into javascript and store them in apps/api/src/database/migrations/generated. The latter are then executed by typeorm.

I hope this solution works for you guys!

[johannesschobel]

If you use VSCode, you can create a .vscode/tasks.json file in your project (top level) and add the following content there:

./.vscode/tasks.json

{
  // See https://go.microsoft.com/fwlink/?LinkId=733558
  // for the documentation about the tasks.json format
  "version": "2.0.0",
  "tasks": [
    {
      "label": "migration:create",
      "type": "shell",
      "command": "npm run migrate:create ${input:availableApps} -n ${input:userInput}"
    },
    {
      "label": "migration:run",
      "type": "shell",
      "command": "npm run migrate:run ${input:availableApps}"
    }
  ],
  "inputs": [
    {
      "type": "promptString",
      "id": "userInput",
      "description": "Enter a Value"
    },
    {
      "type": "pickString",
      "id": "availableApps",
      "description": "Select a nrwl/nx app to debug?",
      "options": ["appA", "appB", "appC"]
    }
  ]
}

Now you can simply use the VSCode Task Runner (Press F1, Select "Task: Run" and then select the defined Task). This will show you some nice GUI elements to select / input the proper information. Please note that you need to adapt the available Options in the availableApps input!

[FrozenPandaz]

Hi, sorry about this.

This was mislabeled as stale. We are testing ways to mark not reproducible issues as stale so that we can focus on actionable items but our initial experiment was too broad and unintentionally labeled this issue as stale.

[boskiv]

I've solved it with next content in package.json

"migration:create": "ts-node -O '{\"module\": \"commonjs\"}' node_modules/typeorm/cli.js migration:create",
"migration:generate": "ts-node -O '{\"module\": \"commonjs\"}' node_modules/typeorm/cli.js migration:generate",
"migration:show": "ts-node -O '{\"module\": \"commonjs\"}' node_modules/typeorm/cli.js migration:show",
"migration:run": "ts-node -O '{\"module\": \"commonjs\"}' node_modules/typeorm/cli.js migration:run -t=false",
"migration:revert": "ts-node -O '{\"module\": \"commonjs\"}' node_modules/typeorm/cli.js migration:revert",
"schema:drop": "ts-node -O '{\"module\": \"commonjs\"}' ./node_modules/typeorm/cli.js schema:drop",
"schema:sync": "ts-node -O '{\"module\": \"commonjs\"}' ./node_modules/typeorm/cli.js schema:sync"

And .env file

TYPEORM_URL=postgres://user:password@localhost/database
TYPEORM_ENTITIES=libs/backend/**/*.entity.ts
TYPEORM_MIGRATIONS=apps/api/src/migrations/*.ts
TYPEORM_MIGRATIONS_DIR=apps/api/src/migrations

➜ yarn schema:drop  
yarn run v1.22.4
$ ts-node -O '{"module": "commonjs"}' ./node_modules/typeorm/cli.js schema:drop
query: START TRANSACTION
query: SELECT 'DROP VIEW IF EXISTS "' || schemaname || '"."' || viewname || '" CASCADE;' as "query" FROM "pg_views" WHERE "schemaname" IN (current_schema()) AND "viewname" NOT IN ('geography_columns', 'geometry_columns', 'raster_columns', 'raster_overviews')
query: SELECT 'DROP TABLE IF EXISTS "' || schemaname || '"."' || tablename || '" CASCADE;' as "query" FROM "pg_tables" WHERE "schemaname" IN (current_schema()) AND "tablename" NOT IN ('spatial_ref_sys')
query: DROP TABLE IF EXISTS "public"."migrations" CASCADE;
query: DROP TABLE IF EXISTS "public"."users" CASCADE;
query: SELECT 'DROP TYPE IF EXISTS "' || n.nspname || '"."' || t.typname || '" CASCADE;' as "query" FROM "pg_type" "t" INNER JOIN "pg_enum" "e" ON "e"."enumtypid" = "t"."oid" INNER JOIN "pg_namespace" "n" ON "n"."oid" = "t"."typnamespace" WHERE "n"."nspname" IN (current_schema()) GROUP BY "n"."nspname", "t"."typname"
query: COMMIT
Database schema has been successfully dropped.
✨  Done in 2.57s.

➜ yarn migration:generate -n Users
yarn run v1.22.4
$ ts-node -O '{"module": "commonjs"}' node_modules/typeorm/cli.js migration:generate -n Users
Migration /Users/i_skiridomov/Projects/nx-koko/apps/api/src/migrations/1592933949014-Users.ts has been generated successfully.
✨  Done in 2.20s.

➜ yarn migration:run
yarn run v1.22.4
$ ts-node -O '{"module": "commonjs"}' node_modules/typeorm/cli.js migration:run -t=false
query: SELECT * FROM "information_schema"."tables" WHERE "table_schema" = current_schema() AND "table_name" = 'migrations'
query: CREATE TABLE "migrations" ("id" SERIAL NOT NULL, "timestamp" bigint NOT NULL, "name" character varying NOT NULL, CONSTRAINT "PK_8c82d7f526340ab734260ea46be" PRIMARY KEY ("id"))
query: SELECT * FROM "migrations" "migrations" ORDER BY "id" DESC
0 migrations are already loaded in the database.
1 migrations were found in the source code.
1 migrations are new migrations that needs to be executed.
query: CREATE TABLE "users" ("id" uuid NOT NULL DEFAULT uuid_generate_v4(), "firstName" character varying NOT NULL, "lastName" character varying NOT NULL, "email" character varying NOT NULL, "password" character varying NOT NULL, "isVerified" boolean NOT NULL DEFAULT false, "resetPasswordToken" character varying DEFAULT null, "createdAt" TIMESTAMP NOT NULL DEFAULT now(), "updatedAt" TIMESTAMP NOT NULL DEFAULT now(), CONSTRAINT "PK_a3ffb1c0c8416b9fc6f907b7433" PRIMARY KEY ("id"))
query: INSERT INTO "migrations"("timestamp", "name") VALUES ($1, $2) -- PARAMETERS: [1592933760663,"Users1592933760663"]
Migration Users1592933760663 has been executed successfully.
✨  Done in 2.59s.

➜ tree libs 
libs
├── api-interfaces
│   ├── README.md
│   ├── jest.config.js
│   ├── src
│   │   ├── index.ts
│   │   └── lib
│   │       └── api-interfaces.ts
│   ├── tsconfig.json
│   ├── tsconfig.lib.json
│   ├── tsconfig.spec.json
│   └── tslint.json
└── backend
    └── users
        ├── README.md
        ├── jest.config.js
        ├── src
        │   ├── index.ts
        │   └── lib
        │       ├── backend-users.controller.spec.ts
        │       ├── backend-users.controller.ts
        │       ├── backend-users.module.ts
        │       ├── backend-users.service.spec.ts
        │       ├── backend-users.service.ts
        │       ├── create-user.dto.ts
        │       └── user.entity.ts
        ├── tsconfig.json
        ├── tsconfig.lib.json
        ├── tsconfig.spec.json
        └── tslint.json

[ghost]

I found how to import the entity in a module with the webpack method. I am currently testing this approach. I inject using the name of entity instead of the class. As example for the entity Project: TypeOrmModule.forFeature(['Project' as any]) @InjectRepository('Project' as any)

[memee]

Another problem may appear if you would like to import some shared code in your entity files like:

// inside user.entity.ts
import { SomeEnum } from "@yourorg/api-interfaces";

I tried with compilerOptions.path but with no success.

[snaerth]

@uRTLy Does anyone know how I can modify my workspace.json to tell Webpack not to bundle into one file

[Gargamil]

For me this has helped: https://github.com/typeorm/typeorm/issues/5458#issuecomment-586771247

[github-actions[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. If we missed this issue please reply to keep it active. Thanks for being a part of the Nx community! 🙏

[gperdomor]

Up

[krekhovetskyi]

One more solution based on @tomastrajan and @boskiv examples.

@memee also it works with if you would like to import some shared code in your entity files.

yarn add --dev tsconfig-paths or npm install --save-dev tsconfig-paths

in root of you project create one more tsconfig, - tsconfig.typeorm.json

{
  "compilerOptions": {
    "baseUrl": ".",
    "module": "commonjs",
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "paths": {
      "@lib/api-interfaces/*": ["libs/api-interfaces/src/lib/*"]
    }
  },
  "ts-node": {
    "require": ["tsconfig-paths/register"]
  }
}

create or update .env file

TYPEORM_CONNECTION=postgres
TYPEORM_URL=postgres://user:password@localhost/database
TYPEORM_SYNCHRONIZE=false
TYPEORM_ENTITIES=apps/api/src/app/**/*.entity.ts
TYPEORM_MIGRATIONS=apps/api/migrations/*.ts
TYPEORM_MIGRATIONS_DIR=apps/api/migrations
TYPEORM_DRIVER_EXTRA='{"ssl":false}'

few more lines to package.json

"scripts": {
    "migration:create": "yarn orm-cli migration:create",
    "migration:generate": "yarn orm-cli migration:generate",
    "migration:show": "yarn orm-cli migration:show",
    "migration:run": "yarn orm-cli migration:run -t=false",
    "migration:revert": "yarn orm-cli migration:revert",
    "schema:drop": "yarn orm-cli schema:drop",
    "schema:sync": "yarn orm-cli schema:sync",
    "orm-cli": "ts-node -P ./tsconfig.typeorm.json node_modules/typeorm/cli.js"
}

typeorm config file apps/api/src/config/typeorm.config.ts

import { TypeOrmModuleOptions } from '@nestjs/typeorm';

const contexts = (require as any).context('../app', true, /\.entity.ts$/);
const entities = contexts
  .keys()
  .map(modulePath => contexts(modulePath))
  .reduce((result, entityModule) => result.concat(Object.keys(entityModule).map(key => entityModule[key])), []);

export const typeOrmConfig: TypeOrmModuleOptions = {
  type: process.env.TYPEORM_CONNECTION,
  url: process.env.TYPEORM_URL,
  synchronize: (process.env.TYPEORM_SYNCHRONIZE === 'true'),
  entities,
  ...JSON.parse(process.env.TYPEORM_DRIVER_EXTRA)
};

apps/api/src/app/app.module.ts

import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { TypeOrmModule } from '@nestjs/typeorm';
import { validate } from '../config/env.validation';
import { typeOrmConfig } from '../config/typeorm.config';
import { MainModule } from './main/main.module';

@Module({
  imports: [
    ConfigModule.forRoot({
      validate,
      cache: true,
      validationOptions: {allowUnknown: false, abortEarly: true}
    }),
    TypeOrmModule.forRoot(typeOrmConfig),
    MainModule
  ]
})
export class AppModule {
}

[github-actions[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. If we missed this issue please reply to keep it active. Thanks for being a part of the Nx community! 🙏

[johannesschobel]

bump

[Chlebamaticon]

Had similar issue, tried the option with require.context but it was undefined for me. I have came up with the following;

declare global {
  const __webpack_modules__: any;
  const __webpack_require__: (moduleId) => any;
}

function isClass(v) {
  return typeof v === 'function' && /^\s*class\s+/.test(v.toString());
}

function accessFunctions(regex: RegExp) {
  const modules = __webpack_modules__;
  const req = __webpack_require__;
  const functions = Object.keys(modules)
    .filter((modulePath) => regex.test(modulePath))
    .map((modulePath) => req(modulePath))
    .reduce(
      (acc, module) => [
        ...acc,
        ...Object.keys(module)
          .map((moduleMemberKey) => module[moduleMemberKey])
          .filter(isClass), // that excludes potential enums from entity files
      ],
      []
    );

  return functions;
}

const typeOrmConfig = {
  entities: accessFunctions(/\.entity\.(ts|js)$/),
};

[itseramin]

There is a possible workaround using webpack context API. ...

I love you <3

[jaysoo]

We are working on generating NestJS apps with esbuild rather than webpack, which will solve the bundling issue with TypeORM or any other packages that need real fs paths.

This isn't an Nx issue, but a Webpack + TypeORM issue (the latter doesn't play well with bundling by design). It is called out on the NestJS docs as well:

Note that webpack won't automatically copy your assets (e.g. graphql files) to the dist folder. Similarly, webpack is not compatible with glob static paths (e.g., the entities property in TypeOrmModule).

I will close this since there is no good solution as long as Webpack/bundling is used. However, for the time being you can use @tomastrajan's require.context solution as outlined here https://github.com/nrwl/nx/issues/803#issuecomment-450642765.

[github-actions[bot]]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.