TypeError: populatorMap[pgType.kind] is not a function

[magick93]

Hello

I'm getting the following error when I try to run kanel:

npx kanel -d "postgresql://postgres:postgres@localhost:54322/postgres" -o ./src/lib/zod
Kanel
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0% | ETA: 0s | 0/103TypeError: populatorMap[pgType.kind] is not a function
    at /home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-pg-schema/build/extractSchemas.js:158:91
    at step (/home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-pg-schema/build/extractSchemas.js:67:23)
    at Object.next (/home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-pg-schema/build/extractSchemas.js:48:53)
    at /home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-pg-schema/build/extractSchemas.js:42:71
    at new Promise (<anonymous>)
    at __awaiter (/home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-pg-schema/build/extractSchemas.js:38:12)
    at /home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-pg-schema/build/extractSchemas.js:153:100
    at Array.map (<anonymous>)
    at /home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-pg-schema/build/extractSchemas.js:153:69
    at step (/home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-pg-schema/build/extractSchemas.js:67:23)

This is using version 3.4.0.

I did have to rename .kanelrc.js to .kanelrc.cjs - I'm not sure if that is a cause, or is relevant.

[kristiandupont]

Hi @magick93, did this happen after an upgrade (I just made some changes today), or is this the first time you are running Kanel?

[magick93]

Hi @kristiandupont

This is a fresh install, and my first time using kanel.

I tried downgrading to v3.3.1, however I got the same error.

[magick93]

I did notice that, even though I've downgraded, the error is still mentioning extract-pg-schema: 4.2.1 - which has only recently been released.

[kristiandupont]

I see. Well, it's a bit hard to debug without your database. Can you try to insert a console.log statement above the error (in the node_modules/extract-pg-schema/build/extractSchemas.js directly? (Or is that not possible with pnpm, which I haven't tried using?)

[magick93]

Good idea.

I made the following modification:

console.log((pgType.name));

                                    try {
                                        return [4 /*yield*/, populatorMap[pgType.kind](db, pgType)];
                                    } catch (error) {
                                        console.log(error);
                                    }

Error parsing view definition for "decrypted_sea
TypeError: Cannot read properties of undefined (reading 'schemaName')
    at /home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-pg-sch3
    at step (/home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/extract-)
    at Object.next (/home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/e)
    at fulfilled (/home/anton/git/yestech/legalease/node_modules/.pnpm/extract-pg-schema@4.2.1/node_modules/ext)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

See https://gist.github.com/magick93/e2dd7499b0868dcea49c58cb42563252

It seems everything in the public schema is fine, but schemas created by extensions, such as Supabase Vault, and Supabase FDW is causing problems.

Is it possible to specific a schema, or exclude schemas?

[kristiandupont]

I see. That is interesting, it should still be fixed. But yes, you can specify which schemas to include with the schemas config key.

[magick93]

I'm not sure if this is related. I tried adding schemas : ['public'], to the config, then it errored in the below section. Specifically this seemed to be when there was a reference to a non-public schema:

reference {
  schemaName: 'auth',
  tableName: 'users',
  columnName: 'id',
  onUpdate: 'NO ACTION',
  onDelete: 'CASCADE',
  name: 'profiles_id_fkey'
}

// 2) If there are references, resolve the type from the targets
    if (c.references && c.references.length > 0) {
        const referencedTypes = c.references.map((reference) => {
            console.log('reference', reference);
            let target = config.schemas[reference.schemaName].tables.find((t) => t.name === reference.tableName);
            if (!target) {
                target = config.schemas[reference.schemaName].views.find((v) => v.name === reference.table);
            }
            if (!target) {
                target = config.schemas[reference.schemaName].materializedViews.find((v) => v.name === reference.tableName);
            }
            if (!target) {
                console.warn('Could not resolve reference', reference);
                return 'unknown';
            }
            const column = target.columns.find((c) => c.name === reference.columnName);
            if (column) {
                return resolveType(column, target, config);
            }
            else {
                console.warn('Could not resolve reference', reference);
                return 'unknown';
            }
        });

[kristiandupont]

That is unrelated. Is the non-public schema the one causing errors before? I don't think I've tried referencing schemas that aren't included, though that should probably be an option..

[kristiandupont]

I am sorry I dropped the ball on this. Did you ever figure it out? It seems that extract-pg-schema returns an item where kind is null even though it should be well-defined. I haven't been able to reproduce the scenario here..

[magick93]

Hi @kristiandupont Thanks for the followup.

No, I didnt manage to resolve this.

[kristiandupont]

It appears that everything in the stripe schema has null kinds. Are they regular tables?

[magick93]

These are foreign tables.

[kristiandupont]

That explains it. So, would you like types out of those, or would you like them ignored?

[magick93]

It's been a while since I looked into this, but from memory this scheme wasn't used.

I thought the issue was extension types and their respective installation scheme.

would you like types out of those, or would you like them ignored?

Ignoring is probably easiest and would be fine for me.

[magick93]

Hi @kristiandupont

I'm not sure if this is related.

When I specify to use schemas: ['public'] I get the TypeError: populatorMap[pgType.kind] is not a function which I think we deduced was due to the use of foreign tables.

However, when I a schema that is not referencing any foreign tables, I get the following:

TypeError: Cannot read properties of undefined (reading 'tables')
    at /home/anton/git/yestech/freedom/node_modules/.pnpm/kanel@3.8.2/node_modules/kanel/build/generators/resolveType.js:37:63
    at Array.map (<anonymous>)
    at resolveType (/home/anton/git/yestech/freedom/node_modules/.pnpm/kanel@3.8.2/node_modules/kanel/build/generators/resolveType.js:36:46)
    at /home/anton/git/yestech/freedom/node_modules/.pnpm/kanel@3.8.2/node_modules/kanel/build/generators/generateProperties.js:56:61
    at Array.map (<anonymous>)
    at generateProperties (/home/anton/git/yestech/freedom/node_modules/.pnpm/kanel@3.8.2/node_modules/kanel/build/generators/generateProperties.js:38:10)
    at /home/anton/git/yestech/freedom/node_modules/.pnpm/kanel@3.8.2/node_modules/kanel/build/generators/makeCompositeGenerator.js:24:65
    at Array.map (<anonymous>)
    at /home/anton/git/yestech/freedom/node_modules/.pnpm/kanel@3.8.2/node_modules/kanel/build/generators/makeCompositeGenerator.js:61:46
    at /home/anton/git/yestech/freedom/node_modules/.pnpm/kanel@3.8.2/node_modules/kanel/build/processDatabase.js:56:22

[magick93]

Ok, I've got a bit further in understanding the problem. With the following:

return [4 /*yield*/, Promise.all(typesToExtract.map(function (pgType) { return __awaiter(_this, void 0, void 0, function () {
    var result, error_1;
    return __generator(this, function (_a) {
        switch (_a.label) {
            case 0:
                _a.trys.push([0, 2, , 3]);
                return [4 /*yield*/, populatorMap[pgType.kind](db, pgType)];
            case 1:
                result = _a.sent();
                (_a = options === null || options === void 0 ? void 0 : options.onProgress) === null || _a === void 0 ? void 0 : _a.call(options);
                return [2 /*return*/, result];
            case 2:
                error_1 = _a.sent();
                console.error(`Error processing type ${pgType.kind}:`, error_1);
                console.error('pgType:', pgType);
                // You can add more logging or error handling here as needed
                return [3 /*break*/, 3];
            case 3: return [2 /*return*/];
        }
    });
}); }))];

I get:

pgType: {
  name: 'payment_intents',
  schemaName: 'public',
  kind: null,
  comment: null
}

The only references I can find to this are:

create foreign table stripe.payment_intents (
  id text,
  customer text,
  amount bigint,
  currency text,
  payment_method text,
  created timestamp,
  attrs jsonb
)
  server stripe_server
  options (
    object 'payment_intents'
  );

[kristiandupont]

Right, so foreign tables are not yet supported. I do mean to add support for them, but I can't guarantee when. If you can live with it for now, I think you should filter them out.

[magick93]

Yes, no problem.