Failed to migrate db OperationalError: timeout expired

[jaiswalshikha]

Describe the bug

While installing unleash through helm chart, I am getting the error as mentioned below: (I am using rds instance here instead of postgresql pod)

[2023-10-15T18:39:37.873] [ERROR] server-impl.js - Failed to migrate db OperationalError: timeout expired at Timeout._onTimeout (/unleash/node_modules/pg/lib/client.js:106:28) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) { cause: Error: timeout expired at Timeout._onTimeout (/unleash/node_modules/pg/lib/client.js:106:28) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7), isOperational: true } [ERROR] Error: timeout expired at Timeout._onTimeout (/unleash/node_modules/pg/lib/client.js:106:28) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7)

Steps to reproduce the bug

No response

Expected behavior

No response

Logs, error output, etc.

[2023-10-15T18:39:37.873] [ERROR] server-impl.js - Failed to migrate db OperationalError: timeout expired
    at Timeout._onTimeout (/unleash/node_modules/pg/lib/client.js:106:28)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7) {
  cause: Error: timeout expired
      at Timeout._onTimeout (/unleash/node_modules/pg/lib/client.js:106:28)
      at listOnTimeout (node:internal/timers:569:17)
      at process.processTimers (node:internal/timers:512:7),
  isOperational: true
}
[ERROR] Error: timeout expired
    at Timeout._onTimeout (/unleash/node_modules/pg/lib/client.js:106:28)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7)

Screenshots

No response

Additional context

No response

Unleash version

5.4.4

Subscription type

Open source

Hosting type

None

SDK information (language and version)

No response

[sjaanus]

Hello @jaiswalshikha ,

Thank you for reaching out and sharing the issue you're facing when installing Unleash via the Helm chart and using an RDS instance.

It seems there's a timeout issue when Unleash tries to migrate the database schema. Here are a few steps and considerations to help diagnose and potentially resolve the issue:

1. Network Connectivity: Ensure that the Kubernetes cluster has proper connectivity to the RDS instance.

   • You can check by exec'ing into one of your pods and attempting to connect to the RDS instance using a tool like psql or any other PostgreSQL client.

2. Database Credentials: Double-check that the credentials (username, password, and database name) you provided are correct and have the necessary permissions.

3. Logs and Monitoring: Check the RDS logs and any available monitoring metrics. There might be issues like CPU spikes or slow queries which could provide more insights.

4. Helm Chart Configuration: Ensure any overrides you've made in the Helm values are correctly structured. Sometimes, a misconfiguration can lead to unexpected behavior.

After you've checked these, if the issue persists, please provide more detailed logs or configurations (ensuring you mask any sensitive data), so we can take a deeper dive.

Thanks for your patience and collaboration. We'll do our best to assist you in resolving this!

[chiragbhatia-tw]

Hi @sjaanus , Hope you're keeping well.

While we have tested the network connectivity and database is reachable from our cluster, but I don't see any way to allow this encryption in helm charts One thing to be noted is that we are using external postgres endpoint with RDS to connect with unleash

[ERROR] error: no pg_hba.conf entry for host "foo", user "user", database "unleash", no encryption

[jaiswalshikha]

Now I'm getting below error (I'm using RDS instead of default postgresql chart)

[ERROR] error: no pg_hba.conf entry for host "10.x.x.x", user "dbuser", database "unleashdb", no encryption at Parser.parseErrorMessage (/unleash/node_modules/pg-protocol/dist/parser.js:287:98) at Parser.handlePacket (/unleash/node_modules/pg-protocol/dist/parser.js:126:29) at Parser.parse (/unleash/node_modules/pg-protocol/dist/parser.js:39:38) at Socket. (/unleash/node_modules/pg-protocol/dist/index.js:11:42) at Socket.emit (node:events:514:28) at addChunk (node:internal/streams/readable:324:12) at readableAddChunk (node:internal/streams/readable:297:9) at Readable.push (node:internal/streams/readable:234:10) at TCP.onStreamRead (node:internal/stream_base_commons:190:23) internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, column: undefined, dataType: undefined, constraint: undefined, file: 'auth.c', line: '543', routine: 'ClientAuthentication' }

[chriswk]

Hi @jaiswalshikha - In the helm values.yaml there is a

dbConfig:
  ssl

value, if you set this to true, it will force ssl connections.

However, I don't think the node process that runs Unleash will trust your rds connection by default. RDS uses custom certificates, so you will also need to mount the CA certificate from AWS RDS into your container and then tell the node process that it should be trusted by injecting the environment variable NODE_EXTRA_CA_CERTS and giving it the path to the RDS CA root certificate file that you mounted in the previous step.

[tylerbrewer2]

@chriswk I'm running into this issue as well. When I attempted to pass SSL configuration through the Helm chart, I received a parsing error. I think my PR should fix the issue

[jaiswalshikha]

@chriswk I followed your instruction. However, I am now stuck at the error mentioned below:

[2023-10-18T07:00:14.579] [ERROR] server-impl.js - Failed to migrate db error: column "type" referenced in foreign key constraint does not exist at Parser.parseErrorMessage (/unleash/node_modules/pg-protocol/dist/parser.js:287:98) at Parser.handlePacket (/unleash/node_modules/pg-protocol/dist/parser.js:126:29) at Parser.parse (/unleash/node_modules/pg-protocol/dist/parser.js:39:38) at TLSSocket. (/unleash/node_modules/pg-protocol/dist/index.js:11:42) at TLSSocket.emit (node:events:514:28) at addChunk (node:internal/streams/readable:324:12) at readableAddChunk (node:internal/streams/readable:297:9) at Readable.push (node:internal/streams/readable:234:10) at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) { length: 138, severity: 'ERROR', code: '42703', detail: undefined, hint: undefined, position: undefined, internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, column: undefined, dataType: undefined, constraint: undefined, file: 'tablecmds.c', line: '11316', routine: 'transformColumnNameList' } [ERROR] error: column "type" referenced in foreign key constraint does not exist at Parser.parseErrorMessage (/unleash/node_modules/pg-protocol/dist/parser.js:287:98) at Parser.handlePacket (/unleash/node_modules/pg-protocol/dist/parser.js:126:29) at Parser.parse (/unleash/node_modules/pg-protocol/dist/parser.js:39:38) at TLSSocket. (/unleash/node_modules/pg-protocol/dist/index.js:11:42) at TLSSocket.emit (node:events:514:28) at addChunk (node:internal/streams/readable:324:12) at readableAddChunk (node:internal/streams/readable:297:9) at Readable.push (node:internal/streams/readable:234:10) at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)

[sjaanus]

Hey @jaiswalshikha,

This looks odd. Could you please provide the list of migrations that have been applied to your database? You can retrieve this by running:

SELECT * FROM migrations;

[jaiswalshikha]

Here is the output:

id | name | run_on ----+-------------------------------------------------------+------------------------- 1 | /20141020151056-initial-schema | 2023-10-18 06:58:19.696 2 | /20141110144153-add-description-to-features | 2023-10-18 06:58:19.705 3 | /20141117200435-add-parameters-template-to-strategies | 2023-10-18 06:58:19.709 4 | /20141117202209-insert-default-strategy | 2023-10-18 06:58:19.72 5 | /20141118071458-default-strategy-event | 2023-10-18 06:58:19.731 6 | /20141215210141-005-archived-flag-to-features | 2023-10-18 06:58:19.735 7 | /20150210152531-006-rename-eventtype | 2023-10-18 06:58:19.74 8 | /20160618193924-add-strategies-to-features | 2023-10-18 06:58:19.746 9 | /20161027134128-create-metrics | 2023-10-18 06:58:19.751 10 | /20161104074441-create-client-instances | 2023-10-18 06:58:19.758 11 | /20161205203516-create-client-applications | 2023-10-18 06:58:19.763 12 | /20161212101749-better-strategy-parameter-definitions | 2023-10-18 06:58:19.774 13 | /20170211085502-built-in-strategies | 2023-10-18 06:58:19.778 14 | /20170211090541-add-default-strategies | 2023-10-18 06:58:19.785 15 | /20170306233934-timestamp-with-tz | 2023-10-18 06:58:19.79 16 | /20170628205541-add-sdk-version-to-client-instances | 2023-10-18 06:58:19.793 17 | /20190123204125-add-variants-to-features | 2023-10-18 06:58:19.797 18 | /20191023184858-flexible-rollout-strategy | 2023-10-18 06:58:19.803 19 | /20200102184820-create-context-fields | 2023-10-18 06:58:19.81 20 | /20200227202711-settings | 2023-10-18 06:58:19.815 21 | /20200329191251-settings-secret | 2023-10-18 06:58:19.819 22 | /20200416201319-create-users | 2023-10-18 06:58:19.825 23 | /20200429175747-users-settings | 2023-10-18 06:58:19.829 24 | /20200805091409-add-feature-toggle-type | 2023-10-18 06:58:19.836 25 | /20200805094311-add-feature-type-to-features | 2023-10-18 06:58:19.84 26 | /20200806091734-add-stale-flag-to-features | 2023-10-18 06:58:19.843 27 | /20200810200901-add-created-at-to-feature-types | 2023-10-18 06:58:19.847 28 | /20200928194947-add-projects | 2023-10-18 06:58:19.852 29 | /20200928195238-add-project-id-to-features | 2023-10-18 06:58:19.857 30 | /20201216140726-add-last-seen-to-features | 2023-10-18 06:58:19.86 (30 rows)

[sjaanus]

Hey @jaiswalshikha,

For the upcoming migration (31th), SQL being ran is following:

CREATE TABLE IF NOT EXISTS tag_types
(
    name        text PRIMARY KEY NOT NULL,
    description text,
    icon        text,
    created_at  TIMESTAMP WITH TIME ZONE DEFAULT now()
);
CREATE TABLE IF NOT EXISTS tags
(
    type  text not null references tag_types (name) ON DELETE CASCADE,
    value text,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT now(),
    PRIMARY KEY (type, value)
);
CREATE TABLE IF NOT EXISTS feature_tag
(
    feature_name varchar(255) not null references features (name) ON DELETE CASCADE,
    tag_type       text not null,
    tag_value      text not null,
    created_at   TIMESTAMP WITH TIME ZONE DEFAULT now(),
    UNIQUE (feature_name, tag_type, tag_value),
    FOREIGN KEY (tag_type, tag_value) REFERENCES tags(type, value) ON DELETE CASCADE
);

From my observation, there seems to be an issue with the third statement where it's trying to reference the type column for the foreign key. This migration is from 2020 and has worked smoothly for everyone, so the migration itself is unlikely to be the problem.

One potential hiccup could be the state of your database. The statement CREATE TABLE IF NOT EXISTS tags means the table will only be created if it doesn't already exist. Is it possible that the tags table already exists in your database but without the type column? Could you verify the existence of the tags table and its columns in your database?

Let me know how it goes!

[jaiswalshikha]

Okay so I was able to get Unleash up and running. However, when I am setting a context path as / in ingress rule, it is accessible but not when it is set to /unleash.

This works: ` hosts:

• host: paths:
  • path: / pathType: Prefix`

But not this

`hosts:

• host: paths:
  • path: /unleash pathType: Prefix`

My UNLEASH_URL is set to <alb-address>/unleash

When I set the context path to /unleash I get 404 error for <alb-address>/static/style-770afaee.css net::ERR_ABORTED 404 (Not Found) and <alb-address>/static/index-17d74ac9.js net::ERR_ABORTED 404 (Not Found)

[sjaanus]

Hey Jaiswalshikha! 😊 To get everything running smoothly, please set the base URI path to /unleash. This will ensure it resolves to <alb-address>/unleash. You can easily set it using the BASE_URI_PATH environment variable. And if you'd like more info, here are docs for configuring unleash: Configuring unleash Hope this helps!

[jaiswalshikha]

Hi @sjaanus thanks for your response. So I tried setting BASE_URI_PATH in the env variable. But then I am getting below error in pod logs and it goes into CrashLoopBackOff state.

[ERROR] TypeError [ERR_INVALID_URL]: Invalid URL at new NodeError (node:internal/errors:405:5) at new URL (node:internal/url:611:13) at findRootUrl (/unleash/node_modules/unleash-server/dist/lib/openapi/index.js:213:21) at createOpenApiSchema (/unleash/node_modules/unleash-server/dist/lib/openapi/index.js:222:17) at new OpenApiService (/unleash/node_modules/unleash-server/dist/lib/services/openapi-service.js:16:102) at createServices (/unleash/node_modules/unleash-server/dist/lib/services/index.js:161:28) at createApp (/unleash/node_modules/unleash-server/dist/lib/server-impl.js:69:52) at Object.start (/unleash/node_modules/unleash-server/dist/lib/server-impl.js:163:27)

Also, do Unleash helm chart support external secret integration so that I can store my db creds in AWS secret manager?

[chriswk]

Hi @jaiswalshikha - it does not support setting up external secrets as of now. If this is important to you, feel free to make a new issue, or better yet add a PR implementing the support you'd like.

[chriswk]

Closing this issue. the base_path issues is here:

• 99

  and the external secret provider is here

• 100

  Please try to stick to one issue per issue.