Exception from a finished function: AbortError: Ready check failed

[shietpas]

Describe the bug I am using Google serverless implementation with utilizes the "api" and "process" functions with PubSub handoff between them and Redis Memorystore cache and Cloud SQL (postgres) backend. The "api" function appears to complete successfully, then 4 minutes later an AbortError appears under the same execution id in the logs with:

Exception from a finished function: AbortError: Ready check failed: Connection forcefully ended and command aborted. It might have been processed. It appears as if a resource connection is being lost behind-the-scenes and resulting in an eventual error; possibly related to Redis cache pooling.

@paveltiunov noted that this may be related to the redis pool. A possible workaround may be to set the environment variable CUBEJS_REDIS_POOL_MAX=0; although a complete resolution is still desired.

To Reproduce Steps to reproduce the behavior:

1. Normal cube.js query execution in "production" mode using Redis Memorystore cache in GCP
2. Stackdriver logs capture AbortError events which occur

Expected behavior Exceptions should not be thrown if functions are completing successfully.

Screenshots The screenshot below shows a single execution ID (call) with AbortError entries lagging the function execution end time by 1.5 minutes:

This screenshot shows frequency of error in a DEV environment with relatively low traffic:

Version: "@cubejs-backend/bigquery-driver": "^0.18.13", "@cubejs-backend/postgres-driver": "^0.18.14", "@cubejs-backend/server": "^0.18.14", "@cubejs-backend/serverless": "^0.18.14", "@cubejs-backend/serverless-google": "^0.18.14", "serverless-express": "^2.0.11",

Additional context Platform: Google Cloud Function runtime: nodejs10

[paveltiunov]

@shietpas Hey Scott! Could you please share your options passed to Handlers constructor?

[shietpas]

@paveltiunov is this sufficient: const GoogleHandlers = require("@cubejs-backend/serverless-google"); // appId 1 = a (BigQuery) // appId 2 = b (Cloud SQL Postgres)

//https://cube.dev/docs/@cubejs-backend-server-core#queue-options const queueOptions = { //Timeout and Interval options are in seconds concurrency: 100, //default = 2 continueWaitTimeout: 60, // long polling interval, default = 5s executionTimeout: 120, //default = 600s (10 minutes) orphanedTimeout: 120, //default = 120s heartBeatInterval: 30 //default=30s, Worker heartbeat interval. If 4*heartBeatInterval time passes without reporting, the query gets cancelled. };

const orchestratorOptions = { queryCacheOptions: { refreshKeyRenewalThreshold: 3600, //Time in seconds to cache the result of refreshKey check; default = 120 (2 minutes) backgroundRenew: true, //immediately return values from cache if available without refreshKey check to renew in foreground. Default value before 0.15.0 was true queueOptions }, preAggregationsOptions: { queueOptions } };

const aRepo = new FileRepository(schema/a); const bRepo = new FileRepository(schema/b);

const postgresDriver = new PostgresDriver({ host: process.env.CUBEJS_A_HOST, database: process.env.CUBEJS_A_DB, port: process.env.CUBEJS_A_PORT, user: process.env.CUBEJS_A_USER, password: process.env.CUBEJS_A_PASS });

const bigQueryDriver = new BigQueryDriver({ projectId: process.env.CUBEJS_DB_BQ_PROJECT_ID, keyFilename: process.env.CUBEJS_DB_BQ_KEY_FILE });

module.exports = new GoogleHandlers({

contextToAppId: ({ authInfo: { appId } }) => CUBEJS_APP_${appId},

dbType: ({ authInfo: { appId } }) => { if (appId == 2) return "postgres";

return "bigquery";

},

driverFactory: ({ authInfo: { appId } }) => { if (appId == 2) return postgresDriver;

return bigQueryDriver;

},

repositoryFactory: ({ authInfo: { appId } }) => { if (appId == 2) return aRepo;

return bRepo;

},

orchestratorOptions });

[paveltiunov]

@shietpas Gotcha. It's due to continueWaitTimeout: 60. I believe it exceeds default timeout for GCP.

[shietpas]

I did extend the cloud function timeout to Max of 540 (9 minutes) to accommodate, so the function should not timeout. Prior to changing I did notice errors for long running operations such as building pre-agg table. I also do background refresh to avoid that particular delay.

Sent from my Verizon, Samsung Galaxy smartphone

---

From: Pavel Tiunov notifications@github.com Sent: Friday, April 3, 2020 10:40:14 PM To: cube-js/cube.js cube.js@noreply.github.com Cc: Hietpas, Scott shietpas@SkylineTechnologies.com; Mention mention@noreply.github.com Subject: Re: [cube-js/cube.js] Exception from a finished function: AbortError: Ready check failed (#556)

@shietpashttps://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fshietpas&data=02%7C01%7Cshietpas%40skylinetechnologies.com%7C7477488891ed4261d59408d7d849e313%7C59a2b115db554735a4d096c18208604f%7C0%7C0%7C637215684172389012&sdata=beYOrhW%2FsHXI7RhHKgBMrKuyt4PyoXwZCu0c5FC%2BeS4%3D&reserved=0 Gotcha. It's due to continueWaitTimeout: 60. I believe it exceeds default timeout for GCP.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcube-js%2Fcube.js%2Fissues%2F556%23issuecomment-608966582&data=02%7C01%7Cshietpas%40skylinetechnologies.com%7C7477488891ed4261d59408d7d849e313%7C59a2b115db554735a4d096c18208604f%7C0%7C0%7C637215684172398965&sdata=dyr2%2BjLKIp5dKlLKqoWKI2KPQXFOSx9L8VLdcTrojk4%3D&reserved=0, or unsubscribehttps://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFKQYETE4EFX3GH5DI24JY3RK2TZ5ANCNFSM4LXTGAMQ&data=02%7C01%7Cshietpas%40skylinetechnologies.com%7C7477488891ed4261d59408d7d849e313%7C59a2b115db554735a4d096c18208604f%7C0%7C0%7C637215684172398965&sdata=qDdHrDtHBr0oD88BzatNoHNjV%2FPKq9KJJl2SYJ9i%2BeQ%3D&reserved=0.

[paveltiunov]

@shietpas I believe function timeout and time allowed to wrap up all background jobs are different parameters. Not sure if GCP has an option for later. You can try to remove this continueWaitTimeout to see what happens. Anyway 60 is too much for all the default settings of Cube.js itself and surrounding environment like API gateways.