Environment variable changes not reflecting in jobs

[alexghattas]

NodeJS 20 - Express with Typescript PostgreSQL: 15.1.0.103 Hosted with Dokku (Heroku OS) Graphile-worker version: 0.16.1

The issue: When I update an environment variable in my remote environments (i.e. staging), the changes are not reflected in the code when it is used by the queue system (i.e. the code is being run by graphile-worker is still using the old env variable values).

Context: We use graphile-worker to download and merge multiple files for our users, we updated the S3 region to use a new bucket, and these details are stored in an env variable. When I download a file not using the queue, it works as expected. When I go to download the files that uses the queue system, it is still referencing the old env variables.

I have redeployed my app a few times, which restarts the graphile-worker, but the issue persists.

Has anyone experienced this before?

Thank you!

[benjie]

Hi Alex; Worker has no caching of environmental variables, so this is not a bug in worker.

Are you using a bundler (webpack or similar) which might be hardcoding the envvars at the time of build into your bundle?

Alternatively, are you using a runner like pm2 which caches the environmental variables that were used when the process was first started, and doesn't reflect changes?

Alternatively, are you actually not restarting worker even though you think you are?

[alexghattas]

Hi Benjie, thanks for getting back to me!

We do not use a bundler on our backend. The env vars are updated throughout our backend application, but they are not being updated when the job runs, it is for whatever reason still using the old env variables.

We do not use anything like pm2 that caches env variables.

The server does seem to restart, here is the log from our server after it gets restarted:

2024-04-17T14:16:50.152743983Z app[web.1]: > visto-backend@1.3.9 start
2024-04-17T14:16:50.152749855Z app[web.1]: > node ./build/src/index.js
2024-04-17T14:16:50.152753982Z app[web.1]:
2024-04-17T14:17:00.326727720Z app[web.1]: 2024-04-17T14:17:00.322Z   info:  Failed to read crontab file '/app/crontab'; cron is disabled
2024-04-17T14:17:00.329925506Z app[web.1]: 2024-04-17T14:17:00.329Z   info:  Registering termination signal handlers (SIGUSR2, SIGINT, SIGTERM, SIGHUP, SIGABRT)
2024-04-17T14:17:00.331877008Z app[web.1]: 2024-04-17T14:17:00.331Z   info:  Spawned
2024-04-17T14:17:00.333220213Z app[web.1]: 2024-04-17T14:17:00.333Z   info:  Graphile Worker started
2024-04-17T14:17:00.345952965Z app[web.1]: 2024-04-17T14:17:00.345Z    info: 🚀 Server ready at 0.0.0.0:5000/graphql {
2024-04-17T14:17:00.345968880Z app[web.1]:     "ANTIVIRUS_DISABLED": false,
2024-04-17T14:17:00.345975552Z app[web.1]:     "ENCRYPTION_ENABLED": true,
2024-04-17T14:17:00.345979935Z app[web.1]:     "ENV": "stg",
2024-04-17T14:17:00.345984174Z app[web.1]:     "LogLevel": "silly"
2024-04-17T14:17:00.345988560Z app[web.1]: }
2024-04-17T14:17:00.366732317Z app[web.1]: 2024-04-17T14:17:00.366Z   info:  Worker connected and looking for jobs... (task names: 'documents.export_application_documents')

and here is what the function looks like that is being called:

export const startGraphileWorker = async () => {
  if (runner) {
    logger.warn("Graphile Worker is already running.");
    return;
  }

  try {
    runner = await run({
      logger: graphileLogger,
      connectionString: process.env.DIRECT_URL,
      taskList: {
        ["documents.export_application_documents"]:
          export_application_documents,
      },
    });

    logger.info("Graphile Worker started");
  } catch (err) {
    logger.error("Failed to start Graphile Worker:", err);
    process.exit(1);
  }
};

You can see that Graphile Worker started gets logged when the server starts.

You are most likely right that it has something to do with our builder or application setup, its just very odd that the env variables is updated everywhere except then used by the job.

We are using dokku to deploy our remote applications, which is dockerized, could that be causing any issues? I have rebuilt the container multiple times.

Thank you again!

[benjie]

No idea; sorry! I suggest that you log the envvars inside the startGraphileWorker function (e.g. where you do Graphile Worker started) - that should make it clear if the envvars are wrong in your script and not just inside Graphile Worker.

[benjie]

(Unrelated...) Note your worker usage is wrong; you need to await runner.promise; see: https://worker.graphile.org/docs/library#add-the-worker-to-your-project