Cloud Tasks Client throws 4 DEADLINE_EXCEEDED

[j-christoffersen]

We're using cloud tasks to enqueue jobs. This is happening for a few jobs that rely on cloud-tasks, the one i'll focus on writes somewhere between 10-25 tasks it creates by reading a file. 10-20% of the time, this job will fail because the cloud tasks client throws with 4 DEADLINE_EXCEEDED.

I've read through the issue here but it seems that was closed over a year ago now and I'm running into this issue with the latest version of cloud tasks (also ran into it before upgrading using major version 3 of the package).

Environment details

• google-cloud/tasks
• Deployed to Gen2 cloud function
• Node 18
• Yarn 1.22.19

Steps to reproduce

Below is my code I'm using to add new tasks

 import { v2beta3 } from "@google-cloud/tasks"

const { GCP_PROJECT } = process.env

if (!GCP_PROJECT) {
  throw new Error("GCP_PROJECT is required")
}

const cloudServiceAccountEmail = process.env.CLOUD_TASK_SERVICE_ACCT_EMAIL;

class CloudTasksClientWrapper {
  private client: v2beta3.CloudTasksClient

  constructor() {
    this.client = new v2beta3.CloudTasksClient()
  }

  async createHttpTaskWithToken({
    queueName,
    payload,
    url,
    location = "us-central1",
  }:
  {
    queueName: string
    payload: string
    url: string
    location?: string
  }) {
    const parent = this.client.queuePath(GCP_PROJECT!, location, queueName)

    const body = Buffer.from(payload).toString("base64")

    const task = {
      httpRequest: {
        httpMethod: "POST" as const,
        url,
        oidcToken: {
          serviceAccountEmail: cloudServiceAccountEmail,
          audience: new URL(url).origin,
        },
        headers: {
          "Content-Type": "application/json",
        },
        body,
      },
    }

    // Send create task request.
    const [response] = await this.client.createTask(
      { parent, task },
      { timeout: 5000 }
    )
    return response
  }
}

export { CloudTasksClientWrapper }

Then I call createHttpTaskWithToken from various jobs, for example, after reading every 50 lines from a 1000-line csv, I'll call this function with a task to process those 50 lines.

[majkelEXE]

Have you tried applying dispatchDeadline property to your task? In my case it solved the error - be wary to stay in range between 15seconds to 30 minutes.

Example:

const task = {
        httpRequest: {
            httpMethod: google.cloud.tasks.v2beta3.HttpMethod.POST,
            url,
            headers: {
                "Content-Type": "application/json",
            },
            oidcToken: {
                serviceAccountEmail: email,
                audience: url,
            },
            body: Buffer.from(JSON.stringify(payload)).toString("base64"),
        },
        dispatchDeadline: { seconds: 60 },
    };

[j-christoffersen]

@majkelEXE Were you seeing that error in client logs or in logs from the cloud task itself? Mine seems to be an issue adding tasks to the queue, not processing them once their in the queue.

[majkelEXE]

In my case it was probably caused by adding hundreds of tasks simultaneously

[Satish-Lakhani]

Did anyone get a resolution to the DEADLINE_EXCEEDED error? I've tried multiple things but have not really been able to find any solution. I see this issue has been discussed since 2020 for older versions too. Is @googleapis-publisher really not able to solve this issue?

Here is the code block that is causing an error: `const authClient = await auth.getClient(); const client = new CloudTasksClient({ authClient: authClient, });

  const parent = client.queuePath(PROJECT, REGION, queue);
  const task = {
    httpRequest: {
      headers: {
        'Content-Type': 'text/plain',
      },
      httpMethod: 'POST',
      url: `${URL}/v1/message`,
    },
    dispatchDeadline: {
      seconds: 30 * 60, //30 minutes
    }
  };

  if (payload) {
    // task.httpRequest.body = Buffer.from(JSON.stringify(payload)).toString('base64');
    task.httpRequest.body = Buffer.from(JSON.stringify(payload));
  }

  if (inSeconds) {
    // The time when the task is scheduled to be attempted.
    task.scheduleTime = {
      seconds: parseInt(inSeconds) + Date.now() / 1000,
    };
  }

  // Send create task request.
  const request = { parent: parent, task: task };
  const [response] = await client.createTask(request);
  // console.log(`Created task ${response.name}`);
  return response.name;`

I have even tried submitting requests in sequence as shown below. But it struggles to create tasks soon after it reaches 5000.

for (contact of recipients) { try { await createHttpTask({ campaign, contact }, queueName); } catch (err) { console.error('Error in createHttpTask::: ', err); continue; } }

I expected it to run for over 50k tasks, but it couldn't even reach 5k tasks.

Here is my runtime env details; Running it on Google cloud run, on node 14, using latest "@google-cloud/tasks": "^5.4.0",

Here is the error log snapshot

Any help/suggestion would really be appreciated.

Thanks ~Satish