@google-cloud/dataflow listJobs missing queued jobs

[fider]

Environment details

• Package: @google-cloud/dataflow@3.0.1 (tested also on v2.x)
• OS: Docker Desktop v4.25.1 on MacOS 14.1.1, M2
• Node.js version: 18.18.2 (Docker image node:18-alpine)
• npm version: 9.8.1

Steps to reproduce

Steps:

1. Start Dataflow job via FlexTemplatesServiceClient.launchFlexTemplate.
2. On GCP dashboard job have status 'Queued'.
3. Using shell $ gcloud dataflow jobs list I can find job on the list with 'STATE' 'Queued'.
4. Using node client JobsV1Beta3Client.getJob(jobId) - I receive job in response with currentState=JOB_STATE_QUEUED.
5. FAIL: Using node client JobsV1Beta3Client.listJobsAsync and listJobs - cannot find queued job in the response with currentState=JOB_STATE_QUEUED. Job can be found on the list after it will change status to other than queued.

Code example

  const dfJobsIterable = this.client.listJobsAsync({
    projectId,
    location,
    filter: Filter.ALL,
    pageSize: 10,
  });

  for await (const dfJob of dfJobsIterable) {
    // 'queued' dfJob cannot be found until change state
  }

Edit:

• JobsV1Beta3Client is configured with fallback: 'rest'

Thanks!

[sofisl]

Hi @fider, could you try setting autopaginate: false?

const request = {request, pageSize: 50}
const [resultArray, nextPageRequest, rawResponse] =
  await client.samplePaginatedMethod(request, {autoPaginate: false});

From docs, if you want to specify pageSize, you need to do it with autopagination.