Transactionize Queries return incomplete Data when using the generators

[mt-ftooth]

Recording here for posterity I was recently working on putting a Sumologic query into a lambda that make use of transactionize, timeshift and join, specifically:

| compare with timeshift 15m 8
| sort by _messagetime asc
| transactionize fid_locale (merge 
...
_messagetime join with "," as ssu_times
)

The query runs and is successful, but if it is run with the _getIterator( generators and for await (const of... as suggested. The returned data in ssu_times only returns the first value and not the other 7 parts of the joined message as expected. This can be observed in the SL dashboard; when running a query like this, the results are initially reported as 1 item and are then filled out with the other 7 parts of the join after a few seconds. My way round this, while rather hacky, was to simply wait some period after starting the job before manually retrieving results. Like so:

  const id = await client.newSearchJob(searchParams);

  return new Promise(((resolve, reject) => {
    // Setting 20s timeout to wait for all transactionize processes to finish
    // Using the current process, we do get results but they are uncomplete!
    setTimeout(resolve, 20000);
  })).then(async () => {
    // eslint-disable-next-line
    const results = await client.getJobResults('record',id,0)

    return results.records.map(message => message.map);
  });

I don't have a suggested solution to this atm, as we generally go off the reported state from Sumologic to check the results are all there, but the state doesn't change between 1 values and 8 values, so I'd call this their bug tbh 🤷 Perhaps there is a newer version of the API, but my solution above works for now.

[mt-james]

Are we using a GET to make the query? A better pattern would be POST and then retrieve the results when it is done.

[mt-ftooth]

Are we using a GET to make the query? A better pattern would be POST and then retrieve the results when it is done.

This is what the iterator does actually do, makes the initial post job and then yields a series of promises to check up on the state and results. However it stops short of having all the readings as it quits out when it gets to the final state STATE_DONE_GATHERING_RESULTS during which the results are still being updated (at least, for this query type)