BTE failing on some queries with TypeError

[tokebe]

BTE is failing on some async requests, only reporting "TypeError: Cannot read properties of undefined (reading 'length')". The actual responses are somewhat hard to parse due to #671.

What we know:

• The failure is due to some attempt to read .length of something which is unexpectedly undefined.
• The same query can fail at one time and then not fail at another time.
• The failure must be happening quite early in execution, as there are no logs in the queue dashboard for queries which fail in this way, and those logs are captured live rather than being sent at the end.

Example here.

This fact that this issue is intermittent makes me think it has something to do either with async handling or thread handling, however there are limited places where .length is read, and many of them seem unlikely to fail under any circumstances. More investigation needed.

[tokebe]

@rjawesome As I'm not going to be super available in the coming days, I'd like you to take a look at this, with priority over #671 & #662.

[rjawesome]

Note that I can't reproduce this locally

Just by looking at the code, this seems to be the one check to .length that could be undefined: https://github.com/biothings/smartapi-kg.js/blob/a0eab7b17447d1ad6014072fbeb6547e5e0ed084/src/operations_builder/sync_operations_builder_with_reasoner.ts#L153 (there could be other ones in dependencies)

Since it is actually able to produce "JobQueueingError" as the issue, that means the error was probably caught here. It looks like there is a console log here. @tokebe is there any way to access the server's stdout/stderr logs? there could possibly be more info about the error in such console logs.

[tokebe]

@rjawesome There isn't a meaningful way to capture console logs on the production server as we don't have direct access and they don't keep long-term persistent logs AFAIK.

We'll have to settle for a preventative fix and monitoring this one, and hopefully by inclusion of #675 and sentry monitoring we'll be able to eventually capture the exact cause.