Proposal backend stops working with Knex timeout error

[simonfernandes]

The proposal system has gone down a couple of times in the past week where where the backend has completely stopped working due to a Knex error:

{"errors":[{"message":"Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?","locations":[{"line":2,"column":3}],"path":["settings"]}],"context":{"req ││ [2024-09-11T07:39:57.291Z] ERROR - GraphQL response contained error(s)

Relevant threads on Teams:

• https://teams.microsoft.com/l/message/19:7bd9b14ff376400bbb59ff21873a73ea@thread.skype/1725546114349?tenantId=3f66361c-a87e-4158-8f61-99e82db3cac8&groupId=403a668a-985f-445b-a4e3-b3a3070bb25b&parentMessageId=1725546114349&teamName=ISIS%20Computing%20Division%20Staff&channelName=FASE&createdTime=1725546114349

• https://teams.microsoft.com/l/message/19:7bd9b14ff376400bbb59ff21873a73ea@thread.skype/1726040398855?tenantId=3f66361c-a87e-4158-8f61-99e82db3cac8&groupId=403a668a-985f-445b-a4e3-b3a3070bb25b&parentMessageId=1726040398855&teamName=ISIS%20Computing%20Division%20Staff&channelName=FASE&createdTime=1726040398855

[simonfernandes]

This is happening quite frequently on dev, we think it's related to the Scheduler sending requests for PDFs, possibly worse on dev because of testing.

As of now, this has happened on these dates (links to Teams threads)

Prod

• Thursday 5th September
• Wednesday 11th September
• Wednesday 18th September
• Thursday 19th September

Dev

• Monday 16th September
• Tuesday 17th September
• Probably more times that I've missed

[simonfernandes]

Closed via https://github.com/UserOfficeProject/user-office-core/pull/779 and https://github.com/UserOfficeProject/user-office-core/pull/781