orinem
commented
2 years ago I added socket.tag.activetunnels to the CHANNEL_OPEN_FAILURE log message. In almost all cases, activetunnels was 8 when a failure occurred with reasonCode of 4. There were a few of 7, but that can be explained by the fact that activetunnels isn't incremented until after a channel is setup, so an eighth channel was likely being setup at the time of the failure.
Given this, it seems that this particular AMT BIOS has a limit of 8 simultaneously open channels.
I suggest only allowing some number of simultaneous channels less than 8. It can easily be done with an "await semaphore" in CIRAHandler.ExecRequest() as follows:
npm install await-semaphoreCIRAHandler.ts:
import { Semaphore } from 'await-semaphore'...
export class CIRAHandler {
xml: string
httpHandler: HttpHandler
username: string
password: string
channel: CIRAChannel
channelState: number = 0
connectAttempts: number = 0
socket: CIRASocket
semaphore: Semaphore
constructor (httpHandler: HttpHandler, username: string, password: string) {
this.username = username
this.password = password
this.httpHandler = httpHandler
// I chose 4 simultaneous channels, half the assumed maximum
this.semaphore = new Semaphore(4)
}...
async ExecRequest (xml: string): Promise<any> {
// If it weren't for the recursive call, we could probably
// wrap the entire function contents with this.semaphore.use()
const release = await this.semaphore.acquire()
if (this.channelState === 0) {
this.channelState = await this.Connect()
}
if (this.channelState === 2) {
try {
if (this.httpHandler.isAuthInProgress == null) {
this.httpHandler.isAuthInProgress = new Promise((resolve, reject) => {
this.httpHandler.authResolve = resolve
})
} else {
await this.httpHandler.isAuthInProgress
}
const data = await this.channel.write(xml)
const parsedData = this.handleResult(data)
release()
return parsedData
} catch (error) {
if (error?.message === 'Unauthorized' || error?.message === 'Closed') {
this.channelState = this.channel.CloseChannel()
release()
return await this.ExecRequest(xml)
} else {
release()
throw error
}
}
}
release()
return null
}Limited testing so far indicates that this works.
The reasonCode 2 failures are likely due to a limit on the number of wsman connections the BIOS can handle simultaneously; if only one, then perhaps the number of pending connections allowed. Either way, reducing the number of simultaneous channels will also reduce the number of connections to wsman too and this error hasn't been seen with the above charge.
matt-primrose
Describe the bug 🪲
AMT resources (enumeration contexts for example) are limited. If too many requests are sent to a computer over CIRA in a given time frame, some will fail, typically with a CHANNEL_OPEN_FAILURE with either code 2 or 4. This results in an exception and the API call fails.
Code 2 is OPEN_CONNECT_FAILED and code 4 is OPEN_RESOURCE_SHORTAGE.
It is particularly bad if the Pull for an enumeration fails as the enumeration context will be lost until it times out. One fewer context makes the problem more likely.
This isn't the fault of the caller. MPS needs to do some kind of flow control and only let so many requests over CIRA be active at once - especially with enumerations. Failures need to be retried - preferably after a delay.
To Reproduce 🪜 Steps to reproduce the behavior:
These exceptions do occur randomly, but the following is almost certain to generate these errors.
Note that error messages will flash on the page, and some data will not be filled in. Further attempts to refresh the page may generate more errors and different data will not be filled in.
The MPS logs will have errors like this:
and later, after restarting everything:
Expected behavior
The page should reload with no errors.
AMT Device (please complete the following information): 🖥️
Service Deployment (please complete the following information): ⛈️