Extension - Channel error will exit that channel but leave extension running with reduced channels

[huntharo]

Motivations

• It was suspected that an exiting channel would not cause the graceful shutdown of the extension
• A unit test has confirmed this is the case
• This can lead to a router running with only 1 or 2 connections when it should have 10 or 20

Acceptance Criteria

• [x] Exit the channel loop when the goaway_received signal is set
  • Under heavy load this will exit the invocation relatively quickly
  • Fixed part of #187:
    • https://github.com/pwrdrvr/lambda-dispatch/pull/187/files#diff-920382ec5579cd4cff9581af7a373a6b9feafce3d4ea6725feef9547bfd8a9b8R97-R101
• [x] Add connection pools for the contained app and for the router
  • This will make instances of this less frequent as a connection error from the contained app will be recoverable unless the contained app has exited (in which case the exec env is going away)
  • This will make it more likely that an /api/chunked/close request will reach the router
  • Being handled in

    • 142

    • 178

• [x] Call /api/chunked/close when the extension sets the goaway_received flag outside of receiving an actual goaway message from the router
  • If the router is still reachable this should exit the invocation quickly
• [x] 💪 If the router is not reachable, set timeouts that will exit the invocation - This is extra credit and can be moved to another ticket if desired

Issue Locations

• RouterChannel loop
  • All the ?; cases will exit the loop while leaving other channels running
  • https://github.com/pwrdrvr/lambda-dispatch/blob/976348ffc63ec4006001c96bfb50590920dd4814/extension/src/router_channel.rs#L475
• LambdaRequest start of channels
  • Task errors with the result of the ChannelRequest.start, but does not signal goaway so other channels can continue
  • https://github.com/pwrdrvr/lambda-dispatch/blob/976348ffc63ec4006001c96bfb50590920dd4814/extension/src/lambda_request.rs#L122
• LambdaRequest is joining all of the channels
  • futures::future::try_join_all
    • This says that if one future returns an Err that all the other futures will be canceled
    • However, the tasks are not actually cancelled and instead the cancellation is cooperative
    • As a result, all the other channel tasks keep running indefinitely   - https://github.com/pwrdrvr/lambda-dispatch/blob/976348ffc63ec4006001c96bfb50590920dd4814/extension/src/lambda_request.rs#L126-L137