Middleware only runs on HTTP API

[sslotsky]

I have configured both pre and pos middleware handlers which work fine when invoking the function that hosts our HTTP API. But when I invoke any other function, it appears the middleware does not run.

Plugins are configured like this:

plugins:
  - serverless-middleware
  - serverless-domain-manager
  - serverless-offline
  - serverless-plugin-aws-alerts
  - serverless-plugin-typescript
  - serverless-plugin-datadog
  - serverless-plugin-resource-tagging

Middleware is configured universally:

custom:
  middleware:
    pre:
      - src/lambdas/middleware/initialize.handler
    pos:
      - src/lambdas/middleware/teardown.handler

The initialize function loads async bindings into my IOC container, and teardown unbinds them. The function that behaves correctly is declared like this:

functions:
  api:
    name: ${self:service}-api-${sls:stage}
    handler: src/api/Api.handle
    timeout: 30 # 30 seconds, limited by API gateway
    events:
      - http:
          path: /{proxy+}
          method: any

When I add console.log statements in my initialize and teardown handlers, I can see the console output when I make a request with Postman. But it does not work correctly when I use aws lambda invoke to test this other function:

  transferOrderShippedHandler:
    handler: src/transfer/queue/transfer-order-shipped-handler/TransferOrderShippedHandlerLambda.handle
    name: transfer-order-shipped-handler-${sls:stage}
    reservedConcurrency: 1 # configured to cap max concurrency, to protect downstreams
    description: Handler for processing transfer order shipments.
    timeout: 300 # 5 min; Not an expensive operation, no need for setting to max timeout
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - FulfillmentOrderShippedQueue
              - Arn
          batchSize: 1

When I invoke this function, I know that initialize does not run because. Partly because I don't see the console output, but also because the exception that gets thrown by the handler indicates that my IOC container was not initialized. Take a look at the bottom line of that stack trace:

    '    at Object.<anonymous> (/Users/sam/src/github.com/deliverr/transfer-service/.build/.middleware/transfer-order-shipped-handler-dev.js:5:93)'

That line shows that the new handler that was generated by serverless-middleware is the one that's being invoked. The contents of that file look exactly like you'd expect them to:

'use strict';
Object.defineProperty(exports, "__esModule", { value: true });
exports.handler = void 0;
const src_lambdas_middleware_initialize = require("../src/lambdas/middleware/initialize");
const src_transfer_queue_transfer_order_shipped_handler_TransferOrderShippedHandlerLambda = require("../src/transfer/queue/transfer-order-shipped-handler/TransferOrderShippedHandlerLambda");
const src_lambdas_middleware_teardown = require("../src/lambdas/middleware/teardown");
const handler = (event, context) => {
    let end = false;
    context.end = () => end = true;
    const wrappedHandler = (handler) => (prev) => {
        if (end)
            return prev;
        context.prev = prev;
        return handler(event, context);
    };
    return Promise.resolve()
        .then(wrappedHandler(src_lambdas_middleware_initialize.handler.bind(src_lambdas_middleware_initialize)))
        .then(wrappedHandler(src_transfer_queue_transfer_order_shipped_handler_TransferOrderShippedHandlerLambda.handle.bind(src_transfer_queue_transfer_order_shipped_handler_TransferOrderShippedHandlerLambda)))
        .then(wrappedHandler(src_lambdas_middleware_teardown.handler.bind(src_lambdas_middleware_teardown)));
};
exports.handler = handler;
//# sourceMappingURL=transfer-order-shipped-handler-dev.js.map%

They chain the original handler together with the middleware handlers, in the exact same way that they do in the handler defined by .build/.middleware/transfer-api-dev.js. So it is extremely puzzling to me that the initialize function is not getting invoked.

[sslotsky]

Also worth noting: when I try this on our staging environment it's the same results, but with one minor difference: the stack trace when I execute the transfer-order-shipped-handler doesn't even appear to show the wrapped handler! No mention of .build/.middleware/ in that stack at all. But the HTTP API works with no issue!

{
  "errorType": "Error",
  "errorMessage": "No matching bindings found for serviceIdentifier: TransferOrderShippedHandler",
  "trace": [
    "Error: No matching bindings found for serviceIdentifier: TransferOrderShippedHandler",
    "    at _validateActiveBindingCount (/var/task/node_modules/inversify/lib/planning/planner.js:82:23)",
    "    at _getActiveBindings (/var/task/node_modules/inversify/lib/planning/planner.js:68:5)",
    "    at _createSubRequests (/var/task/node_modules/inversify/lib/planning/planner.js:103:26)",
    "    at plan (/var/task/node_modules/inversify/lib/planning/planner.js:154:9)",
    "    at /var/task/node_modules/inversify/lib/container/container.js:620:46",
    "    at Container._get (/var/task/node_modules/inversify/lib/container/container.js:590:38)",
    "    at Container._getButThrowIfAsync (/var/task/node_modules/inversify/lib/container/container.js:593:27)",
    "    at Container.get (/var/task/node_modules/inversify/lib/container/container.js:374:21)",
    "    at Object.<anonymous> (/var/task/src/transfer/queue/transfer-order-shipped-handler/TransferOrderShippedHandlerLambda.js:7:135)",
    "    at Module._compile (node:internal/modules/cjs/loader:1105:14)",
    "    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)",
    "    at Module.load (node:internal/modules/cjs/loader:981:32)",
    "    at Function.Module._load (node:internal/modules/cjs/loader:822:12)",
    "    at Module.require (node:internal/modules/cjs/loader:1005:19)",
    "    at Module.Hook.Module.require (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/ritm.js:72:33)",
    "    at require (node:internal/modules/cjs/helpers:102:18)"
  ]```

[sslotsky]

On coming back from the weekend I'm entirely sure that this is my fault. The TransferOrderShippedHandler that I'm trying to wrap calls iocContainer.get() on import, and the handler that's generated by this plugin obviously needs to import the handler at the very beginning, so we therefore try to access the container before the middleware can even be invoked. Will have to think on that.