WARNING: Never try to work with this plugin

[vijay-ashok]

I tried to integrate AWS MSK with serverless using this plugin and after trying 3+ days, I got nothing. NOTE to author: I highly appreciate creating such plugins but if its not maintained at least give a heads-up saying its not man. Totally frustrated using this plugin

My serverless.yml file

EventsProcessor:
  handler: src/handlers/events.handler
  layers:
    - {Ref: PythonRequirementsLambdaLayer}
  timeout: 300
  events:
    - msk: 
        arn: '<arn of AWS MSK>'
        topic: <topic name>
        batchSize: 100
        startingPosition: LATEST
        enabled: true # this will turn it off

custom.yml

serverless-offline-msk:
  allowAutoTopicCreation: true # If we can create topics in kafka automatically
  brokers: # list of brokers to use for kafka
    - "AWS MSK broker 1"
    - "AWS MSK broker 2"

[lukevers]

Hey @vijay-ashok -- are you saying you tried to use this AWS MSK directly with this plugin? It's intended to use with a local kafka instance (hence "offline", similar to serverless-offline-aws-eventbridge).

There's a working example here: https://github.com/lukevers/serverless-offline-msk/tree/main/examples/docker-compose-single-broker

If it's still not working for you, if you setup a reproduction repository I can pull it down and help isolate the issue (assuming you're not trying to connect to AWS MSK brokers directly -- again, this plugin is to simulate AWS MSK in a local environment).

[vijay-ashok]

My bad. I was trying to use it with AWS MSK. I guess there is no way I can use serverless-offline plugin with AWS MSK then

Thanks for the update @lukevers

[lukevers]

@vijay-ashok If you're trying to connect directly to MSK from your own computer I would be surprised if you were able to do that unless you had a VPN into your VPC with the right access setup -- but in my experience when I work with MSK I like to purposely not allow any direct traffic and instead setup security groups to be used with Lambda (using Lambda for both producers and consumers).

It looks like they may allow public access if you want..but it seems like a bad idea to open it up to the internet https://docs.aws.amazon.com/msk/latest/developerguide/public-access.html

Even if you did attempt to use this directly against the MSK brokers over VPN, I'm unsure if it work or not (I haven't tried it myself).

[vijay-ashok]

@lukevers This is for my staging environment and the docker container is running inside a kubernetes namespace and inside my VPC only and has access to all my AWS resources. Problem is serverless-offline doesnt support AWS MSK events and I need exactly this 😢