Question: Kinesis Option 1 - Producer Demo Application

[aeggbeg]

Hi, This is not a bug. I am trying to find the consumer node.js/JavaScript application for Kinesis Dat Streaming Option 1 in the repo and associated documentation to modify the script - not sure where to look. I visited this repo from below link

https://docs.aws.amazon.com/solutions/latest/aws-streaming-data-solution-for-amazon-kinesis/comps-api-kds.html

also is there a producer application that will generate sample data to be sent to API gateway? I have used the CF template to launch option 1. But I am not sure how to test it by sending data using JavaScript from my own web application. Can you please provide some guidance or links?

thanks.

[dscpinheiro]

Hi @aeggbeg,

The consumer is located on this file (https://github.com/awslabs/aws-streaming-data-solution-for-amazon-kinesis-and-amazon-msk/blob/main/source/lambda/kds-lambda-consumer/index.js), and it is very similar to the kinesis-process-record blueprint available on the AWS Lambda console. If you make any changes to it, you might also have to modify this file (https://github.com/awslabs/aws-streaming-data-solution-for-amazon-kinesis-and-amazon-msk/blob/main/source/patterns/apigw-kds-lambda.ts#L118-L123), where the stream and function are configured to work with each other.

We don't have a demo producer for Kinesis option 1, but the Amazon API Gateway documentation has some examples on how to invoke your REST API through generated SDKs (https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-generate-sdk-javascript.html). One thing to keep in mind is that by default the solution uses IAM authentication for the API. If you're going to make it available to non-AWS users, it might be worth to look into Amazon Cognito user pools as authorizer (https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-integrate-with-cognito.html).

For testing, you can also leverage the API Gateway console (https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-test-method.html) or the test-invoke-method API. Here's an example using boto3:

apigateway = boto3.client('apigateway')
apigateway.test_invoke_method(
    restApiId='restApiId',
    resourceId='resourceId',
    httpMethod='POST',
    pathWithQueryString='/record',
    body=json.dumps({ 'data': 'abc', 'partitionKey': 'pk001' })
)

[aeggbeg]

Thank you very much for your answer. I am able to test option 1 successfully. Yes my use case is for non-AWS users. Is there any plan for including Cognito in the template set e.g. option 5? It will be great to follow best practice then build on my own.

Many thanks again for your help.

[dscpinheiro]

Thanks for the feedback @aeggbeg, we'll look at enhancing option 1 in a future release so that you can select the authentication type (IAM or Cognito User Pools) for the Rest API.

[aeggbeg]

YES PLEASE :) I am happy to help test it or outline my use case here in detail if you like. Many thanks.

[dscpinheiro]

@aeggbeg,

We released a new version of the solution (v1.5.0), and option 1 now includes an Amazon Cognito user pool as the default authorizer for API Gateway (https://docs.aws.amazon.com/solutions/latest/aws-streaming-data-solution-for-amazon-kinesis/option1-api-kds-lambda.html).

By default, the solution will create the user pool and user pool client, but we also added instructions in case you want to use existing resources:

https://github.com/awslabs/aws-streaming-data-solution-for-amazon-kinesis-and-amazon-msk/blob/bd40e28cd2010ceb30e0188bff9ff1259ed4ecb0/source/patterns/apigw-kds-lambda.ts#L75-L81

[crowdsense]

Many thanks for the update. I will do a fresh install and come back to you with feedback and new feature request 👯‍♂️ :)