Kafka on AKS sample, EH+Kafka sample, README.md on simulator

[rasavant-ms]

Work done during OneWeek, November 2019

1. Kafka on AKS -> Databricks -> CosmosDB
2. EH+Kafka -> Azure Functions -> CosmosDB
3. README.md on simulator

[algattik]

I've made some updates to this base into https://github.com/Azure-Samples/streaming-at-scale/tree/algattik/pr-79-integration:

• merged some parallel work (Premium Functions, AKS in VNET configuration from the Flink branch)
• replaced solution "eventhubskafkaproducer-functions-cosmosdb" with a toggle in solution "eventhubs-functions-cosmosdb" (-g: Protocol used by event generator: 'eventhubs' or 'kafka'. Default=eventhubs) to reduce level of redundant code

Please merge that branch into the PR and make the fixes below.

• akskafka-databricks-cosmosdb

  • Deployment is not idempotent (Creating kubectl namespace Error from server (AlreadyExists): namespaces "kafka" already exists) update 2019-11-11: added helm chart to make deployment idempotent
  • Performance is too low. With-t 1, solution churns around 800-900 events/second and fails integration test (-s V) update 2019-11-11: seems to be resolved by increasing from 1 to 3 replicas in the helm chart
  • Operator takes about 10 minutes to spin Kafka cluster. If the generator and processor are started in the meantime, they get configured with <none> as broker host. update 2019-11-11: added code to wait for IP
  • Kafka gets configured with an external LB public IP. For security and performance (direct communication between clients and topic brokers) we should use internal IPs. Thanks to my merged work above this should be easy, hdinsightkafka-databricks-sqldw has scripts to peer solution VNET with Databricks VNET.
  • Please add information in the README file, regarding the AKS deployment and the templates used

• eventhubskafka-functions-cosmosdb

  • Performance is too low. With-t 1, solution churns around 700-800 events/second and fails integration test (-s V). (This was measured after I merged the change from Dedicated to Premium Functions, but Premium Functions work well in other -functions- solution) update 2019-11-11: found issue to be that the scale controller for Premium Functions does not work with the Kafka trigger yet, reverted to use Dedicated plan for this solution