Revamp Obsrv Sink to Index All Data in Elastic search

[manjudr]

Project Details

What is Sunbird Obsrv?

Sunbird Obsrv comprises several pluggable tools and microservices that come together to enable observability features on any platform/solution. This includes the ability to capture granular events via telemetry, create measures, and observe various events/actions carried out by the system/users/devices (like IoT devices) on any platform/solution.

Sunbird Obsrv can be utilized as an independent building block by adopters or as part of a system that employs other Sunbird building blocks. Sunbird Obsrv comes with a set of microservices, APIs, and some utility SDKs to make it easy for adopters to rapidly enable powerful data processing and aggregation infrastructure to process telemetry data, validate telemetry stream data, as well as aggregate and generate actionable insights via APIs. It also has built-in open data cataloguing and publishing capability. It is built keeping extensibility in mind so that adopters have the flexibility to adapt the telemetry and tools to their specific use cases. More details are here

Features to be implemented

Provision the Elasticsearch using a Helm chart and configuring the Obsrv sink to index all data to Elasticsearch.

• Current Scenario: At present, Obsrv reads messages from the Kafka sink topic to index all data into analytical storage apache druid.

• Acceptance Criteria: Improve Obsrv by modifying its configuration to enable the reading of messages from the sink topic and indexing all messages to Elasticsearch.

Learning Path

• Complexity - Medium
• Skills Required - Elasticsearch, Kafka, Kubernetes, Docker, Helm
• Name of Mentors - @manjudr @anandp504
• Project size - 8 Weeks

Product Set Up

• Scripts - https://github.com/Sunbird-Obsrv/obsrv-automation/blob/main/README.md
• API Documentation - https://github.com/Sanketika-Obsrv/obsrv-api-service/tree/main/swagger-doc
• More details about the sunbird-obsrv - - https://obsrv.sunbird.org/use/installation-guide

Milestones

• [ ] Understanding of the existing apache druid sink configuration
• [ ] Understanding elastic search index mapping spec
• [ ] Understanding of elastic search Basic APIs
• [ ] Create a helm chart to provision the elastic search
• [ ] Create an index mapping file to index the data into an elastic search by reading from sink topic
• [ ] Query the result from the elastic search and validate the result.

[wasup-yash]

I can work on this in case #4 is not free to be assigned.

[NEGRAH-S]

I am interested to contribute on the above issue