Importing data from Elasticsearch

camunda-community-hub / zeeqs

GraphQL API for Zeebe data

Apache License 2.0

62 stars 15 forks source link

Importing data from Elasticsearch #5

Open saig0 opened 4 years ago

saig0 commented 4 years ago

import the data that is provided by the Elasticsearch exporter

asaf commented 4 years ago

@saig0 first, i'd like to say that it's a great effort as there's currently no way to read Zeebe data easily.

I would like to clarify this, when ES exporter is enabled, Operate uses ES transformers in order to create its own data, so data gets built on the fly when it is pushed by the exporter.

Since the approach you chose is to create a local DB, I was wondering what would be the approach to support importing data from ES.

saig0 commented 4 years ago

@asaf, my idea is similar to Operate.

pull ES
aggregate data
store data in internal DB (which is used to query from)
pull next data from ES

The difference to Operate is that the data is stored in a database that is maybe not ES.

asaf commented 4 years ago

@saig0 ah, I thought operate puts some "processors" on ES somehow, yea fair enough, I guess you just need some kind of cursor / store the last document fetched as with Hazelcast sequence

mohit2494 commented 4 years ago

Hi there, I'm looking forward to contribute to this through Hacktoberfest. Can someone help me out chart the details, so that I can make incremental code reviews for this. Best :)

saig0 commented 4 years ago

@mohit2494 awesome :tada:

The basic idea here is to import the data from ES and store it similar to the Hazelcast importer.

create a new module for the ES importer similar to /hazelcast-importer
create a new importer to poll the data from the different ES indexes and store them in the same way as /hazelcast-importer/src/main/kotlin/io/zeebe/zeeqs/importer/hazelcast/HazelcastImporter.kt
store the current position of the ES index to continue after a restart

The data in ES are generated by the ES exporter: https://github.com/zeebe-io/zeebe/tree/develop/exporters/elasticsearch-exporter Here are the templates for the ES indexes: https://github.com/zeebe-io/zeebe/tree/develop/exporters/elasticsearch-exporter/src/main/resources

Please ask if you've any question or need help :)

mohit2494 commented 4 years ago

Thank you so much for the reply. It looks interesting, though a bit challenging. Can I have an email id where I can get in touch.

Best. Mohit

saig0 commented 4 years ago

@mohit2494 you can get in touch using the Zeebe Slack channel. You can ask questions in general or contact me directly.

fromz commented 2 years ago

@saig0 is there any idea of an ETA for this at all?

saig0 commented 2 years ago

@fromz no ETA. Currently, I don't have any plans to work on it because I don't have a concrete need for it.

But I'm open to contributions from the community. :rocket: