elastic / elasticsearch-hadoop

:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
https://www.elastic.co/products/hadoop
Apache License 2.0
10 stars 989 forks source link

ELK to Databricks Data load #2141

Closed Sasidhar0101 closed 1 year ago

Sasidhar0101 commented 1 year ago

Hi,

I'm trying to load data into data bricks from ELK with the help of API using python script. Can you suggest the best option so I can read the huge data like 200 million in a single shot? The method I'm using now allows me only 5 million of data even that is running for 40 min approx. The run is happening on a single core and not spreading between the cluster cores in Databricks. Please help me with the code if you have any.

Databricks #Hadoop #ApacheSpark

Thanks Sasidhar

### Tasks
masseyke commented 1 year ago

I don't think I understand the problem you're running into. This sounds like a good topic for discussion on https://discuss.elastic.co/ rather than a confirmed bug, right?