Multiple indexes setting for 'es.resource'

[mungeol]

'es.resource' = 'apache-2014.09./apache-access' or 'es.resource' = 'apache-2014.09.29,apache-2014.09.30/apache-access' are not working well for 'select count(*) from test' which is HiveQL. The count result is not right. 'es.resource' configuration should support multiple indexes setting. Or, at least give an error message.

[costin]

Could you explain what the expectation is and what is the actual result? es-hadoop does minimal interpretation of the index/type and feeds the information directly to Elasticsearch.

[mungeol]

The hive table I created is like below

CREATE EXTERNAL TABLE test ( date timestamp, clientip string, request string ) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES ( 'es.resource' = 'apache-2014.09.29/apache-access', -- or -- 'es.resource' = 'apache-2014.09.30/apache-access', 'es.mapping.names' = 'date:@timestamp' );

and I used 'select count() from test;' which is a hive query to count the total number of rows of the table. the result is same with ES count. the count result are 1454536 and 215564 for each apache-2014.09.29 and apache-2014.09.30 index then, I changed 'es.resource' = 'apache-2014.09.29/apache-access' to 'es.resource' = 'apache-2014.09./apache-access' or 'es.resource' = 'apache-2014.09.29,apache-2014.09.30/apache-access' for including multiple indexes. and I used 'select count(*) from test;' again to count the total number of documents of the indexes, but the result is different with ES count. the count result is 2919161 which should be 1670100 (1454536 + 215564).

---

environmental information

• centos base 6.4 64-bit / java version "1.7.0_55"
• CDH-5.1.2-1.cdh5.1.2.p0.3
• hive 0.12.0
• elasticsearch-hadoop-2.0.1
• 3 nodes' hadoop and es cluster

[costin]

Hi,

Sorry for the delay in picking this up. I've tried reproducing this but can't - maybe it has something to do with the dataset or potentially the way the counting is done. I'm not sure where that number (2919161) is coming from - clearly some data is being returned but not properly processed. Can you please confirm the following:

• the count result when done directly to ES through curl. Double check that the indices are properly refreshed (i.e there's no inflight data)
• after you have loaded the data, start a separate Hive query with logging turned all the way to TRACE level on org.elasticsearch.hadoop package (as mentioned in the docs). There will be a lot of data so please be patient - archive the results and let me know where you have uploaded them.

Thanks!

[mungeol]

Hi,

I created new indexes with small data for easy test. I indexed same three documents into each cars-01 and cars-02 indexes with same type name transactions

POST /cars-01/transactions/_bulk { "index": {}} { "price" : 10000, "color" : "red", "make" : "honda", "sold" : "2014-10-28" } { "index": {}} { "price" : 20000, "color" : "red", "make" : "honda", "sold" : "2014-11-05" } { "index": {}} { "price" : 30000, "color" : "green", "make" : "ford", "sold" : "2014-05-18" }

POST /cars-02/transactions/_bulk { "index": {}} { "price" : 10000, "color" : "red", "make" : "honda", "sold" : "2014-10-28" } { "index": {}} { "price" : 20000, "color" : "red", "make" : "honda", "sold" : "2014-11-05" } { "index": {}} { "price" : 30000, "color" : "green", "make" : "ford", "sold" : "2014-05-18" }

'GET cars-01/_search?search_type=count' returns 3 hits 'GET cars-02/_search?search_type=count' returns 3 hits 'GET cars-*/_search?search_type=count' returns 6 hits

and created table like below

CREATE EXTERNAL TABLE cars ( price bigint, color string, make string, sold timestamp ) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES ( 'es.resource' = 'cars-*/transactions' );

'select * from cars' and 'select count(*) from cars' returns different result every time example 1 (9 rows)

10000 red honda 2014-10-28 00:00:00 20000 red honda 2014-11-05 00:00:00 30000 green ford 2014-05-18 00:00:00 20000 red honda 2014-11-05 00:00:00 30000 green ford 2014-05-18 00:00:00 10000 red honda 2014-10-28 00:00:00 10000 red honda 2014-10-28 00:00:00 20000 red honda 2014-11-05 00:00:00 30000 green ford 2014-05-18 00:00:00

example 2 (10 rows)

10000 red honda 2014-10-28 00:00:00 20000 red honda 2014-11-05 00:00:00 30000 green ford 2014-05-18 00:00:00 10000 red honda 2014-10-28 00:00:00 10000 red honda 2014-10-28 00:00:00 10000 red honda 2014-10-28 00:00:00 20000 red honda 2014-11-05 00:00:00 30000 green ford 2014-05-18 00:00:00 20000 red honda 2014-11-05 00:00:00 30000 green ford 2014-05-18 00:00:00

I uploaded two logs. es_hadoop.log file includes logs after I added 'log4j.category.org.elasticsearch.hadoop.hive=TRACE' into log4j.properties file hive.log file includes logs after I changed 'hadoop.root.logger=INFO' to 'hadoop.root.logger=TRACE' here is the links for both logs https://raw.githubusercontent.com/mungeol/log/master/es_hadoop.log https://raw.githubusercontent.com/mungeol/log/master/hive.log hope it could help

Thanks.

[romanmar1]

I get this issue as well, but from a Spark perspective. I have two exactly identical indices. I use a query that return exactly 4 documents. If i set es.resource to be "index1/mydoc", everything works as expected:

1. The count of the rdd is 4.
2. collecting partitions reveals that there is a single partition of size 4.

If i set es.resource to be "index1,index2/mydoc", things get a bit wierd:

1. The count is 16 (instead of the expected 8).
2. There are two partitions, each of size 8, with the same documents.

Moving forward, if i add an additional index, "index1,index2,index3/mydoc", the count will be 36, with 3 identical partitions of size 12 each.

[costin]

Folks, can you try the latest Beta (4) and see whether it addressed your issue? There have been several updates on this front.

Thanks,

[mungeol]

It is working now. Thanks,

[costin]

Closing the issue...