search

elastic / stream2es

Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)
355 stars 60 forks source link

Problem running against current master #45

Closed spinscale closed 9 years ago

spinscale commented 9 years ago

I cannot run stream2es against the current master due to the latest mapping changes

Tried to run

stream2es wiki -d 5000000 --source file:///Users/data/enwiki-latest-pages-articles.xml.bz2

resulting in this on the client side

clojure.lang.ExceptionInfo: clj-http: status 400 {:object {:trace-redirects ["http://localhost:9200/wiki"], :request-time 808, :status 400, :headers {"content-type" "application/json; charset=UTF-8", "content-length" "360"}, :body "{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"mapping [_default_]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}},\"status\":400}"}, :environment {client #<client$wrap_output_coercion$fn__1005 clj_http.client$wrap_output_coercion$fn__1005@3f8f9dd6>, req {:request-method :put, :url "http://localhost:9200/wiki", :body "{\"settings\":{\"index.refresh_interval\":\"5s\",\"index.number_of_shards\":2,\"query.default_field\":\"text\",\"index.number_of_replicas\":0},\"mappings\":{\"_default_\":{\"_size\":{\"enabled\":true,\"store\":true},\"_all\":{\"enabled\":false}}}}"}, map__927 {:trace-redirects ["http://localhost:9200/wiki"], :request-time 808, :status 400, :headers {"content-type" "application/json; charset=UTF-8", "content-length" "360"}, :body "{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"mapping [_default_]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}},\"status\":400}"}, resp {:trace-redirects ["http://localhost:9200/wiki"], :request-time 808, :status 400, :headers {"content-type" "application/json; charset=UTF-8", "content-length" "360"}, :body "{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"mapping [_default_]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}},\"status\":400}"}, status 400}}
    at clj_http.client$wrap_exceptions$fn__926.invoke(client.clj:111)
    at clj_http.client$wrap_accept$fn__1046.invoke(client.clj:380)
    at clj_http.client$wrap_accept_encoding$fn__1052.invoke(client.clj:394)
    at clj_http.client$wrap_content_type$fn__1041.invoke(client.clj:370)
    at clj_http.client$wrap_form_params$fn__1090.invoke(client.clj:481)
    at clj_http.client$wrap_nested_params$fn__1108.invoke(client.clj:505)
    at clj_http.client$wrap_method$fn__1085.invoke(client.clj:464)
    at clj_http.cookies$wrap_cookies$fn__631.invoke(cookies.clj:118)
    at clj_http.links$wrap_links$fn__661.invoke(links.clj:50)
    at clj_http.client$wrap_unknown_host$fn__1117.invoke(client.clj:524)
    at clj_http.client$put.doInvoke(client.clj:633)
    at clojure.lang.RestFn.invoke(RestFn.java:423)
    at stream2es.es$put.invoke(es.clj:30)
    at stream2es.main$ensure_index.invoke(main.clj:310)
    at stream2es.main$main.invoke(main.clj:318)
    at stream2es.main$_main.doInvoke(main.clj:336)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at stream2es.main.main(Unknown Source)
2015-05-04T14:37:07.217+0200 ERROR unexpected exception: clojure.lang.ExceptionInfo: clj-http: status 400 {:object {:trace-redirects ["http://localhost:9200/wiki"], :request-time 808, :status 400, :headers {"content-type" "application/json; charset=UTF-8", "content-length" "360"}, :body "{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"mapping [_default_]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}},\"status\":400}"}, :environment {client #<client$wrap_output_coercion$fn__1005 clj_http.client$wrap_output_coercion$fn__1005@3f8f9dd6>, req {:request-method :put, :url "http://localhost:9200/wiki", :body "{\"settings\":{\"index.refresh_interval\":\"5s\",\"index.number_of_shards\":2,\"query.default_field\":\"text\",\"index.number_of_replicas\":0},\"mappings\":{\"_default_\":{\"_size\":{\"enabled\":true,\"store\":true},\"_all\":{\"enabled\":false}}}}"}, map__927 {:trace-redirects ["http://localhost:9200/wiki"], :request-time 808, :status 400, :headers {"content-type" "application/json; charset=UTF-8", "content-length" "360"}, :body "{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"mapping [_default_]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}},\"status\":400}"}, resp {:trace-redirects ["http://localhost:9200/wiki"], :request-time 808, :status 400, :headers {"content-type" "application/json; charset=UTF-8", "content-length" "360"}, :body "{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"mapping [_default_]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"Mapping definition for [_size] has unsupported parameters:  [store : true]\"}},\"status\":400}"}, status 400}}
2015-05-04T14:37:07.367+0200 INFO  00:01.370 0.0d/s 0.0K/s (0.0mb) indexed 0 streamed 0 errors 0

and this on the server side

[2015-05-04 14:37:07,157][DEBUG][action.admin.indices.create] [Count Nefaria] [wiki] failed to create
MapperParsingException[mapping [_default_]]; nested: MapperParsingException[Mapping definition for [_size] has unsupported parameters:  [store : true]];
    at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:382)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:383)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: MapperParsingException[Mapping definition for [_size] has unsupported parameters:  [store : true]]
    at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:285)
    at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:279)
    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:259)
    at org.elasticsearch.index.mapper.DocumentMapperParser.parseCompressed(DocumentMapperParser.java:211)
    at org.elasticsearch.index.mapper.DocumentMapperParser.parseCompressed(DocumentMapperParser.java:196)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:216)
    at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:379)
    ... 6 more
drewr commented 9 years ago

This was fixed in 978252e!