What versions of elasticsearch is stream2es compatible with?

[maxrothman]

We're investigating using stream2es to migrate from ES 0.90 to 5.1. Is stream2es compatible with both versions?

[drewr]

Cross-version compat with the es stream hasn't been focused on, unfortunately. There's a chance you could --tee-bulk it with a 0.90-compatible version from that cluster and curl it to the 5.x cluster. GNU parallel makes this fast if you have many cores.

time find /path/to/bulks -name \*.gz \
   | parallel "curl -s -XPOST 5xcluster:9200/_bulk --data-binary @<(gzip -cd {}) >/dev/null"

That doesn't check for bulk errors, but you get the idea.

[damienalexandre]

I also have an error with 5.1:

java -jar stream2es es --target http://localhost:9200/wikipedia_copy --source http://localhost:9200/wikipedia
clojure.lang.ExceptionInfo: clj-http: status 400 {:request-time 16, :repeatable? false, :streaming? true, :chunked? true, :headers {"content-type" "application/json; charset=UTF-8", "transfer-encoding" "chunked"}, :orig-content-encoding nil, :status 400, :length -1, :body "{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"unknown setting [index.creation_date] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"unknown setting [index.creation_date] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\"},\"status\":400}", :trace-redirects ["http://localhost:9200/wikipedia_copy"]}
    at slingshot.support$stack_trace.invoke(support.clj:201)
    at clj_http.client$wrap_exceptions$fn__2502.invoke(client.clj:196)
    at clj_http.client$wrap_accept$fn__2662.invoke(client.clj:565)
    at clj_http.client$wrap_accept_encoding$fn__2668.invoke(client.clj:579)
    at clj_http.client$wrap_content_type$fn__2657.invoke(client.clj:555)
    at clj_http.client$wrap_form_params$fn__2739.invoke(client.clj:726)
    at clj_http.client$wrap_nested_params$fn__2753.invoke(client.clj:756)
    at clj_http.client$wrap_method$fn__2709.invoke(client.clj:670)
    at clj_http.cookies$wrap_cookies$fn__207.invoke(cookies.clj:124)
    at clj_http.links$wrap_links$fn__1129.invoke(links.clj:51)
    at clj_http.client$wrap_unknown_host$fn__2762.invoke(client.clj:776)
    at clj_http.client$put.invokeStatic(client.clj:897)
    at clj_http.client$put.doInvoke(client.clj:893)
    at clojure.lang.RestFn.invoke(RestFn.java:423)
    at stream2es.http$put.invokeStatic(http.clj:17)
    at stream2es.http$put.invoke(http.clj:16)
    at stream2es.es$fn__6447.invokeStatic(es.clj:155)
    at stream2es.es$fn__6447.invoke(es.clj:130)
    at stream2es.es$fn__6325$G__6295__6332.invoke(es.clj:17)
    at stream2es.main$ensure_index.invokeStatic(main.clj:313)
    at stream2es.main$ensure_index.invoke(main.clj:303)
    at stream2es.main$main.invokeStatic(main.clj:321)
    at stream2es.main$main.invoke(main.clj:317)
    at stream2es.main$_main.invokeStatic(main.clj:339)
    at stream2es.main$_main.doInvoke(main.clj:337)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at stream2es.main.main(Unknown Source)
2017-01-04T15:57:16.063+0000 ERROR unexpected exception: see above
2017-01-04T15:57:16.269+0000 INFO  00:00.357 0.0d/s 0.0K/s (0.0mb) indexed 0 streamed 0 errors 0

unknown setting [index.creation_date] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

So I just created the index myself and now I see this:

java -jar stream2es es --target http://localhost:9200/wikipedia_copy --source http://localhost:9200/wikipedia
clojure.lang.ExceptionInfo: clj-http: status 400 {:request-time 6, :repeatable? false, :streaming? true, :chunked? true, :headers {"content-type" "application/json; charset=UTF-8", "transfer-encoding" "chunked"}, :orig-content-encoding nil, :status 400, :length -1, :body "{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"No search type for [scan]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"No search type for [scan]\"},\"status\":400}", :trace-redirects ["http://localhost:9200/wikipedia/_search"]}
    at slingshot.support$stack_trace.invoke(support.clj:201)
    at clj_http.client$wrap_exceptions$fn__2502.invoke(client.clj:196)
    at clj_http.client$wrap_accept$fn__2662.invoke(client.clj:565)
    at clj_http.client$wrap_accept_encoding$fn__2668.invoke(client.clj:579)
    at clj_http.client$wrap_content_type$fn__2657.invoke(client.clj:555)
    at clj_http.client$wrap_form_params$fn__2739.invoke(client.clj:726)
    at clj_http.client$wrap_nested_params$fn__2753.invoke(client.clj:756)
    at clj_http.client$wrap_method$fn__2709.invoke(client.clj:670)
    at clj_http.cookies$wrap_cookies$fn__207.invoke(cookies.clj:124)
    at clj_http.links$wrap_links$fn__1129.invoke(links.clj:51)
    at clj_http.client$wrap_unknown_host$fn__2762.invoke(client.clj:776)
    at clj_http.client$get.invokeStatic(client.clj:879)
    at clj_http.client$get.doInvoke(client.clj:875)
    at clojure.lang.RestFn.invoke(RestFn.java:423)
    at stream2es.http$get.invokeStatic(http.clj:29)
    at stream2es.http$get.invoke(http.clj:28)
    at stream2es.es$scan1.invokeStatic(es.clj:113)
    at stream2es.es$scan1.invoke(es.clj:110)
    at stream2es.es$scan.invokeStatic(es.clj:127)
    at stream2es.es$scan.invoke(es.clj:124)
    at stream2es.stream.es$make_callback$fn__6689.invoke(es.clj:84)
    at stream2es.main$stream_BANG_.invokeStatic(main.clj:245)
    at stream2es.main$stream_BANG_.invoke(main.clj:241)
    at stream2es.main$main.invokeStatic(main.clj:333)
    at stream2es.main$main.invoke(main.clj:317)
    at stream2es.main$_main.invokeStatic(main.clj:339)
    at stream2es.main$_main.doInvoke(main.clj:337)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at stream2es.main.main(Unknown Source)
2017-01-04T16:00:20.351+0000 ERROR unexpected exception: see above
2017-01-04T16:00:20.546+0000 INFO  00:00.290 0.0d/s 0.0K/s (0.0mb) indexed 0 streamed 0 errors 0

No search type for [scan]

I know there is a _reindex API now, but this tool is useful when dealing with two separated clusters and it looks like it's not compatible with Elasticsearch 5.