how to convert msearch request to be runnable from esrally

[TomonoriSoejima]

Suppose you had a request like this on Console in Dev tools.

GET hotel/_search
{
  "query": {
    "term": {
      "busy_month": {
        "value": "JANUARY"
      }
    }
  }
}

And pressing Command+I right on the text will give you

{"query":{"term":{"busy_month.keyword":{"value":"JANUARY"}}}}

Now that you need to escape the quotes when you want to run it from _msearch inside rally's configuration.

This process was a bit tiresome and after many attempts of string play I ended up with something like this.

$ echo '{"query":{"term":{"busy_month.keyword":{"value":"JANUARY"}}}}' | sed -e s,\",z-,g | tr -s 'z-' '\\"'

{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"JANUARY\"}}}}

$

Also, rally will want the request body to be a single line for _msearch

{
  "name": "msearch",
  "operation": {
    "operation-type": "raw-request",
    "path": "/hotel/_msearch",
    "header": {
      "Content-Type": "application/x-ndjson"
    },
    "body": "{}\n{"query": {"term": {"busy_month": "APRIL"}}}{}{"query": {"term": {"busy_month": "APRIL"}}}\n"
  },
  "warmup-iterations": 100,
  "iterations": 100
}

Note that the existence of newline \n there.

So but if you added it by hand and iterate like this, \n will get lost as you can see.

cat make.string.sh

for key in JANUARY FEBRUARY MARCH APRIL MAY JUNE JULY AUGUST SEPTEMBER OCTOBER NOVEMBER DECEMBER; do
    echo '{}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"month_name\"}}}}\n' | sed -e s,month_name,$key,
done

• result

$ sh make.string.sh | head -6 | paste -sd' '  -

{} {\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"JANUARY\"}}}}  {} {\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"FEBRUARY\"}}}}

$

So the workaround was to replace \n with other characters such as @ and I came up with this.

cat make.string.sh

for key in JANUARY FEBRUARY MARCH APRIL MAY JUNE JULY AUGUST SEPTEMBER OCTOBER NOVEMBER DECEMBER; do
    echo '{}\@{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"month_name\"}}}}\@' | sed -e s,month_name,$key,
done |  paste -sd' '  - | tr '@' 'n'

Then now you get the complete text. This should be very handy if you are dealing with a much more lengthy request.

$ sh make.string.sh

{}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"JANUARY\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"FEBRUARY\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"MARCH\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"APRIL\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"MAY\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"JUNE\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"JULY\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"AUGUST\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"SEPTEMBER\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"OCTOBER\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"NOVEMBER\"}}}}\n {}\n{\"query\":{\"term\":{\"busy_month.keyword\":{\"value\":\"DECEMBER\"}}}}\n

$

[TomonoriSoejima]

Attached is my rally config and it can be executed like esrally --track--path=./my_rally/ --pipeline=benchmark-only --target-hosts=localhost:9200

my_rally.zip

[TomonoriSoejima]

But I ended up in using https://codebeautify.org/json-escape-unescape to do it after all.