Unicode issues

[andrelaszlo]

When I submit the following query, that works in Sense:

GET /<redacted>/_search?pretty
{
  "query": {
    "match": {
      "my_field": "☃"
    }
  },
  "size": 1
}

I get this error:

Debugger entered--Lisp error: (error #("Multibyte text in HTTP request: GET /<redacted>/_search?pretty HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: localhost:9200
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs
Content-Type: application/json; charset=UTF-8
Content-length: 79

{
  \"query\": {
    \"match\": {
      \"my_field\": \"\342\230\203\"
    }
  },
  \"size\": 1
}
" 32 35 (fontified t face font-lock-builtin-face)))
  signal(error (#("Multibyte text in HTTP request: GET /<redacted>/_search?pretty HTTP/1.1
\nMIME-Version: 1.0
\nConnection: keep-alive
\nExtension: Security/Digest Security/SSL
\nHost: localhost:9200
\nAccept-encoding: gzip
\nAccept: */*
\nUser-Agent: URL/Emacs
\nContent-Type: application/json; charset=UTF-8
\nContent-length: 79
\n
\n{\n  \"query\": {\n    \"match\": {\n      \"my_field\": \"\342\230\203\"\n    }\n  },\n  \"size\": 1\n}\n" 32 35 (fontified t face font-lock-builtin-face))))
  error("Multibyte text in HTTP request: %s" #("GET /<redacted>/_search?pretty HTTP/1.1
\nMIME-Version: 1.0
\nConnection: keep-alive
\nExtension: Security/Digest Security/SSL
\nHost: localhost:9200
\nAccept-encoding: gzip
\nAccept: */*
\nUser-Agent: URL/Emacs
\nContent-Type: application/json; charset=UTF-8
\nContent-length: 79
\n
\n{\n  \"query\": {\n    \"match\": {\n      \"my_field\": \"\342\230\203\"\n    }\n  },\n  \"size\": 1\n}\n" 0 3 (face font-lock-builtin-face fontified t)))
  url-http-create-request()
  url-http([cl-struct-url "http" nil nil "localhost" 9200 "/<redacted>/_search?pretty" nil nil t nil t] es-result--handle-response (nil "*ES: es-queries.es*"))
  url-retrieve-internal("http://localhost:9200/<redacted>/_search?pretty" es-result--handle-response (nil "*ES: es-queries.es*") nil nil)
  url-retrieve("http://localhost:9200/<redacted>/_search?pretty" es-result--handle-response ("*ES: es-queries.es*"))
  es--execute-region()
  es-execute-request-dwim(nil)
  funcall-interactively(es-execute-request-dwim nil)
  call-interactively(es-execute-request-dwim nil nil)
  command-execute(es-execute-request-dwim)

I'm using GNU Emacs 25.1.1, ES

[dakrone]

Thanks for reporting this, I'll take a look!

[dakrone]

After doing some research on this and playing around, I think maybe switching to request might be the easiest solution here. I'll play around with it (shouldn't be too hard to switch)

[andrelaszlo]

Looks like a really nice lib, what are you using now?

[dakrone]

what are you using now?

Right now I used Emacs' built-in url.el to execute the request

[dakrone]

@andrelaszlo I pushed a branch (switch-to-request) that switches to using request.el and also fixes this unicode issue for .es files. I need to switch ob-elasticsearch.el (org babel) to use it as it still has the unicode issue, but when I do then I'll push a new version.

Feel free to check it out and make sure it works for you!

[greenonion]

@dakrone hello, I thought I'd post here instead of creating a new issue!

After the switch to request.el, requests containing unicode payload work fine. However, the response is not displayed correctly. I've tried to look into it a bit. It turns out that request.el specifies (set-process-coding-system proc 'binary 'binary) before issuing the request to curl (see here). If you remove this line, the output gets parsed correctly. However I think we might want the coding system of the request to actually be set to 'binary.

It turns out that we can handle this inside es-mode.el. I gave it a shot as a proof of concept and it seems to work. However my elisp skills are mediocre at best and that's why I'm not actually opening a PR.

The relevant change is in function es--execute-string. Similarly to the :data handler, we can change the :complete handler to:

(cl-function
  (lambda (&key data response error-thrown &allow-other-keys)
    (with-current-buffer results-buffer
      (let ((utf-data (decode-coding-string data 'utf-8)))
        (es-result--handle-response utf-data response error-thrown)))))

and the response gets parsed correctly!

Let me know if I can do anything more to help - including opening a PR. Oh, and thanks for your great work on this plugin!

[dakrone]

Hi @greenonion, definitely opening a PR would be great!

[greenonion]

Great, will do!