Response content ends with ^M

[rejeep]

Hi,

It seems elnode adds a ^M and the end of the response. To reproduce, save this to server.el:

(require 'elnode)

(defun some-handler (httpcon)
  (elnode-http-start httpcon 200 '("Content-Type" . "application/octet-stream"))
  (elnode-send-file httpcon load-file-name))

(elnode-start 'some-handler :port 8028 :host "localhost")

(while t (sit-for 100))

Run with:

$ emacs --script server.el -Q

Eval in Emacs:

(pop-to-buffer (url-retrieve-synchronously "http://127.0.0.1:8028"))

The buffer will contain (note the ^M at the end):

HTTP/1.1 200 Ok
Transfer-Encoding: chunked
Content-Type: application/octet-stream

(require 'elnode)

(defun some-handler (httpcon)
  (elnode-http-start httpcon 200 '("Content-Type" . "application/octet-stream"))
  (elnode-send-file httpcon load-file-name))

(elnode-start 'some-handler :port 8028 :host "localhost")

(while t (sit-for 100))
^M

[nicferrier]

Have you tried with curl? I am pretty confident about Elnode's chunked encoding implementation. The end of the stream is signaled with:

0\r\n\r\n

If a user agent isn't interpreting chunked encoding properly this can be a problem. If you packet trace the elnode traffic I bet you'll see it doing it properly.

If url-retrieve is broken, sorry. I have an alternate client called "web" that I wrote because of problems with url-retrieve.

[rejeep]

It can be an issue with url.el, for sure. I'm not sure curl will help since the ^M is not visible outside of Emacs and copying the output from curl don't necessarily preserve it.

I'm using elnode for a fake ELPA server. The reason I filed the issue here is because if I do this:

(pop-to-buffer (url-retrieve-synchronously "http://melpa.milkbox.net/packages/wrap-region-20130529.728.el"))

I don't get the ^M.

[sabof]

I also had bad experience using url-retrieve-synchronously directly. url-copy-file', andurl-insert-file-contents' seem to handle strange characters better.

[nicferrier]

@rejeep what Melpa does has nothing to do with it because Melpa is serving a file with a content length which means that no chunked encoding is done. I suspect it's chunked encoding which is the problem for 'url. I use chunked encoding all the time with elnode, I basically couldn't build any of the streaming apps I build without it so I'm really confident it is not broken.

What you could do is collect the data together, render it to whatever form it's going to be sent as, and then take it's content length and pass that to elnode in the elnode-http-start call. Then elnode will not use chunked encoding and all will be well.

It's much less efficient of course.

[rejeep]

Allright, I actually tried to set the content length, but I used elnode-send-file instead of elnode-http-return. Works better now!

Thanks for the help!