search

dash-docs-el / helm-dash

Browse Dash docsets inside emacs
511 stars 59 forks source link

Cannot retrieve list of official docsets #178

Open dchrzanowski opened 5 years ago

dchrzanowski commented 5 years ago

I've narrowed down that helm-install-docset executes helm-dash-read-json-from-url that seems to fail to retrieve data from "https://api.github.com/repos/Kapeli/feeds/contents/" with the error: Wrong type argument: integer-or-marker-p, nil. Possibly a change in the way url-http-end-of-headers is set/not-set?

kidd commented 5 years ago

This looks like a dup of #132. It is a connectivity issue trying to fetch that json.

I can't reproduce it in my machine, and even we could add some protection so it doesn't crash hard (PRs accepted), I'm not sure it'd solve the real issue.

dchrzanowski commented 5 years ago

I'll try again today, see if there is problem in reproducing the error.

buhudu commented 5 years ago

Same problem here! Any suggestions?

kidd commented 5 years ago

depending on your elisp debugging skills you can try to call edebug-defun on the breaking function and try to see why the variable url-http-end-of-headers is nil.

There's this reported bug https://emacs.stackexchange.com/questions/32950/why-isnt-url-http-end-of-headers-set-correctly-when-using-url-automatic-caching, but the behavior is different, so it being nil probably means we got no content at all from the request. But there's no reason why you're getting failures from downloading from that url.

dchrzanowski commented 5 years ago

Yes I've already looked at this post from stack before. There really is no reason to not get any data back since the url is live.

buhudu commented 5 years ago

Weird, when I step through the debugger on url-retrieve in url.el I can get to the point where I select a docset to install. But then I get a "Bad URL" error.

Outside of the debugger I don't even make it to that step.

dchrzanowski commented 5 years ago

That's some odd behaviour. My Elisp fu is not good enough to tackle this.

buhudu commented 5 years ago

It works. Thank you.

dchrzanowski commented 5 years ago

Alternative:

You can also go to the list of feeds here. Click the desired xml file. Then download the tgz file from any of the url's from the xml file. Once you have the file downloaded, you can run M-x helm-dash-install-docset-from-file and point it to the downloaded file.

kidd commented 5 years ago

Here's another alternative. I rewrote the function so that it uses curl but returns the same result. Works on my machine (TM)

(defun helm-dash-read-json-from-url (url)
  (shell-command (concat "curl -s " url) "*helm-dash-download*")
  (with-current-buffer "*helm-dash-download*"
    (json-read)))
jeeger commented 5 years ago

I've replaced all the URL functions with calls to curl, since it seems curl's proxy handling is a bit better than url.el's. The change is available here in the windows-fixes branch (should also work on Linux☺).

jbaum98 commented 5 years ago

I think that this a problem with url-retrieve-synchronously. This works:

(url-retrieve "https://api.github.com/repos/Kapeli/feeds/contents"
  (lambda (status) (switch-to-buffer (current-buffer)))))

As a side note, if we do want to switch to curl I think we should use request.el

jeeger commented 5 years ago

I think that this a problem with url-retrieve-synchronously. This works:

(url-retrieve "https://api.github.com/repos/Kapeli/feeds/contents"
  (lambda (status) (switch-to-buffer (current-buffer)))))

Nope, this fails here as well.

As a side note, if we do want to switch to curl I think we should use request.el

Yes, that would make much more sense. I'm not familiar with request.el, but I'll have a look at it if my ad-hoc curl patching fails ☺.

jbaum98 commented 5 years ago

Nope, this fails here as well.

What do you mean by "here"? I just meant that evaluating that using eval-expression gives me a buffer with a response in it, whereas

(switch-to-buffer (url-retrieve-synchronously "https://api.github.com/repos/Kapeli/feeds/contents"))

doesn't work.

jeeger commented 5 years ago

I mean that evaluating this expression does not return any content on this PC, since url.el is broken for using HTTPS over a proxy. This is why I've worked around url.el by using curl.

jbaum98 commented 5 years ago

Maybe I've misunderstood, but this makes me think that the proxy is not the issue. Because url-retrieve works for me even though url-retrieve-synchronously doesn't. So at least there are 2 issues: the proxy messes up everything, but even without the proxy I'm experiencing this issue with helm-dash.

jeeger commented 5 years ago

Interesting, here it's definitely the proxy. request.el definitely looks like a good alternative here, maybe gated with an availability check so it's only used when installed.

kidd commented 5 years ago

@jeeger , I'd buy into it if it's gated. I don't want to make an extra dependency for everyone.

Total naive question. Is a fix for proxy over http possible by plumbing some elisp? or we'd need hardcore networking knowledge for that? upstream emacs would benefit from this fix if we'd have it.

jeeger commented 5 years ago

I've had a look at the url.el elisp, and it's a tangled mess for me. I think that bug that might well have it's origin in the basic structure of url.el. TLS and Proxies are both kinds of "stream processors" that might not be able to be "stacked" recursively. I don't see me fixing url.el in the time I have available, even though it would of course be preferable to using an external package.

Using request.el would be much easier, and it could be easily gated behind a featurep invocation.

Edit: What makes this more complicated is that it seems this bug only occurs with redirects.