search

alphapapa / plz.el

An HTTP library for Emacs
GNU General Public License v3.0
187 stars 9 forks source link

Persistent `Curl error: 23 Write error.` for search.brave.com URL? #46

Closed 9viz closed 4 months ago

9viz commented 4 months ago

Hello Adam,

I am trying to use plz for GET-ing a URL but only for these URLs do I always get back a "Curl error: 23. Write error.":

(require 'plz)
(require 'eww)
(require 'url-http)

(plz 'get "https://search.brave.com/search?q=emacs+lisp" :as 'buffer
  :decode nil
  :timeout 6000
  :headers `(("Accept" . ,eww-accept-content-types)
     ("User-Agent" . ,(url-http--user-agent-default-string))))

It does not matter if I change the query in the URL to something else, it always fails with the backtrace:

Debugger entered--Lisp error: (plz-curl-error "Curl error" #s(plz-error :curl-error (23 . "Write error.") :response nil :message nil))
  signal(plz-curl-error ("Curl error" #s(plz-error :curl-error (23 . "Write error.") :response nil :message nil)))
  (if (let* ((cl-x data)) (progn (or (let* ((cl-x cl-x)) (progn (and (memq ... cl-struct-plz-error-tags) t))) (signal 'wrong-type-argument (list 'plz-error cl-x))) (aref cl-x 2))) (signal 'plz-http-error (list "HTTP error" data)) (signal 'plz-curl-error (list "Curl error" data)))
  (let ((data val)) (if (let* ((cl-x data)) (progn (or (let* ((cl-x cl-x)) (progn (and ... t))) (signal 'wrong-type-argument (list 'plz-error cl-x))) (aref cl-x 2))) (signal 'plz-http-error (list "HTTP error" data)) (signal 'plz-curl-error (list "Curl error" data))))
  (if (let* ((cl-x val)) (progn (and (memq (type-of cl-x) cl-struct-plz-error-tags) t))) (let ((data val)) (if (let* ((cl-x data)) (progn (or (let* (...) (progn ...)) (signal 'wrong-type-argument (list ... cl-x))) (aref cl-x 2))) (signal 'plz-http-error (list "HTTP error" data)) (signal 'plz-curl-error (list "Curl error" data)))) (let ((else val)) else))
  (let* ((val (process-get process :plz-result))) (if (let* ((cl-x val)) (progn (and (memq (type-of cl-x) cl-struct-plz-error-tags) t))) (let ((data val)) (if (let* ((cl-x data)) (progn (or (let* ... ...) (signal ... ...)) (aref cl-x 2))) (signal 'plz-http-error (list "HTTP error" data)) (signal 'plz-curl-error (list "Curl error" data)))) (let ((else val)) else)))
  (let ((inhibit-quit nil)) (if (and process stderr-process) nil (error "Process unexpectedly nil")) (while (accept-process-output process)) (while (accept-process-output stderr-process)) (if (eq :plz-result (process-get process :plz-result)) (progn (plz--sentinel process "finished\n") (if (eq :plz-result (process-get process :plz-result)) (progn (error "Plz: NO RESULT FROM PROCESS:%S  ARGS:%S" process rest))))) (let* ((val (process-get process :plz-result))) (if (let* ((cl-x val)) (progn (and (memq (type-of cl-x) cl-struct-plz-error-tags) t))) (let ((data val)) (if (let* ((cl-x data)) (progn (or ... ...) (aref cl-x 2))) (signal 'plz-http-error (list "HTTP error" data)) (signal 'plz-curl-error (list "Curl error" data)))) (let ((else val)) else))))
  (condition-case nil (let ((inhibit-quit nil)) (if (and process stderr-process) nil (error "Process unexpectedly nil")) (while (accept-process-output process)) (while (accept-process-output stderr-process)) (if (eq :plz-result (process-get process :plz-result)) (progn (plz--sentinel process "finished\n") (if (eq :plz-result (process-get process :plz-result)) (progn (error "Plz: NO RESULT FROM PROCESS:%S  ARGS:%S" process rest))))) (let* ((val (process-get process :plz-result))) (if (let* ((cl-x val)) (progn (and (memq ... cl-struct-plz-error-tags) t))) (let ((data val)) (if (let* (...) (progn ... ...)) (signal 'plz-http-error (list "HTTP error" data)) (signal 'plz-curl-error (list "Curl error" data)))) (let ((else val)) else)))) (quit (setq quit-flag t) (eval '(ignore nil) t)))
  (unwind-protect (condition-case nil (let ((inhibit-quit nil)) (if (and process stderr-process) nil (error "Process unexpectedly nil")) (while (accept-process-output process)) (while (accept-process-output stderr-process)) (if (eq :plz-result (process-get process :plz-result)) (progn (plz--sentinel process "finished\n") (if (eq :plz-result (process-get process :plz-result)) (progn (error "Plz: NO RESULT FROM PROCESS:%S  ARGS:%S" process rest))))) (let* ((val (process-get process :plz-result))) (if (let* ((cl-x val)) (progn (and ... t))) (let ((data val)) (if (let* ... ...) (signal ... ...) (signal ... ...))) (let ((else val)) else)))) (quit (setq quit-flag t) (eval '(ignore nil) t))) (if (eq as 'buffer) nil (kill-buffer process-buffer)) (kill-buffer (process-buffer stderr-process)))
  (if sync-p (unwind-protect (condition-case nil (let ((inhibit-quit nil)) (if (and process stderr-process) nil (error "Process unexpectedly nil")) (while (accept-process-output process)) (while (accept-process-output stderr-process)) (if (eq :plz-result (process-get process :plz-result)) (progn (plz--sentinel process "finished\n") (if (eq :plz-result ...) (progn ...)))) (let* ((val (process-get process :plz-result))) (if (let* (...) (progn ...)) (let (...) (if ... ... ...)) (let (...) else)))) (quit (setq quit-flag t) (eval '(ignore nil) t))) (if (eq as 'buffer) nil (kill-buffer process-buffer)) (kill-buffer (process-buffer stderr-process))) process)
  (let* ((data-arg (cond ((eq body-type 'binary) (let nil "--data-binary")) ((eq body-type 'text) (let nil "--data")) (t (let ((x6367 body-type)) (error "No clause matching `%S'" x6367))))) (curl-command-line-args (append plz-curl-default-args (list "--config" "-"))) (curl-config-header-args (let* ((--cl-var-- headers) (value nil) (key nil) (--cl-var-- nil)) (while (consp --cl-var--) (progn (setq value (car --cl-var--)) (setq key (car-safe ...))) (setq --cl-var-- (cons (cons "--header" ...) --cl-var--)) (setq --cl-var-- (cdr --cl-var--))) (nreverse --cl-var--))) (curl-config-args (append curl-config-header-args (list (cons "--url" url)) (if connect-timeout (progn (list (cons "--connect-timeout" ...)))) (if timeout (progn (list (cons "--max-time" ...)))) (cond ((eq method 'get) (let nil (list ...))) ((memq method '...) (let nil (list ... ... ...))) ((eq method 'delete) (let nil (list ... ...))) ((eq method 'head) (let nil (list ... ...)))))) (curl-config (let* ((--cl-var-- curl-config-args) (value nil) (key nil) (--cl-var-- "")) (while (consp --cl-var--) (progn (setq value (car --cl-var--)) (setq key (car-safe ...))) (setq --cl-var-- (concat --cl-var-- (format "%s \"%s\"\n" key value))) (setq --cl-var-- (cdr --cl-var--))) --cl-var--)) (decode (if (eq as 'binary) 'nil (let nil decode))) (default-directory temporary-file-directory) (process-buffer (generate-new-buffer " *plz-request-curl*")) (stderr-process (make-pipe-process :name "plz-request-curl-stderr" :buffer (generate-new-buffer " *plz-request-curl-stderr*") :noquery t :sentinel #'plz--stderr-sentinel)) (process (make-process :name "plz-request-curl" :buffer process-buffer :coding 'binary :command (append (list plz-curl-program) curl-command-line-args) :connection-type 'pipe :sentinel #'plz--sentinel :stderr stderr-process :noquery noquery)) sync-p) (if (eq 'sync then) (progn (progn (setq sync-p t) (setq then #'(lambda (result) (process-put process :plz-result result))) (setq else nil)))) (progn (let* ((v process)) (process-put v :plz-then (let* ((pcase-0 #'...)) (cond ((memq as ...) (let nil ...)) ((eq as ...) (let nil ...)) ((eq as ...) (let nil ...)) ((eq as ...) (let nil ...)) ((consp as) (let* ... ...)) ((functionp as) (funcall pcase-0)) (t (let ... ...)))))) (let* ((v process)) (process-put v :plz-else else)) (let* ((v process)) (process-put v :plz-finally finally)) (let* ((v process)) (process-put v :plz-sync sync-p)) (let* ((v process)) (process-put v :plz-args (apply #'list method url rest))) (let* ((v process)) (process-put v :plz-result :plz-result))) (process-send-string process curl-config) (if body (progn (cond ((stringp body) (process-send-string process body)) ((bufferp body) (save-current-buffer (set-buffer body) (process-send-region process (point-min) (point-max))))))) (process-send-eof process) (if sync-p (unwind-protect (condition-case nil (let ((inhibit-quit nil)) (if (and process stderr-process) nil (error "Process unexpectedly nil")) (while (accept-process-output process)) (while (accept-process-output stderr-process)) (if (eq :plz-result (process-get process :plz-result)) (progn (plz--sentinel process "finished\n") (if ... ...))) (let* ((val ...)) (if (let* ... ...) (let ... ...) (let ... else)))) (quit (setq quit-flag t) (eval '(ignore nil) t))) (if (eq as 'buffer) nil (kill-buffer process-buffer)) (kill-buffer (process-buffer stderr-process))) process))
  (progn (setq decode (if (and decode-s (not decode)) nil decode)) (setq headers (cons (cons "Expect" "") headers)) (let* ((data-arg (cond ((eq body-type 'binary) (let nil "--data-binary")) ((eq body-type 'text) (let nil "--data")) (t (let (...) (error "No clause matching `%S'" x6367))))) (curl-command-line-args (append plz-curl-default-args (list "--config" "-"))) (curl-config-header-args (let* ((--cl-var-- headers) (value nil) (key nil) (--cl-var-- nil)) (while (consp --cl-var--) (progn (setq value ...) (setq key ...)) (setq --cl-var-- (cons ... --cl-var--)) (setq --cl-var-- (cdr --cl-var--))) (nreverse --cl-var--))) (curl-config-args (append curl-config-header-args (list (cons "--url" url)) (if connect-timeout (progn (list ...))) (if timeout (progn (list ...))) (cond ((eq method ...) (let nil ...)) ((memq method ...) (let nil ...)) ((eq method ...) (let nil ...)) ((eq method ...) (let nil ...))))) (curl-config (let* ((--cl-var-- curl-config-args) (value nil) (key nil) (--cl-var-- "")) (while (consp --cl-var--) (progn (setq value ...) (setq key ...)) (setq --cl-var-- (concat --cl-var-- ...)) (setq --cl-var-- (cdr --cl-var--))) --cl-var--)) (decode (if (eq as 'binary) 'nil (let nil decode))) (default-directory temporary-file-directory) (process-buffer (generate-new-buffer " *plz-request-curl*")) (stderr-process (make-pipe-process :name "plz-request-curl-stderr" :buffer (generate-new-buffer " *plz-request-curl-stderr*") :noquery t :sentinel #'plz--stderr-sentinel)) (process (make-process :name "plz-request-curl" :buffer process-buffer :coding 'binary :command (append (list plz-curl-program) curl-command-line-args) :connection-type 'pipe :sentinel #'plz--sentinel :stderr stderr-process :noquery noquery)) sync-p) (if (eq 'sync then) (progn (progn (setq sync-p t) (setq then #'(lambda ... ...)) (setq else nil)))) (progn (let* ((v process)) (process-put v :plz-then (let* ((pcase-0 ...)) (cond (... ...) (... ...) (... ...) (... ...) (... ...) (... ...) (t ...))))) (let* ((v process)) (process-put v :plz-else else)) (let* ((v process)) (process-put v :plz-finally finally)) (let* ((v process)) (process-put v :plz-sync sync-p)) (let* ((v process)) (process-put v :plz-args (apply #'list method url rest))) (let* ((v process)) (process-put v :plz-result :plz-result))) (process-send-string process curl-config) (if body (progn (cond ((stringp body) (process-send-string process body)) ((bufferp body) (save-current-buffer (set-buffer body) (process-send-region process ... ...)))))) (process-send-eof process) (if sync-p (unwind-protect (condition-case nil (let ((inhibit-quit nil)) (if (and process stderr-process) nil (error "Process unexpectedly nil")) (while (accept-process-output process)) (while (accept-process-output stderr-process)) (if (eq :plz-result ...) (progn ... ...)) (let* (...) (if ... ... ...))) (quit (setq quit-flag t) (eval '... t))) (if (eq as 'buffer) nil (kill-buffer process-buffer)) (kill-buffer (process-buffer stderr-process))) process)))
  (progn (let ((--cl-keys-- rest)) (while --cl-keys-- (cond ((memq (car --cl-keys--) '(:headers :body :else :finally :noquery :as :then :body-type :decode :connect-timeout :timeout :allow-other-keys)) (if (cdr --cl-keys--) nil (error "Missing argument for %s" (car --cl-keys--))) (setq --cl-keys-- (cdr (cdr --cl-keys--)))) ((car (cdr (memq ... rest))) (setq --cl-keys-- nil)) (t (error "Keyword argument %s not one of (:headers :body :else :finally :noquery :as :then :body-type :decode :connect-timeout :timeout)" (car --cl-keys--)))))) (progn (setq decode (if (and decode-s (not decode)) nil decode)) (setq headers (cons (cons "Expect" "") headers)) (let* ((data-arg (cond ((eq body-type ...) (let nil "--data-binary")) ((eq body-type ...) (let nil "--data")) (t (let ... ...)))) (curl-command-line-args (append plz-curl-default-args (list "--config" "-"))) (curl-config-header-args (let* ((--cl-var-- headers) (value nil) (key nil) (--cl-var-- nil)) (while (consp --cl-var--) (progn ... ...) (setq --cl-var-- ...) (setq --cl-var-- ...)) (nreverse --cl-var--))) (curl-config-args (append curl-config-header-args (list (cons "--url" url)) (if connect-timeout (progn ...)) (if timeout (progn ...)) (cond (... ...) (... ...) (... ...) (... ...)))) (curl-config (let* ((--cl-var-- curl-config-args) (value nil) (key nil) (--cl-var-- "")) (while (consp --cl-var--) (progn ... ...) (setq --cl-var-- ...) (setq --cl-var-- ...)) --cl-var--)) (decode (if (eq as 'binary) 'nil (let nil decode))) (default-directory temporary-file-directory) (process-buffer (generate-new-buffer " *plz-request-curl*")) (stderr-process (make-pipe-process :name "plz-request-curl-stderr" :buffer (generate-new-buffer " *plz-request-curl-stderr*") :noquery t :sentinel #'plz--stderr-sentinel)) (process (make-process :name "plz-request-curl" :buffer process-buffer :coding 'binary :command (append (list plz-curl-program) curl-command-line-args) :connection-type 'pipe :sentinel #'plz--sentinel :stderr stderr-process :noquery noquery)) sync-p) (if (eq 'sync then) (progn (progn (setq sync-p t) (setq then #'...) (setq else nil)))) (progn (let* ((v process)) (process-put v :plz-then (let* (...) (cond ... ... ... ... ... ... ...)))) (let* ((v process)) (process-put v :plz-else else)) (let* ((v process)) (process-put v :plz-finally finally)) (let* ((v process)) (process-put v :plz-sync sync-p)) (let* ((v process)) (process-put v :plz-args (apply #'list method url rest))) (let* ((v process)) (process-put v :plz-result :plz-result))) (process-send-string process curl-config) (if body (progn (cond ((stringp body) (process-send-string process body)) ((bufferp body) (save-current-buffer ... ...))))) (process-send-eof process) (if sync-p (unwind-protect (condition-case nil (let (...) (if ... nil ...) (while ...) (while ...) (if ... ...) (let* ... ...)) (quit (setq quit-flag t) (eval ... t))) (if (eq as 'buffer) nil (kill-buffer process-buffer)) (kill-buffer (process-buffer stderr-process))) process))))
  (let* ((headers (car (cdr (plist-member rest ':headers)))) (body (car (cdr (plist-member rest ':body)))) (else (car (cdr (plist-member rest ':else)))) (finally (car (cdr (plist-member rest ':finally)))) (noquery (car (cdr (plist-member rest ':noquery)))) (as (car (cdr (or (plist-member rest ':as) '(nil string))))) (then (car (cdr (or (plist-member rest ':then) '(nil sync))))) (body-type (car (cdr (or (plist-member rest ':body-type) '(nil text))))) (decode-s (plist-member rest ':decode)) (decode (if decode-s (prog1 (car (cdr decode-s)) (setq decode-s t)) t)) (connect-timeout (car (cdr (or (plist-member rest ':connect-timeout) (list nil plz-connect-timeout))))) (timeout (car (cdr (or (plist-member rest ':timeout) (list nil plz-timeout)))))) (progn (let ((--cl-keys-- rest)) (while --cl-keys-- (cond ((memq (car --cl-keys--) '...) (if (cdr --cl-keys--) nil (error "Missing argument for %s" ...)) (setq --cl-keys-- (cdr ...))) ((car (cdr ...)) (setq --cl-keys-- nil)) (t (error "Keyword argument %s not one of (:headers :body :else :finally :noquery :as :then :body-type :decode :connect-timeout :timeout)" (car --cl-keys--)))))) (progn (setq decode (if (and decode-s (not decode)) nil decode)) (setq headers (cons (cons "Expect" "") headers)) (let* ((data-arg (cond (... ...) (... ...) (t ...))) (curl-command-line-args (append plz-curl-default-args (list "--config" "-"))) (curl-config-header-args (let* (... ... ... ...) (while ... ... ... ...) (nreverse --cl-var--))) (curl-config-args (append curl-config-header-args (list ...) (if connect-timeout ...) (if timeout ...) (cond ... ... ... ...))) (curl-config (let* (... ... ... ...) (while ... ... ... ...) --cl-var--)) (decode (if (eq as ...) 'nil (let nil decode))) (default-directory temporary-file-directory) (process-buffer (generate-new-buffer " *plz-request-curl*")) (stderr-process (make-pipe-process :name "plz-request-curl-stderr" :buffer (generate-new-buffer " *plz-request-curl-stderr*") :noquery t :sentinel #'plz--stderr-sentinel)) (process (make-process :name "plz-request-curl" :buffer process-buffer :coding 'binary :command (append ... curl-command-line-args) :connection-type 'pipe :sentinel #'plz--sentinel :stderr stderr-process :noquery noquery)) sync-p) (if (eq 'sync then) (progn (progn (setq sync-p t) (setq then ...) (setq else nil)))) (progn (let* ((v process)) (process-put v :plz-then (let* ... ...))) (let* ((v process)) (process-put v :plz-else else)) (let* ((v process)) (process-put v :plz-finally finally)) (let* ((v process)) (process-put v :plz-sync sync-p)) (let* ((v process)) (process-put v :plz-args (apply ... method url rest))) (let* ((v process)) (process-put v :plz-result :plz-result))) (process-send-string process curl-config) (if body (progn (cond (... ...) (... ...)))) (process-send-eof process) (if sync-p (unwind-protect (condition-case nil (let ... ... ... ... ... ...) (quit ... ...)) (if (eq as ...) nil (kill-buffer process-buffer)) (kill-buffer (process-buffer stderr-process))) process)))))
  plz(get "https://search.brave.com/search?q=emacs+lisp" :as buffer :decode nil :timeout 6000 :headers (("Accept" . "text/html, text/plain, text/sgml, text/css, application/xhtml+xml, */*;q=0.01") ("User-Agent" . "URL/Emacs Emacs/30.0.50 (X11; x86_64-pc-linux-gnu)")))
  (progn (plz 'get "https://search.brave.com/search?q=emacs+lisp" :as 'buffer :decode nil :timeout 6000 :headers (list (cons "Accept" eww-accept-content-types) (cons "User-Agent" (url-http--user-agent-default-string)))))
  eval((progn (plz 'get "https://search.brave.com/search?q=emacs+lisp" :as 'buffer :decode nil :timeout 6000 :headers (list (cons "Accept" eww-accept-content-types) (cons "User-Agent" (url-http--user-agent-default-string))))) t)
  elisp--eval-last-sexp(nil)
  eval-last-sexp(nil)
  (eros--eval-overlay (eval-last-sexp eval-last-sexp-arg-internal) (point))
  eros-eval-last-sexp(nil)
  funcall-interactively(eros-eval-last-sexp nil)
  call-interactively(eros-eval-last-sexp nil nil)
  command-execute(eros-eval-last-sexp)

However, if I leave out the query part, I get no curl error. For other URLs with a query part, no curl error is signalled e.g., https://www.google.com/search?q=asd.

If I try out the same thing in the shell as,

% echo "--header \"Expect: \"
--header \"User-Agent: URL/Emacs Emacs/30.0.50 (X11; x86_64-pc-linux-gnu)\"
--header \"Accept: text/html, text/plain, text/sgml, text/css, application/xhtml+xml, */*;q=0.01\"
--url \"https://search.brave.com/search?q=emacs+lisp\"
--connect-timeout \"5\"
--max-time \"60\"
--dump-header \"-\"
" |curl --silent --compressed --location --config -

then curl dumps the request as expected.

Is this a bad cockpit error, am I missing something very obvious? AFAICT, it doesn't have to do with the headers since it runs fine from the shell.

I am using plz @ f402bcc93446fe1629dd2e64d7e147e22fe034e8, and Emacs master (same error with 29.2).

alphapapa commented 4 months ago

According to the man page: 

23     Write error. Curl couldn't write data to a local filesystem or similar.

My analysis:

???

:)

Since it only seems to come from search.brave.com, you should probably look at the extreme level of debug info, including TLS stuff. Also, remove anything not necessary to reproduce the problem, standard debugging stuff.

9viz commented 4 months ago

Thanks for the pointer to the verbose flag. I completely forgot to check the exit code of the curl command when running from the shell, and it does indeed exit with 23.

Removing --compressed does the job for me, and it might be related to https://github.com/curl/curl/issues/5200. I will just end in 'Accept-Encoding: gzip' header and do the decompression in Emacs side like url-http does. Thank you again, and sorry about the noise.

alphapapa commented 4 months ago

No problem. Thanks.