HTTP 403 response from powerthesaurus.org

[magne-hov]

I've started receiving HTTP 403 errors for the requests that https://github.com/SavchenkoValeriy/emacs-powerthesaurus performs.

F.ex for M-x powerthesaurus-lookup-synonyms-dwim RET the RET I get the following messages:

[error] request--callback: peculiar error: 403
[error] request-default-error-callback: https://www.powerthesaurus.org/the/synonyms error

This reproduces with GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0):

(load-file "/home/mhov/.emacs.d/straight/repos/emacs-request/request.el")
(load-file "/home/mhov/.emacs.d/straight/build/s/s.el")
(load-file "/home/mhov/.emacs.d/straight/repos/emacs-powerthesaurus/powerthesaurus.el")

I'd be interested to know if others see the same thing.

[magne-hov]

When doing similar GET queries with curl I also get HTTP 403, and the following

HTML content

```

www.powerthesaurus.org

Checking if the site connection is secure

www.powerthesaurus.org needs to review the security of your connection before proceeding.

Ray ID: 781b532c0ebe7330

Performance & security by Cloudflare

```

The following excerpt might be relevant to a potential upstream change?

Enable JavaScript and cookies to continue

[bridgesense]

That's a shame. Sometimes, you can spoof the system to think it's a browser, but the javascript block is probably a deal breaker. What a sad way to end! This was such an excellent plugin! :(

EDIT: Looking at the code, the web dev's are actively preventing this type of traffic altogether.

EDIT 2: As a writer, I really appreciate the wordnut extension. Until there might be a workaround or concession by the website dev, wordnet is pretty great.

[mj-ramirez]

Interestingly it still works on my M1 setup but not on the intel OSX. Go figure...

[magne-hov]

Interestingly it still works on my M1 setup but not on the intel OSX.

Interesting. Does it also work from your M1 setup if your use curl directly?

[mj-ramirez]

Same curl output on Intel OSX (bigsur), request-default-error-callback in emacs, and M1 OSX (monterey), still working as expected in emacs. mirror straight.el .emacs init file for both systems.

~ % curl -I https://www.powerthesaurus.org/echo

HTTP/2 403 date: Fri, 06 Jan 2023 21:48:36 GMT content-type: text/html; charset=UTF-8 permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=() referrer-policy: same-origin x-frame-options: SAMEORIGIN cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 expires: Thu, 01 Jan 1970 00:00:01 GMT report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=c%2BnStweRWPWVDzDjxF8fi1Z3Nm%2Bccxe0ILc3Z2Unhzroip4AqKWDmQxdi1P%2F%2BLHTN263iIWt1vUXlZD8wvs0BuLeVnGZJhcAz7hogJLhuTuxpUQANXEAvgaR4Y3Yd%2BWjUzzF9rK9JPw%3D"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} server: cloudflare cf-ray: 7857a8a76e798e14-LHR

[mj-ramirez]

I have updated the OSX BigSur System via Brew with curl 7.87.0 (x86_64-apple-darwin20.6.0) and checked that emacs is using that version via its shell (some /etc/paths editing required). Unfortunately same error message [error] request-default-error-callback:. Powerthesaurus.el still works flawless on M1 setup.

[ziova]

is there any alternative thesaurus that users can use until this is addressed?

[SavchenkoValeriy]

Hey folks, thanks for reporting this! The only system that I have is the M1 Mac (Monterey), so it works for me. I'm pretty sure that we can "dress up" the request so it works in other setups as well. NOTE: it's so weird that this bug is system dependent.

[SavchenkoValeriy]

curl has builtin support for cookies. On my system, even though we don't specify anything special for it, request.el still adds cookie flags to the curl command. So, it runs something like:

curl --silent --location --cookie ./curl-cookie-jar --cookie-jar ./curl-cookie-jar --include --compressed --header "User-Agent: Chrome/74.0.3729.169" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" --header "Accept-Language: en,en-US;q=0.5" --header "Upgrade-Insecure-Requests: 1" --url https://www.powerthesaurus.org/marvelous/synonyms

Can anyone facing this problem try this request and see if this request has the same outcome? (Maybe --cookie flags simply got lost). Then it would be an easier problem. If not, we'll need to somehow "convince" powerthesaurus that we enabled Javascript.

[bridgesense]

It looks good to me. :)

I'm not sure how that translates to your code yet, though.

[SavchenkoValeriy]

It looks good to me. :)

I'm not sure how that translates to your code yet, though.

Just to double-check, this is the HTML page that you got with that curl request I posted before on a system where emacs-powerthesaurus gets error 403? Am I right here?

If this is so, can you please execute powerthesaurus-debug-connection, request the synonyms for "marvelous", and send back everything related to power thesaurus from your *Messages* buffer? Especially the message that looks like a curl request. Thanks!

[bridgesense]

Just to clarify. Yes, I ran your line above and outputted it to a local HTML file. that is what you are looking at.

I've downloaded the latest version of powerthesaurus, unfortunately, that command isn't available to me.

[SavchenkoValeriy]

Just to clarify. Yes, I ran your line above and outputted it to a local HTML file. that is what you are looking at.

I've downloaded the latest version of powerthesaurus, unfortunately, that command isn't available to me.

powerthesaurus-debug-connection is not interactive, so you can't call it via M-x. You can evaluate it directly from minibuffer M-; (powerthesaurus-debug-connection), or in the *scratch* buffer by pasting (powerthesaurus-debug-connection) and using C-x C-e after the closing parenthesis.

Sorry, for the confusion, I should've been clearer.

[mnazaal]

Hi, I've been having the same issue. Finding synonyms for "marvelous" after running powerthesaurus-debug-connection gives the following:

1. HTML part alone:
2. Actual output:

   
   debug
   [debug] request--curl: --silent --location --cookie /<<MYUSER>>/.emacs.d/request/curl-cookie-jar --cookie-jar /<<MYUSER>>/.emacs.d/request/curl-cookie-jar --include --write-out \n(:num-redirects %{num_redirects} :url-effective "%{url_effective}") --compressed --header User-Agent: Chrome/74.0.3729.169 --header Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 --header Accept-Language: en,en-US;q=0.5 --header Upgrade-Insecure-Requests: 1 --url https://www.powerthesaurus.org/marvelous/synonyms
   [debug] request--curl-callback: event finished

[debug] request--callback: UNPARSED HTTP/2 403 date: Sat, 11 Mar 2023 18:08:35 GMT content-type: text/html; charset=UTF-8 permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=() referrer-policy: same-origin x-frame-options: SAMEORIGIN cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 expires: Thu, 01 Jan 1970 00:00:01 GMT report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=yQ8AZJL0kOO1zysEn3zK4foc67ibxslvlf7h1UqlCh5Q0KDCBxQxW9nDEvFilq93HVP1roS4u20lIC43lwjKzegrmC1xIlsz%2FzPDRE0BIgCpMh3BDTrRCyYWbYeZsYjOSq30jBXQKdY%3D"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} vary: Accept-Encoding server: cloudflare cf-ray: 7a65be5e3f2d3766-HEL content-encoding: br

<>

[error] request--callback: peculiar error: 403 [debug] request--callback: executing error [error] request-default-error-callback: https://www.powerthesaurus.org/marvelous/synonyms error

[SavchenkoValeriy]

A tiny update, I reproduced the issue on a local machine and in CI: https://github.com/SavchenkoValeriy/emacs-powerthesaurus/actions/runs/4392647435 It does look like this problem manifests itself only on Intel Macs (at least judging from this issue and CI). I managed to get curl requests working correctly with GraphQL requests from https://github.com/clarencecastillo/alfred-powerthesaurus. I'll rewrite the package to use API. Hopefully it should fix the issue for some time.

[SavchenkoValeriy]

Hey folks, #25 should resolve the issue (at least it did in CI). I'd really appreciate if you can update to version 0.3.0 and confirm. Cheers!

[mnazaal]

Hi, thanks for the quick reply. I have updated to 0.3.0 (via straight.el - confirmed the version via find-library).

I get the following error now when using powerthesaurus-lookup-dwim (trace from toggle-debug-on-error below):

Debugger entered--Lisp error: (error "Unknown query type ’:synonyms’")
  signal(error ("Unknown query type ’:synonyms’"))
  error("Unknown query type '%s'" ":synonyms")
  powerthesaurus--query("powerthesaurus" ":synonyms" #f(compiled-function (results) #<bytecode 0x1592866d1aa5>))
  powerthesaurus-lookup("powerthesaurus" ":synonyms" nil nil)
  powerthesaurus-lookup-dwim(nil)
  funcall-interactively(powerthesaurus-lookup-dwim nil)
  call-interactively(powerthesaurus-lookup-dwim record nil)
  command-execute(powerthesaurus-lookup-dwim record)
  execute-extended-command(nil "powerthesaurus-lookup-dwim" nil)
  funcall-interactively(execute-extended-command nil "powerthesaurus-lookup-dwim" nil)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Other functions (powerthesaurus-lookup-synonyms-dwim, powerthesaurus-lookup-definitions-dwim etc) work fine, thanks :)

Just in case : I'm on Ubuntu 20.04

[SavchenkoValeriy]

@mnazaal Whoops, that's my bad. I'll fix it in a second. Update to 0.3.1 and it's going to be alright. Closing this issue since it got resolved.

[mj-ramirez]

Intel Mac back on track 👍 :)