Enable cookies

[gapmiss]

It appears that any website that uses certain Cloudflare security checks returns:

Please enable cookies.

You are unable to access example.com

Why have I been blocked? This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.

What can I do to resolve this? You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.

Is it possible to enable cookies?

Thank You

[mrusme]

Thanks, could you provide me with a URL to test implementation?

[gapmiss]

examples: https://medium.com/@JasonWyatt/squeezing-performance-from-sqlite-explaining-the-virtual-machine-2550ef6c5db https://www.producthunt.com/posts/easyscrape

^^ could be "medium.com" sites

Also, reddit appears to blocks all requests.

https://www.reddit.com/r/mullvadvpn/comments/swimwp/what_does_the_malware_protection_on_20221beta_1_do/ https://www.reddit.com/r/mullvadvpn/comments/sxewrh/whatismyipaddress_showing_confirmed_proxy_ip/

Example of command I am running:

./reader -a "Safari: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" -i 'https://example.com' > ~/pkm/\@TEST/$KMVAR_Local__Title.md

Thx

[mrusme]

Please try again with latest master:

reader -a "Safari: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" https://medium.com/@JasonWyatt/squeezing-performance-from-sqlite-explaining-the-virtual-machine-2550ef6c5db

[gapmiss]

I am currently using the "reader_0.1.2_darwin_amd64.tar.gz" release. I do not have the capabilities to compile the source code. I will wait till you release the next version and download. Thank you again.

[gapmiss]

I've used the latest release(v0.1.3) and can report that "medium.com" and "producthunt.com" are now working great.

However, when trying the reddit pages again, they return what looks like encoded binary data.

For example:

./reader -a 'Safari: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15' \
-i https://www.reddit.com/r/mullvadvpn/comments/sxewrh/whatismyipaddress_showing_confirmed_proxy_ip/

returns this:

xœÜ½Ù–âH²(ú^\_Á®^uº²WˆÒ,‘yªÖfFÌó´W-¡ ��„Æý\[÷ý~Ù•„ \\€"#«ºOeEH>˜»›™››™›ý”rÿûßÿUhåû“v15·W

<< TRUNCATED >>

 iü;�žú®Ùq'ÃŽÓq½^NÐñ3c¶B¬hY\]6h�-gàMF=Ëí°þ7œ#�D¯=²†Ï�½Óñ&�Aß?7ûtíñQÁKJ‰Éx

I will keep testing w/ other websites and report any findings.

Thank you

[mrusme]

@gapmiss I believe I have fixed this issue with one of the latest releases. Please give it a try again sometime and report back if it is still happening.

[gapmiss]

version: 0.2.1

Seems to still be a cookie issue.

This (w/ and w/out the user-agent flag):

./reader -a 'Safari: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15' -i https://www.reddit.com/r/mullvadvpn/comments/sxewrh/whatismyipaddress_showing_confirmed_proxy_ip/

returns:

Blocked

reddit's awesome and all, but you may have a bit of a problem.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please contact us at this email address mailto:ratelimit@reddit.com?Subject=Blocked%20198.54.130.117.

when contacting us, please include your ip address which is: 198.54.130.117 and reddit account

I am able to view the above reddit URL in the browser without the blocking.

Let me know if I can test further.

[mrusme]

For reference:

• https://www.reddit.com/r/redditdev/comments/t8e8hc/getting_nothing_but_429_responses_when_using_go/
• https://www.reddit.com/r/redditdev/comments/uncu00/go_golang_clients_getting_403_blocked_responses/

@gapmiss are you able to reproduce this behavior on any other site but reddit?

[gapmiss]

are you able to reproduce this behavior on any other site but reddit?

@mrusme ~ No, I have not experienced this behavior w/ any other sites

[AbeEstrada]

@mrusme I can reproduce using v0.3.0 with this link:

https://gearmoose.com/best-tactical-gifts-2/

[AbeEstrada]

I don't know if Cookies is enough to access to these sites, some require JavaScript:

https://www.bloomberg.com/news/articles/2022-11-02/apple-to-keep-qualcomm-chips-in-2023-in-reversal-of-expectations

[mrusme]

I'd recommend using the following command until this was fixed:

wget -O - https://gearmoose.com/best-tactical-gifts-2/ | reader -i -

[mrusme]

I have just release v0.4.0 which includes a fix that should work for most sites. I've tested the gearmoose and bloomberg examples from above and they were loading.

I'll close this issue for now, as there is...

• this fix
• the wget/curl pipe workaround described above

In case anyone feels like the majority of sites still won't work, feel free to re-open it!