Can't install an npm package because the server considers me a bot (403 forbidden via 4G)

devoln commented 5 months ago

Current behavior

Scripts (including npm install) can't download the link https://download.cypress.io/desktop/13.6.6?platform=linux&arch=x64 because of bot protection.

Debug logs

119.3 npm ERR! [TITLE]  Downloading Cypress      0% 0s
119.3 npm ERR! The Cypress App could not be downloaded.
119.3 npm ERR!
119.3 npm ERR! Does your workplace require a proxy to be used to access the Internet? If so, you must configure the HTTP_PROXY environment variable before downloading Cypress. Read more: https://on.cypress.io/proxy-configuration
119.3 npm ERR!
119.3 npm ERR! Otherwise, please check network connectivity and try again:
119.3 npm ERR!
119.3 npm ERR! ----------
119.3 npm ERR!
119.3 npm ERR! URL: https://download.cypress.io/desktop/13.6.6?platform=linux&arch=x64
119.3 npm ERR! Error: Failed downloading the Cypress binary.
119.3 npm ERR! Response code: 403
119.3 npm ERR! Response message: Forbidden
119.3 npm ERR!
119.3 npm ERR! ----------

Cypress Version

13.6.6

Node version

18

Package Manager

npm

Package Manager Version

10.2.4

Operating system

Linux

Operating System Version

Ubuntu 22.04

Other

I use 4G modem every day because I don't have access to cable internet connection where I live. And it seems that my mobile operator puts a lot of users behind a single IP address which makes many websites to think that I'm a bot. I often see captchas and Cloudflare's managed challenges when browsing but it has never prevented me from installing packages. Until today.

And now I want to run a Docker container to host locally Compiler Explorer. I ran docker build and it failed because it can't download Cypress. I tried to open the link from the npm log manually in my browser. First I saw something like "Are you a bot?" but in a few seconds the download started. So I can download the file but I don't know how to use it to successfully build the container. I tried to repeat building the image many times, but it always fails.

For me it doesn't seem logical to have a bot protection for files that are expected to be downloaded by scripts. Could use consider removing it?

I can't even use VPN to avoid it because my provider blocks VPN protocols.

jennifer-shehane commented 5 months ago

@devoln I'm sorry to hear about the restrictions you're having to deal with. Are you under any corporate proxy? Is it possible this is something on your end? (I don't think the error looks that way to me).

I'm not sure if we have geo-based blocking, but that could be a possibility depending on your location.

There are some advanced installation guides, like for mirroring the location of download. If you can download it manually maybe there's some way to set it up that way. https://docs.cypress.io/guides/references/advanced-installation

csvan commented 5 months ago

This does not look like bot protection, it looks like your proxy is blocking the download

devoln commented 5 months ago

I work remotely from home and I haven't setup any proxy here. It is definitely a bot protection set on your domain with Cloudflare. When I opened your link I got this "Are you a bot" Cloudflare check again. It finishes automatically in browser without requiring me to solve captchas or checking "I'm not a robot" checkbox but it prevents automatic downloads from happening.

You probably have activated some sort of DDoS protection on your domain and subdomain cypress.io and Cloudflare may have some internal logic how to detect bots. I also use Cloudflare so I know that there are a lot of things to tweak. You can probably add an exception to remove download.cypress.io and cdn.cypress.io from checks.

I tried wget this URL from Windows Powershell and got an interesting output containing something about Cloudflare challenges:

wget "https://download.cypress.io/desktop/13.6.6?platform=linux&arch=x64"

wget :

Enable JavaScript and cookies to continue

(function(){window._cf_chl_opt={cvId: '3',cZone: "download.cypress.io",cType: 'managed',cNounce: '96030',cRay: '8636c63e29cd690a',cHash: '955982906fc1f14',cUPMDTk: "\/desktop\/13.6.6?platform=linux&arch=x64&cf_chl_tk=PMkbqYVWjmlFOqwk hrobrmJrG5QkOj3.Z.wHPTwlZM8-1710277862-0.0.1.1-1578",cFPWv: 'g',cTTimeMs: '1000',cMTimeMs: '375000',cTplV: 5,cTplB: 'cf',cK: "visitor-time",fa: "\/desktop\/13.6.6?platform=linux&arch=x64&cf_chl_f_tk=PMkbqYVWjmlFOqwkhrobrmJrG5QkOj3.Z. wHPTwlZM8-1710277862-0.0.1.1-1578",md: "BJS6XrfhPhy0.E0i9L3ra8Q1tH8hcuPJ4U_Vvztshhk-1710277862-1.1.1.1-EqHSyGtIsDnbHxIt7vOb3.4_lHOrxh36Qw6SVyA2cdHY_abVW1b_uWxnmOpAdIXzTP8PPFpeskxWk0OVhkf.nbtB4jIM1wOZqLU4hO2t7reokDe3VbK0VL5FoPrZ3pIgQ2CC hA_JmCZEzZhP7av1EyqZF8dsdA8SgmSkbSphFzx31rcoHX_TW9W_kshv8TX.m7CUSX.4471kTJQ1Sw_S_kHUbPVHDFVibXNdiGj1OrO3t7FKWDSpC3HnTBoPeIkReDrUnOKWEhgxvluKICpWmtStzmEOTbgYhSTcyqeqs6u3JsH60vfbmhcZNFszF8ETlQHx6CvxoioU66FveXCLzPZ3ELN83GHizZxlhPIBHoNgcUR .ss8eb01r_BLKmDASuEe1lXpgh2ck136uSlKbRWqc4Ke9L.pp0m0HfHTeJRdAfcdteFhpTZ_SdCsCzw3Jud0VOKIGN5P.DaJBa.eeCxzWtskBIKZVEMMGta4VzNXHKfqtQ5Djmnz2zpTJQ_d3M26M7h5VPzpwJEYvB6T4MPNiuJ8Xji79OPiFW6795JFxVzySVlNA3Cmo9QpLVwmWj3hvWmuEMR2LLdSOFi0lVCf7OJ I0WcjoYZBbqxWAiuJVg7PEiCiwUeAcJP3nZgeyiCIb0YVoTXcGNkSQXnCD2RLngR_RWisAsi3Q3w0_hNRCFKLBThQpUg5XhkeReQd7TTIfwrLebtRQPMfZnTCOV22wbR2waoIF.AkhZW8gxnGvm0BmiOQPaNmTf_HfnRqmjev4nchrG.BmrloqfvNNGS4pmm7w6zezK5qw80XBnVNkV1WL4N07m6uU7HBciCMvbscPf JSlEeIeKKI861aZ9LOsM8Uu8_znYqEbyJF1qfsV0vIwOoUEqIwZ.VZmHayGX9rQvNmRjRAyma7Jc4BnmY3GWC8czypWhMC3wSv8g7eURxXihT6Ao34atKGg8EXzYUxADtpigJoBNnCOUirIoNzAikUSZrPpzF4DUKo8pdyyMGAvGMeHOBDsM_5uAJH_rxt847SciqQHHxrgtKhiZDuB4wp1oivKUwoPhX3dzHtExCiy 5yK3DiWTmUnlVKEaThyVNRrzsLH24A6g_l.fBKewRKZhpgqU0aYNVz46cd_k6vUwICs93NsqYzl8PM1VMdt_rq3Tt2zS9q0QLzRvDRrrT9zH1UGSzgDEey.flhsg3kADdTFkPshgrd6eUdEajdN4jA2F.cFq243WC0TERYC5RKlBL2t3TBpkMa1_gVY2oEt8psQftgrLUp78YlKgpJnHZ9vbi_2abNStkXwt_b1A91N 3msGtwBu.zePkwM4ymmwyk5LaZO0Zy.XSDaVWF4Fc.SBN_U3TthB52wtLkSMR_mcT2.V_FV3d1.eeR6D3dfCr12lC8E8tf46OQ1YLiRcPGzJuXOtLMzdlz1mPff3cn_kKr0LgYOlqkR0pUAqZAN6NpRt7TQSLlGg0ov_yrD6u6dWxY4dhAO0P9lXRGxfmq7RidJnj6MASB6PoY0E",mdrd: ".liC62R.Sy6WPI3d36 g943OLuc7J8EWGKwmQFBny7.0-1710277862-1.1.1.1-YrQy_jeYCEBzyuaxfWapSQRWYONdpeANGCkVUlmYTH4e.XApO.T6Kb4SgWpYMIWNFrjFNBPd0M2XI8JxFxcDjYko16L8pypeCdkbBANrE_gMPQpacjZg0AgZfzDk0g9KnKPQuiIWfJf2XHnJkvRXae3hRFjbicRMRlQmdM7v1XFc19Kg0cAQ4ypmz..U0U ETCpLnjDaMNrGq1CO96cVfSnpgX3BhHiYasdQvoD4okbTPJNXhEZ8niI2TbSgBcgeYbWoMrhojAZxqFNw.X.yrQQZAnOoK3oJFZSwyML7R8ytrlvZdN4_vl51oprpjiHDNkCmMxmrB3SRbKNkm3UAa2uydbx9BkFJ6yg_S0UiGnrnnJJGXUCCkimDBzlbeJIgXyWIRFS_u50f5df573UNyls8pTg9w29DCybhvhNBAM YJs4oyBKjZUP6D8s7Xy33MpSPFNuy0M.gE5dyVPAQdOA9XbiqFZxQjVsI.Dzfb5w674RkBNnzNQaANdMCkGZjy9svAUxIf2FSgNOLnsFevbD_Rtr0pvCTjz24Rp.6er5ixPEv9_xRZaEYqSQRb0SBgxIBeBGBsRh0r.iQHSVdFIJ819JylupKGX6w.tvF1I5rKLZ3r7RzKoRU7_XAeMgxb76wxTkaN6S9SkTsejCRm7 zq8aJP.qv_VoBqALcYwS9PTVTqIAgwns0Aboinwe.6oUlGZMOtAZ6_2MlEOjdPT5_wTtBgwvJEKwdTXOnslChtz6RwW6LduDlttRFhA82f_n9RvvxghR4BlYUUidpcdErhkolvGwFOHCNLdKx_Lg3lPhcZdVe05s5KKzgwgNTfgrcI1jJ.WaC8XLz8kXDnHFBeZKKCcuJb2Z2BWwHSW7pamx4PhgRM8HHYEH25hNvQS 8slg6ecJ8pIHNZBaqQgXqMpwgH25frHeKDzrrOMS0hwuUBQK54RkpdQvH14PNZEH._KKrRQD0dtx2J5.vYk475jklOPrMtWJsqp2R99WV1u0AoDS9l5YxQaKcu1357SEyXyY5.lZyP_6wa4b1dhotUY_qlGvYOgifn32wHlV6SUs3TjajAiw2sllT3resq7dsOFdrZ3DdVQbin8oXpKxqu1KwsuhhOrl3vtlaV5ma qIUqDzH__O73kdHuMSFABPNmjOYP.b_0QojGtsMQvHv5cmycCX1oksxN1o5ihC9z6CQYdtGVtfj3tcrNvLmvERmOWMRotlr6kQoHjFo1rfhK6UY6FHEzHRYYJng08_4YwVvF8oGiciY7RdjyI9m3EIFsg2N0iw7rSIvYa2aMybTBaaJ8x2hIDMWMmzwTn7fzGMcb4adISO6TLuuwFLbR.oSV..TrMPFkSCpl6Q77mER HqnLYTg3QN4kKSO3ZM99zd2NICylGhGHfNE1Lxs.ME0GgC0wA53REOZQtBZTaetvmvzD9wMZsfILCDkjwpPyHAUuz9xcCNvB5OTw6AvKA4CmeMJg0pK.cXxXR1qLc1lqlKwMma.TGglUvOuvmHAInpSQeB.d1HaQLU1rMUrcQ_gqRYG27ioUr1jM0cRgJbPMZ7DeEzb4_ymJP11fpgMctvawQmFPT38ql4uWLHxoCk4 nPom2tNGc_A8_JG8go.3cbFCySataTCCgXj1nGMQ6.Qaeu5lwFpbHxdEIqd8.HSkH.mvVlqcpTliBWTy.Zeudad7Tkyy4YaCjneau1rCsZaS045U1EYNamY1Sq_G8ufkQEKXMYJt8HiyRLAUZZa8zSwFb_ttzWTKr9XDrrPInFUnvhHRShI5gM5pRNRXabrMg3dBB16cnehlv4qBdEHFnhmUL77oPCLCw1C0AeeKrER W32U.ABkwGZuK4sKTG2X8yHCU9aIpJBQ_1f0hUri3.yhGNT3hJrinfbeLZ2r3EBqnUHlzZS04VyRL5S0OBn3ufUi3O9pmwp0sj_MbxFlTkqfvGnFsU_EB8de0t9xiudYSWKgJ7J7GBw2ttxl3voYtKcSczSc.Se6jvuB78pOSjeyh10MqKRAVYn8wI2d7mUoZt9YtvvzzpSTNom5_WoN3.Qe2wRlYZJBaIy.eXrEsxF Dlk8l1evY_mGpuX0QiPKGkI6vuxvlcYhXNDk3sVBmVbaH_pB9G7F6tmADXemhhk_GhPy8Z4SwuRgbWGP0iP8DGQPkcI2GvbeSO9DKTCfQOrCW14wJH15G.X3qCO37.bx5O5SjFsQcZBG1Y0d.xsiI",cRq: {ru: 'aHR0cHM6Ly9kb3dubG9hZC5jeXByZXNzLmlvL2Rlc2t0b3AvMTMuNi42P3BsYXRmb3JtPWxpb nV4JmFyY2g9eDY0',ra: 'TW96aWxsYS81LjAgKFdpbmRvd3MgTlQ7IFdpbmRvd3MgTlQgMTAuMDsgcnUtUlUpIFdpbmRvd3NQb3dlclNoZWxsLzUuMS4yMjYyMS4yNTA2',rm: 'R0VU',d: 'JeLJ3shQ6CCF+fKbLFpKqgz6lSGsz0CE2ov9tfD0qFUUXEiKsI8DiAOmuZ9MsUMseyYQIqrWB63Z8Zr6E3HTTGn9 uoO6aEYv1mjvuU34bmvzgkfQRKnt4PaiGPA1pkgZw9Q6Epr7PNzORQ54H/A1cX0hFdQspB27tW40E+DhtKVZPI7jDFpJ2/deQ3u/x2I9WA9W6QZcOKa5cq7e+JnVn3ljr3d3nSXG70zmYI2UzFubxVeI8Da2eoOsusKXt3/SIAK01q06Ou07iUi6EPsbSXnrCVAaP3j5oGEu9jlCtDZIfs6cR7B7LtbK3M1u8DfgY7y 63rYKJFUl2rtGRR/HVslEssqZJCAPRFpzuAMZtc+0SGKIs/eUgyZ6GKh5udtoJeR/XpQGnUfHACRkv+6/zd7vz/17zVyxb34apJZxsPAC6gErA0QQzZ8f6AiA8s5AvbDa2EmIO3FxlFf5t0LUMUHS3TeqvgmSHUgMIyqtXkIPJyRkbEu5tOZd6ilj/W7C',t: 'MTcxMDI3Nzg2Mi4xMTMwMDA=',cT: Math.floor (Date.now() / 1000),m: 'tRKky1ac4WlfWlBY9BwJW4w6Zlz72EHdXERWNaRKoVg=',i1: 'BPCQskbSQafcHiKbOzgWOQ==',i2: '0DAdY9ZNaR7lMtY6/xR+MA==',zh: '16g56Sl9OErXzDbei3pVjkQ3vX5oN5JUbnAVMy/FDfQ=',uh: 'ZcLu8/8hGnYsignjivRIMHMoPTaWYh1pSSY+9bZtKy0=',h h: 'x+dVsAyy4/zcI6ySIIeGqyLMI2nfvXzQJPeyNjs48Cg=',}};var cpo = document.createElement('script');cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=8636c63e29cd690a';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location .search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null, "\/desktop\/13.6.6?platform=linux&arch=x64&__cf_chl_r t_tk=PMkbqYVWjmlFOqwkhrobrmJrG5QkOj3.Z.wHPTwlZM8-1710277862-0.0.1.1-1578" + window._cf_chl_opt.cOgUHash);cpo.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(cpo);}()); строка:1 знак:1
wget "https://download.cypress.io/desktop/13.6.6?platform=linux&arch= ...
+ CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Manual downloading and setting CYPRESS_INSTALL_BINARY helped me to work around this problem and build my Docker container that depended on Cypress.

jennifer-shehane commented 5 months ago

@devoln Thanks for the update. I'm glad the workaround did work. Sorry for the issues. I'll close this and suggest the workaround for now. We haven't gotten other reports of this, so think it may be isolated.

cypress-io / cypress