detecting automation tools to browse the website

puzzledvacuum commented 5 years ago

Anyone getting http status code 403?

status: 403,
  response:
   Response {
     _events: [Object: null prototype] {},
     _eventsCount: 0,
     _maxListeners: undefined,
     res:
      IncomingMessage {
        _readableState: [ReadableState],
        readable: false,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        socket: [TLSSocket],
        connection: [TLSSocket],
        httpVersionMajor: 1,
        httpVersionMinor: 1,
        httpVersion: '1.1',
        complete: true,
        headers: [Object],
        rawHeaders: [Array],
        trailers: {},
        rawTrailers: [],
        aborted: false,
        upgrade: false,
        url: '',
        method: null,
        statusCode: 403,
        statusMessage: 'Unauthorized',
        client: [TLSSocket],
        _consuming: true,
        _dumped: false,
        req: [ClientRequest],
        text:
         '<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Access to this page has been denied.</title> <!-- Custom CSS -->  <link href="https://fonts.googleapis.com/css?family=Open+Sans:300" rel ="stylesheet"> <style> html,body{ margin: 0; padding: 0; font-family: \'Open Sans\', sans-serif; color: #000; } a{ color: #c5c5c5; text-decoration: none; } .container{ align-items: center; display: flex; flex: 1; justify-content: space-between; flex-direction: column; height: 100%; } .container > div { width: 100%; display: flex; justify-content:center; } .container > div > div { display: flex; width: 80%; } .customer-logo-wrapper{ padding-top: 2rem; flex-grow: 0; background-color: #fff; visibility: hidden; } .customer-logo{ border-bottom: 1px solid #000; } .customer-logo > img{ padding-bottom: 1rem; max-height: 50px; max-width: auto; } .page-title-wrapper{ flex-grow: 2; } .page-title { flex-direction: column-reverse; } .content-wrapper{ flex-grow: 5; } .content{ flex-direction: column; } .page-footer-wrapper{ align-items: center; flex-grow: 0.2; background-color: #000; color: #c5c5c5; font-size: 70%; } @media (min-width:768px){ html,body{ height: 100%; } } </style> <script src="https://www.google.com/recaptcha/api.js" async defer></script> </head> <body> <section class="container"> <div class="customer-logo-wrapper"> <div class="customer-logo"> <img src=\'\' alt="Logo"/> </div> </div> <div class="page-title-wrapper"> <div class="page-title"> <h1>Please verify you are a human</h1> </div> </div> <div class="content-wrapper"> <div class="content"> <p> Please click "I am not a robot" to continue </p> <div class="g-recaptcha" data-sitekey="6Lcj-R8TAAAAABs3FrRPuQhLMbp5QrHsHufzLf7b" data-callback="handleCaptcha" data-theme="dark"> </div> <p> Access to this page has been denied because we believe you are using automation tools to browse the website. </p> <p> Th is may happen as a result of the following: </p> <ul> <li> Javascript is disabled or blocked by an extension (ad blockers for example) </li> <li> Your browser does not support cookies </li> </ul> <p> Please make sure that Javascript and cookies are enabled on your browser and that you are not blocking them from loading. </p> <p> Reference ID: #8fa4f810-4e34-11e9-a626-f91b39262958 </p> </div> </div> <div class="page-footer-wrapper"> <div class="page-footer"> <p> Powered by <a href="https://www.perimeterx.com">PerimeterX</a> , Inc. </p> </div> </div> </section> <!-- Px --> <script> ( function (){ window._pxAppId = \'PXZHh9f9x0\'; var clientSrc = \'//client.perimeterx.net/\' + window._pxAppId + \'/main.min.js\'; var p = document.getElementsByTagName("script")[0], s = document.createElement("script"); s.async = 1; s.src = clientSrc; p.parentNode.insertBefore(s, p); } () ); </script> <script> window.px_vid="";window.px_uuid="8fa4f810-4e34-11e9-a626-f91b39262958"; function handleCaptcha(response){ var vid = ""; var uuid = "8fa4f810-4e34-11e9-a626-f91b39262958"; var name = "_pxCaptcha"; var expiryUtc = new Date(Date.now()+1000*10).toUTCString(); var cookie = name + \'=\' + JSON.stringify({r: response, v: vid, u: uuid}) + \'; expires=\' + expiryUtc + \'; path=/\'; document.cookie = cookie; location.reload(); } </script> </body> </html>' }

riazXrazor commented 5 years ago

it doesn't happen always, still havent found a way

puzzledvacuum commented 5 years ago

Yup, tested extensively afterwards. Turns out that I was not pausing between starting multiple downloads simultaneously. I now add some human-like pauses between and haven’t been blocked again.

On Mon, 14 Oct 2019 at 19:31, Riaz Laskar notifications@github.com wrote:

it doesn't happen always, still havent found a way

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/riazXrazor/udemy-dl/issues/50?email_source=notifications&email_token=AEDT7KKSB5IADW4OJZTFF3TQOSUHNA5CNFSM4HAWBDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBFWOPI#issuecomment-541812541, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDT7KOWCZBDV5LJQ4VKBN3QOSUHNANCNFSM4HAWBDXA .

riazXrazor / udemy-dl

detecting automation tools to browse the website #50