search

samc621 / SneakerBot

All-in-one bot, with auto captcha-solving and proxy management, using Node.js and Puppeteer.
MIT License
753 stars 194 forks source link

No sites work and proxy doesnt work #33

Closed authsoftware closed 3 years ago

authsoftware commented 3 years ago

I have tried to use proxies but none worked and it keeps using local to open the sites. Also I cannot get anything to actually checkout, footlocker keeps saying product not found on both product id and link, nike wont add to bag, shopify keeps giving an error of 

[6/9/2021, 9:52:39 PM] - Error: Evaluation failed: TypeError: Cannot read property 'id' of undefined
    at <anonymous>:3:72

on champs it is stuck on closing modal. Supreme this is the error I receive

[6/9/2021, 10:50:33 PM] - Navigating to URL
[6/9/2021, 10:50:36 PM] - Error: Evaluation failed: TypeError: Cannot read property 'querySelector' of null
    at <anonymous>:9:36
    at ExecutionContext._evaluateInternal (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\ExecutionContext.js:217:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async ExecutionContext.evaluate (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\ExecutionContext.js:106:16)
    at async Object.exports.guestCheckout (C:\Users\dev\Desktop\SneakerBot-master\sites\supremenewyork.js:229:7)
    at async C:\Users\dev\Desktop\SneakerBot-master\helpers\cluster.js:85:34
samc621 commented 3 years ago

@authsoftware thanks for opening this ticket. Regarding the proxy issues, please see #29. And regarding the nike issues, see #28. I am looking into the other issues.

samc621 commented 3 years ago

Can you also provide some details to reproduce the issue on Footlocker? Preferably a Task object I can test with? I cannot reproduce it from my end.

I recently fixed an unrelated issue with the footsites here.

samc621 commented 3 years ago

I am also not having any issues with shopify or supreme. Please provide a Task object for testing.

authsoftware commented 3 years ago

Yes I will send you the tasks I have so you could test them on Foots, Shopify, Supreme

authsoftware commented 3 years ago

Supreme

{
            "id": 24,
            "site_id": 5,
            "url": "https://www.supremenewyork.com/shop/shirts/b29vjuoq3",
            "style_index": null,
            "size": null,
            "shipping_speed_index": null,
            "billing_address_id": 1,
            "shipping_address_id": 1,
            "notification_email_address": "sessismith94@gmail.com",
            "created_at": "2021-06-09T19:29:52.576Z",
            "updated_at": "2021-06-09T19:29:52.576Z",
            "is_deleted": false,
            "auto_solve_captchas": true,
            "product_code": null,
            "site_name": "supremenewyork"
        },

Shopify

{
            "id": 19,
            "site_id": 3,
            "url": "https://kith.com/collections/kith-hoodies/products/kh2657-603",
            "style_index": null,
            "size": null,
            "shipping_speed_index": null,
            "billing_address_id": 1,
            "shipping_address_id": 1,
            "notification_email_address": "sessismith94@gmail.com",
            "created_at": "2021-06-09T18:49:04.645Z",
            "updated_at": "2021-06-09T18:49:04.645Z",
            "is_deleted": false,
            "auto_solve_captchas": true,
            "product_code": null,
            "site_name": "shopify"
        },

Footlocker

{
            "id": 16,
            "site_id": 2,
            "url": "https://footlocker.com",
            "style_index": null,
            "size": null,
            "shipping_speed_index": null,
            "billing_address_id": 1,
            "shipping_address_id": 1,
            "notification_email_address": "sessismith94@gmail.com",
            "created_at": "2021-06-09T18:33:10.523Z",
            "updated_at": "2021-06-09T18:33:10.523Z",
            "is_deleted": false,
            "auto_solve_captchas": true,
            "product_code": "W2288111",
            "site_name": "footsites"
        },
samc621 commented 3 years ago

@authsoftware using a style_index and size of null will not work for these products. For products without more than one style, you can use 0 for the style_index. I would apply the same for the shipping_speed_index. But you will still need to apply a size.

The reason I made these fields nullable is for site integrations or individual products where they are N/A e.g no shipping speed or style options on Supreme, or products without size options.

authsoftware commented 3 years ago

Okay thank you, I will test that and let you know

authsoftware commented 3 years ago

Okay so after fixing that and fixing the proxies issue, this is the error I have now

[6/15/2021, 11:45:23 PM] - Searching for product by product code
[6/15/2021, 11:45:23 PM] - Error: net::ERR_CONNECTION_RESET at https://www.champssports.com/search?query=EY2763
    at navigate (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:115:23)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async FrameManager.navigateFrame (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:90:21)
    at async Frame.goto (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:416:16)
    at async Page.goto (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\Page.js:821:16)
    at async searchByProductCode (C:\Users\dev\Desktop\SneakerBot-master\sites\footsites.js:276:5)
    at async Object.exports.guestCheckout (C:\Users\dev\Desktop\SneakerBot-master\sites\footsites.js:323:7)
    at async C:\Users\dev\Desktop\SneakerBot-master\helpers\cluster.js:94:34

task object

{
        "id": 25,
        "site_id": 2,
        "url": "https://www.champssports.com/",
        "style_index": 0,
        "size": "12",
        "shipping_speed_index": 0,
        "billing_address_id": 1,
        "shipping_address_id": 1,
        "notification_email_address": "sessismith94@gmail.com",
        "created_at": "2021-06-15T20:45:11.970Z",
        "updated_at": "2021-06-15T20:45:11.970Z",
        "is_deleted": false,
        "auto_solve_captchas": true,
        "product_code": "EY2763"
    }
samc621 commented 3 years ago

@authsoftware thanks for this, I will test with this Task object.

I have seen this error from puppeteer before, particularly when using headless mode. Some websites will reject the headless user agent so we need to manually set the user agent when the page is created. I will verify that this is the issue and then put up a quick fix for it.

samc621 commented 3 years ago

@authsoftware this looks to be the case, can you confirm that it is working for you in headful mode?

authsoftware commented 3 years ago

@samc621 I did enable headful mode by making headless false in cluster.js but i have an issue with almost any site whenever you go to the product page it keeps going back to the main page for example on champs it goes to the product page then redirects to the home page then this issue comes up 

TimeoutError: waiting for selector `div#bluecoreActionScreen` failed: timeout 30000ms exceeded
    at new WaitTask (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\DOMWorld.js:509:34)
    at DOMWorld.waitForSelectorInPage (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\DOMWorld.js:420:26)
    at Object.internalHandler.waitFor (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\QueryHandler.js:31:77)
    at DOMWorld.waitForSelector (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\DOMWorld.js:313:29)
    at Frame.waitForSelector (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:841:51)
    at Page.waitForSelector (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\Page.js:1335:33)
    at closeModal (C:\Users\dev\Desktop\SneakerBot-master\sites\footsites.js:256:16)
    at Object.exports.guestCheckout (C:\Users\dev\Desktop\SneakerBot-master\sites\footsites.js:320:5)
    at C:\Users\dev\Desktop\SneakerBot-master\helpers\cluster.js:95:57
samc621 commented 3 years ago

@authsoftware this is happening for you with the sample Champs task above? It is working fine for me, at least in headful mode.

authsoftware commented 3 years ago

@samc621 Yes it is happening to me in all sites

samc621 commented 3 years ago

Based on the selector in that stack trace, it sounds like the modal which sometimes appears on the footsites is not appearing for you. Using timeout: 0 on that selector might help. But this promise is not awaited so its weird that it wouldn't at least attempt to ATC before this error occurred. I might need to see a video of what you are seeing.

I understand that you're seeing this behaviour across sites, but I will need stack traces from the others, then. This specific error is with a selector on the footsites.

labboy0276 commented 3 years ago

I am seeing this error as well (on headless) @samc621:

TimeoutError: waiting for selector `div#bluecoreActionScreen` failed: timeout 30000ms exceeded

Here is the task JSON I used:

{
        "site_id": 2,
        "url": "https://www.champssports.com/product/converse-run-star-hike-platform-high-top-womens/170913C.html",
        "style_index": 0,
        "size": "10.0",
        "shipping_speed_index": 0,
        "billing_address_id": 1,
        "shipping_address_id": 1,
        "notification_email_address": "labboy0276@yahoo.com"
}

edit: I am seeing this as a the body response via:

      let bodyHTML = await page.evaluate(() =>  document.documentElement.outerHTML);
      console.log(bodyHTML);
<html><head>
    <title>429 Too Many Requests</title>
  </head>
  <body>
    <h1>Error 429 Too Many Requests</h1>
    <p>Too Many Requests</p>
    <h3>Guru Mediation:</h3>
    <p>Details: cache-ewr18145-EWR 1625508467 2084869219</p>
    <hr>
    <p>Varnish cache server</p>

Is there a way to set the user agent dynamically with somehow? Maybe that is still the issue?

labboy0276 commented 3 years ago

After playing around this a little more, I was able to get past the error with the following:

  1. Made sure my proxies were being used. (I had burned through the list)
  2. Added the following package to my setup: user-agents && puppeteer-extra-plugin-anonymize-ua
  3. I then add this to the helpers/cluster.js file:
const Ua = require('puppeteer-extra-plugin-anonymize-ua');
const UserAgent = require("user-agents");

puppeteer.use(Ua());
const userAgent = new UserAgent();

    const puppeteerOptions = {
      headless: true,
      defaultViewport: null,
      args: [
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-web-security',
        '--disable-features=IsolateOrigins,site-per-process',
        '--user-agent=' + userAgent
      ]
    };

This got the timeout errors to go away. (note I edited this a couple times)

samc621 commented 3 years ago

@labboy0276 thanks for the updates here! There's a lot we can do around randomizing browser fingerprints, and starting with the UA is a great idea. If this has been tested and confirmed to help address a problem you are facing, please feel free to open a Pull Request. I am happy to try it on my end and then merge it as long as it works.

samc621 commented 3 years ago

@labboy0276 as for this error:

TimeoutError: waiting for selector `div#bluecoreActionScreen` failed: timeout 30000ms exceeded

Can you try the suggestion I mentioned above, which is to add timeout: 0 to the waitFor args?

So line 256 would look like:

await page.waitForSelector(modalSelector, { visible: true, timeout: 0 });

What I think is going on here is that this modal only shows on some of the footsites, but not Champs. Using the default timeout from Puppeteer, it will timeout after 30s, which will break your task midway. But with the timeout of 0, it will basically never time out, but listen asynchronously (non-blocking) and close the modal if it does ever appear.

I suggest trying this and see if it works. If it does, seems like a quick fix we can get in there (feel free to open a PR).

labboy0276 commented 3 years ago

@samc621 I put in that PR. I didn't have issues with that timeout after I fixed everything I mentioned. If it does happen again, I can try your suggestion.

samc621 commented 3 years ago

@labboy0276 thanks for that! So just to confirm, this fix is just for the footsites issue, right? Or does this address the nike issue as well?

labboy0276 commented 3 years ago

@samc621 the user agent issue fixes all sites for peoples who use headless.

cimthog commented 3 years ago

I keep getting Error 429, too many requests, whenever there's a drop.. for footsites.

Anyone knows a way round this?

samc621 commented 3 years ago

@labboy0276 thank you! I am testing that PR now.

13ROY commented 3 years ago

@samc621 and @labboy0276 , based on the discussions and the closed PR https://github.com/samc621/SneakerBot/pull/39, I am guessing we can close this issue?

samc621 commented 3 years ago

@13ROY yes, I am closing this issue now.