search

Hari-Nagarajan / fairgame

Tool to help us buy hard to find items.
GNU General Public License v3.0
2.44k stars 805 forks source link

ASIN values stopped working #490

Closed theaddies closed 3 years ago

theaddies commented 3 years ago

Of the original list of AISN values I was using from https://docs.google.com/document/d/1grN282tPodM9N57bPq4bbNyKZC01t_4A-sLpzzu_7lM/edit Only 2 for the 3080 cards still work, B08HR7SV3M and B08HR5SXPS . All others show no results. Is there a way to determine new AISN values or does the program not work anymore.

unapproachable commented 3 years ago

@theaddies I'm inclined to tell you that the program doesn't work any more. However, instead, I'll ask that you provide an updated list of ASIN for the community to use once you figure them out. Please post back here when you have an updated list. We will leave this issue open for you.

anoriega211 commented 3 years ago

This may not be a complete list, but I used this tracker's lists for 3070's and 3080's from Amazon and have been using those ASIN's - https://www.nowinstock.net/computers/videocards/nvidia/rtx3080/. Hope that helps

miketest123123 commented 3 years ago

Also having the same issue OP is having. Pages are errored out (infamous Dog landing page), or it goes to the ASIN but does not know how to look at prices anymore due to the price pop up.

josephsmi2013 commented 3 years ago

I'm confirming for @miketest123123 we well. Popup has an additional "Add-to-cart" button which needs to be selected.

The command window will just output "failed to load prices for #ASIN, going to next ASIN"

Any help would be much appriciated!

SkebengZA commented 3 years ago

Yeah looks like amazon added a redirect to the page that defeats the scraping now.

AdamBLevy commented 3 years ago

The sad irony here is that Amazon and Best Buy will defeat the genuine customers just resorting to botting to make legitimate purchases, while the well-funded, paid bot creators will defeat their efforts and perpetuate the cycle of scalping.

afpcondor13 commented 3 years ago

Is this project still being worked on? This seems like a trivial thing for the creator to fix.

DakkJaniels commented 3 years ago

It is still being worked on, but we all have lives outside of this project and would love for other people to help us with it. What is it that you think is so trivial? and if it is so trivial, why don't you submit a PR and fix it?

unapproachable commented 3 years ago

@afpcondor13 Do let us know when you finish the trivial fix and get it submitted. Judging from your github activity, you're well equipped for the task at hand. Also, if you could state what it is that you believe the issue to be it would help other people contribute to your trivial fix. Thank you for all your efforts.

afpcondor13 commented 3 years ago

I wish I could but I don't have the programming knowledge yet, I'm working on it and hopefully will in the not-too-distant future. What I meant about it being trivial is Amazon just changed the way the buy options open up and appear to have disabled them opening up on their own page, it just seems like the bot just needs to be changed to look on the sidebar that opens on the main product page. I apologize if it seemed like I trivialized the work that's been done on this bot, I'm amazed at the work you guys have done and very much appreciate what you are giving back to the community. I hope to join the open source community soon and help give back like you all are doing.

afpcondor13 commented 3 years ago

My initial comment was more geared towards the comments above that seemed to imply that the bot was dead because of Amazon's change and it didn't appear that way to me. I'm very glad to see that it's not, thank you guys for replying so quickly, and again, I apologize for how my comment came off.

afpcondor13 commented 3 years ago

@afpcondor13 Do let us know when you finish the trivial fix and get it submitted. Judging from your github activity, you're well equipped for the task at hand. Also, if you could state what it is that you believe the issue to be it would help other people contribute to your trivial fix. Thank you for all your efforts.

After looking through the issues on the page, I can see the issue appears to have been fixed in the Feature/xpath branch, but when I look through the branches I don't see it available. I'm guessing it has been merged into another branch, can you tell me which branch currently has this fix?

Clearly I don't know git very well yet, sorry if I should be able to see this info somewhere.

DakkJaniels commented 3 years ago

It was merged into development. You can find this by looking at closed pull requests.

afpcondor13 commented 3 years ago

It was merged into development. You can find this by looking at closed pull requests.

Ah awesome, thank you!

afpcondor13 commented 3 years ago

After pulling the Development branch it looks like it's working now, thank you guys.

Some additional info that could be useful, I downloaded and set up the bot Thursday afternoon, I'm shooting for the 5950x. I let it run overnight into Friday on 1 second refreshes, sometime Friday morning it broke, I didn't see it until I got home early Friday afternoon. This is interesting because looking back at the different issues on here, it doesn't appear to be new, it looks like it's at least a few weeks old, and yet mine ran for almost a full day before Amazon appeared to break it.

I just thought this might be interesting to you guys, it looks like Amazon may be making slight changes for users that appear to be running bots on their page.

gtrak commented 3 years ago

Hi!

Will look at it in more detail tomorrow, but there's a couple things I noticed here.

One, the xpath for the offers page, seems to have changed: This can find something for me:

                 test = self.driver.find_element_by_xpath(
-                    '//*[@id="olpOfferList"]/div/p'
+                    "//span[@data-action='show-all-offers-display']"
                 )

2, I think we're doing some needless manual looping and waiting in this block, when selenium has timeouts built-in:

        while True:
            atc_buttons = self.driver.find_elements_by_xpath(
                '//*[@name="submit.addToCart"]'
            )
            if atc_buttons:
                # Early out if we found buttons
                break

            test = None
            try:
                test = self.driver.find_element_by_xpath(
                    "//span[@data-action='show-all-offers-display']"
                )
            except sel_exceptions.NoSuchElementException:
                log.info("Offers element not found")
                pass

            if test and (test.text in NO_SELLERS):
                return False
            if time.time() > timeout:
                log.info("TIME:", time.time())
                log.info(f"failed to load page for {asin}, going to next ASIN")
                return False

This will look for an element by xpath for 10 seconds:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

WebDriverWait(driver, 10).until(
        EC.visibility_of_element_located((By.XPATH, xpath))
    )

I can probably get something working locally and put up a PR for this case, but I bet amazon will keep changing stuff, or might use different strategies for different items. Has that happened in the past? Should we try a few known things before bailing out?

I was testing it against a page like this: image

gtrak commented 3 years ago

Ah, I guess I should look at the development branch :eyes: . Thanks for working on this!

DakkJaniels commented 3 years ago

@gtrak Regarding Item 1, does that just take you to the offers page? Amazon has been redirecting off of that inconsistently so probably not worth it. We have a different idea in a different branch we are working on, but it's not ready for development yet.

Regarding Item 2, I think selenium basically does the same thing, it just has a built in polling frequency when you call that. Regardless, we are moving to that style of call in the branch I mentioned above.

gtrak commented 3 years ago

Re item 1, clicking it would open up a pane on the right with a list of offers, but I like the old style URL munging approach you have here. If it works consistently, it's better than trying to navigate the more complex UI and guess its states. I was thinking the presence of that element could just be used to know if the offers are available at all.

On Sat, Feb 6, 2021, 11:17 PM DakkJaniels notifications@github.com wrote:

@gtrak https://github.com/gtrak Regarding Item 1, does that just take you to the offers page? Amazon has been redirecting off if that inconsistently so probably not worth it. We have a different idea in a different branch we are working on, but it's not ready for development yet.

Regarding Item 2, I think selenium basically does the same thing, it just has a built in polling frequency when you call that. Regardless, we are moving to that style of call in the branch I mentioned above.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Hari-Nagarajan/fairgame/issues/490#issuecomment-774591610, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGQFOULWIV5YVA47AJCI7LS5YH4XANCNFSM4XC3LXRA .

DakkJaniels commented 3 years ago

Ah - we were using the offers listing page (OLP) originally, but Amazon is sunsetting that for the all offers display (AOD), which is the list that shows up on the right. Development branch can currently deal with that. Unless you get lucky with getting the OLP to load, master branch doesn't work very well at the moment.

gtrak commented 3 years ago

The development branch seems to be working well for me.

I had to turn off OTP and add a sleep that was long enough to manually solve a captcha, but now I have it working.

SirDystic commented 3 years ago

Seeing a similar ASIN issue with German, France and Spain ASINs for the PS5. Trying to buy a PS5 digital or normal, the ASIN url is directed to focus on a PS5 controller or camera instead of the console:

So https://www.amazon.de/dp/B08H93ZRK9/ref=olp_dp_redir focuses on the wrong item

To see the correct ASIN this link now seems to be required: https://www.amazon.de/dp/B08H93ZRK9/?th=1

Also a minor issue on the Amazon Italy site:

"ERROR Ordine in preparazione is not a known title, please create issue indicating the title with a screenshot of page"

theaddies commented 3 years ago

@gtrak

The development branch seems to be working well for me.

I had to turn off OTP and add a sleep that was long enough to manually solve a captcha, but now I have it working.

What is OTP? The code is generally beyond my capabilities but I might be able to get your suggestion to work for me. Thanks.

gtrak commented 3 years ago

OTP is the security jargon term 'one time pad' used for 2-factor-authentication, the text code that amazon will send to your phone for verification on login, I noticed fairgame wouldn't handle this on master, but not sure if development has code to handle it.

Disable 2FA: https://www.amazon.com/gp/help/customer/display.html?nodeId=GLEC8Z5M8NNXC9B2

Deckoz2302 commented 3 years ago

both main and dev handle waits for OTP. They don't however handle OTP when using '--headless'.

theaddies commented 3 years ago

I managed to get the development branch working however I get these spurious errors. It does however keep running.

Traceback (most recent call last):
  File "C:\Users\thead\Documents\bootcamp\fairgame\stores\amazon.py", line 567, in check_stock
    WebDriverWait(self.driver, timeout=DEFAULT_MAX_TIMEOUT).until(
  File "C:\Users\thead\.virtualenvs\fairgame-laGWzWY4\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
2021-02-09 16:06:49|0.6.0.dev5|ERROR|Timed out waiting for the flyout to open and populate.  Is the connection slow?  Do you see the flyout populate?
2021-02-09 16:07:00|0.6.0.dev5|ERROR|Timed out waiting for the flyout to open and populate.  Is the connection slow?  Do you see the flyout populate?
2021-02-09 16:07:10|0.6.0.dev5|ERROR|Timed out waiting for the flyout to open and populate.  Is the connection slow?  Do you see the flyout populate?
2021-02-09 16:07:21|0.6.0.dev5|ERROR|Timed out waiting for the flyout to open and populate.  Is the connection slow?  Do you see the flyout populate?
2021-02-09 16:07:31|0.6.0.dev5|ERROR|Timed out waiting for the flyout to open and populate.  Is the connection slow?  Do you see the flyout populate?
2021-02-09 16:07:42|0.6.0.dev5|ERROR|Timed out waiting for the flyout to open and populate.  Is the connection slow?  Do you see the flyout populate?
Extolus commented 3 years ago

Hey folks! So is the bot still functionally? Or it's dead?