Does the script now get stuck at the login page?

[HubbabbuH]

Both with the current version that includes the $600 tax check and also with the previous version of the script, ever since yesterday afternoon for the past two days, the script no longer works for me.

When I check the login() function's "response.html" file, the script always get stuck at the Vine login page, with the Amazon error message of: "There was a problem. Your email or password was incorrect. Please try again." But my email and password that I use in the script are correct, and I can manually login through the browser with the same credentials.

The script apparently thinks the login was successful and now always says "Found 0 items" for both queues. But since I am not actually logged in and the script is not aware that I am not logged in, when it tries to access the "Your Queue" and "Vine for All" queues, it hits the login page yet again and, not surprisingly, then results in the "Found 0 items" errors because the script is looking for items via: soup.find_all('tr', {'class':'v_newsletter_item'})

So ever since yesterday, the script can no longer login using my credentials, even though I can login without problems when I manually use any browser. I think that Amazon changed something with the login process because up until yesterday afternoon, the script was working fine on a daily basis for me. After I got this error, I updated the script to include the $600 tax check code, and the same issue is still there.

So is this just happening to me, or can anyone else duplicate this?

[User2020]

Same thing happening to me. Except the debug.html shows an error "Please enable cookies" on it.

[timur-tabi]

In my case, I still have some pending reviews to write, so I can't debug this until I finish them.

[HubbabbuH]

Thanks for the reply, Timur. Amazon definitely changed something with their login page starting around yesterday morning. Up until yesterday morning, there were no problems.

In reply to User2020, the initial Amazon error of:

"There was a problem. Your email or password was incorrect. Please try again."

happens because the amazon-vine script cannot get past the login page. The response.html file that is saved by the login() function shows the email has been filled in, but the password field is still blank.

The script is not aware that the login failed and then tries to access the queue pages. And so in the "def get_list(br, url, name)" function, if you dump out the "soup = download_vine_page(br, url, name)" response, you will still see the Amazon login page, this time with the email field blanked out and the error of "Please Enable Cookies to Continue".

So if the script can be fixed to correctly login, I think the other parts of the script are okay once you are actually logged in.

[HubbabbuH]

Timur,

In login() function,

if 'There was an error with your E-Mail/ Password combination' in html:

should be changed to:

if 'Your email or password was incorrect' in html:

That will let the script properly check for correct login credentials. And:

if 'Your email or password was incorrect' in html:
    print 'Invalid userid or password'
    sys.exit(1)

should be placed after saving "response.html". So the order should be:

with open('AmazonQC_responseLogin.htm', 'w') as f:
    print >>f, html
# Check for bad password
# Fixme: not robust
if 'Your email or password was incorrect' in html:
    print 'Invalid userid or password'
    sys.exit(1)
soup = BeautifulSoup(html)

But the current problem is still that the script cannot get past the signIn form, and I don't see why this is happening.

[timur-tabi]

The weird thing is that if I download the login form and load it into my browser, I can log in. There must be something specifically wrong with the way Mechanize is processing the page that breaks it.

[timur-tabi]

Guys, I'm stumped. I've been comparing the HTTP output between my script and Firefox, and I don't see any real differences, and many of those differences are beyond my control because they're Mechanize internals.

[User2020]

You don't have any old copies of Amazon's page to see what changed, do you? I wish I did so I can help.

[timur-tabi]

No, I never thought to do that. I will do that from now on. The script currently doesn't really provide meaningful logs anyway. I've fixed that, so that if I ever figure out what's wrong, I'll be prepared for next time.

I don't think the problem is Vine-specific. It seems like general Amazon logins are broken. Maybe I can find another Python script that logs into Amazon to see what it does.

[User2020]

I really appreciate your work. Thanks for looking into this!

[masterac]

hie, i was able to change the script to log in on the french version, it work but i get :

Opening Votre file d'attente website Reading response Parsing response Found 0 items Opening Vine pour tous website Reading response Parsing response Found 0 items Cannot get list of items

so it can't find item i guess, i got 3 in my personnal and 1 in last chance. but it mean amazon france didn't change the code yet i guess, since i am not stuck, its interesting, so it mean maybe you can use the code of the french page, must be written in english anyway since the script work

[ghost]

Related: http://stackoverflow.com/questions/37409455/mechanize-perl-amazon-login-via-forms-unsuccessful

[timur-tabi]

masterac: can you give me the URLs for the French version?

[masterac]

yes there you go

https://www.amazon.fr/gp/vine/newsletter?ie=UTF8&tab=FR_LastChance https://www.amazon.fr/gp/vine/newsletter?ie=UTF8&tab=FR_Default

Date: Fri, 10 Jun 2016 12:10:47 -0700 From: notifications@github.com To: amazon-vine@noreply.github.com CC: master_a_c@hotmail.com; comment@noreply.github.com Subject: Re: [timur-tabi/amazon-vine] Does the script now get stuck at the login page? (#6)

masterac: can you give me the URLs for the French version?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[timur-tabi]

Looks like I can't log in to the French Vine page with my userid/password.

[masterac]

Yeah wont work since its a different country.

I was to make the app run but doesnt show product, i will have a look at the code, can you do a version without the tax ? We dont have it in france , i set the limit a 5000 but just in case i didnt do it correctly

Mar

Date: Tue, 14 Jun 2016 21:27:39 -0700 From: notifications@github.com To: amazon-vine@noreply.github.com CC: master_a_c@hotmail.com; comment@noreply.github.com Subject: Re: [timur-tabi/amazon-vine] Does the script now get stuck at the login page? (#6)

Looks like I can't log in to the French Vine page with my userid/password.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[ghost]

Anyone ever figure this out? I had a play around today and got nowhere, the page constantly claimed my password was wrong (it wasn't).

If someone fancies having another shot at this, here are a couple of links that might help (or hinder)

https://blog.jverkamp.com/2015/07/02/scraping-kindle-highlights/ http://toddhayton.com/2014/12/08/form-handling-with-mechanize-and-beautifulsoup/

[timur-tabi]

I believe the problem is in Mechanize itself. Amazon is expecting something specific.

Someone would need to debug the low-level HTTP/HTML interaction to determine what specifically Amazon is looking for that Mechanize does not produce. I could do that, but I'm not interested enough any more in Amazon Vine to bother. There are lots of other people with Amazon log-in scripts that also fail, so I'm waiting for someone else to figure out what's wrong.

[User2020]

Well, I want to thank timur-tabi for his script while it did work. It saved me a lot of trouble, but I adapted and now check manually all the time and seem to get better items now without the script, strangely enough. I think Vine was punishing me for using a script... although, I don't know how they would have known since I changed it to log in at random times.

[ghost]

The Kindle script, doesn't use Mechanise, and that too fails, which, I think means Mechanise isn't the culprit.....

[timur-tabi]

I have to hand it to Amazon. I don't know if they did it on purpose, but they managed to break every scraper app without breaking real web browsers, even those that have Javascript disabled.

[oliverx0]

It seems like it might have to do with cookies not being set correctly. I copied the cookies from my browser into my mechanize agent and it worked. So if you are simply trying to log in, this is a small hack that for some cases might be enough.

[timur-tabi]

Can you tell me how to do that? I tried playing with Mechanize's cookie support, but I couldn't figure it out.

[oliverx0]

Sure, just fyi, I am using the Ruby version of Mechanize. I was having exactly the same problem of not being able to log into Amazon. These are the cookies that I saw in my browser (it is for amazon.es, I can try amazon.com too, but the spanish version was giving me the same problems):

• __utma
• __utmz
• a-ogbcbff
• amznacsleftnav-845d2874-6afe-3651-bffc-0492b9d51b9b
• amznacsleftnav-d352bc15-5266-3350-8eb7-71dc34b73fee
• at-acbes
• cam-hit
• lc-acbes
• s_dslv
• s_fid
• s_nr
• s_pers
• s_vnum
• sess-at-acbes
• session-id
• session-id-time
• session-token
• ubid-acbes
• x-acbes

I put them in a CSV file, and this is the code to load them into mechanize and log in:

 @agent = Mechanize.new() do |agent|
    agent.user_agent_alias = 'Mac Safari'
    agent.follow_meta_refresh = true
    agent.redirect_ok = true
  end

  csv = CSV.read 'data.csv', col_sep: "\t"
  csv.to_a.map do |row| 
        cookie = Mechanize::Cookie.new({
        :domain => row[2], 
        :name => row[0], 
        :value => row[1], 
        :path => '/', 
        :time => row[4]
         })
        @agent.cookie_jar << cookie
  end

  page = @agent.get('https://www.amazon.es/ap/signin?_encoding=UTF8&openid.assoc_handle=esflex&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.ns.pape=http%3A%2F%2Fspecs.openid.net%2Fextensions%2Fpape%2F1.0&openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.es%2Fdp%2FB001FU1ECM', [], nil, {'User-Agent' => 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'})

 log_in_form = page.form('signIn')
 log_in_form.email = 'email@example.com'
 log_in_form.password = 'password'

 page = @agent.submit(log_in_form, log_in_form.buttons.first)

[timur-tabi]

It looks to me like most of those cookies appear only after you successfully log in. It appears that you're logging into Amazon via your web browser, and then "transferring" that login session to your script. So I don't see how this can help. Every time you want to run the script, you'll need to log via a web browser.

[timur-tabi]

Well, using this technique, I was able to get my script to work, but it's very clunky, and I don't know if it will work in the long term. I'll test it and see.

[oliverx0]

Yeah, like I said its more of a hack that can help some people. In my case, I only use my account and the cookies expire after a long time, so it helps.

[ghost]

I'm using python, so couldn't use the above directly, but tried the technique, but couldn't get it to fly🤔

I used a chrome extension to export my cookies as a cookies.txt file, and then loaded the file like this:

br = mechanize.Browser()     
print "loading cookies"
cj = mechanize.MozillaCookieJar()
cj.filename = os.path.join(os.environ["HOME"], "vinecookies.txt")
# Attempt to load the cookie file at this point
if os.path.exists(cj.filename):
  cj.load()
print ("cookies = %s " % cj)

br.set_cookiejar(cj)
# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

If anyone got this working with python, can they share code? Thanks

[timur-tabi]

The trick is to import only the .amazon.com cookies. If you export all cookies from Chrome, you need to strip out the non-Amazon ones during the import.

I think I need to write a program that imports cookies properly.

[timur-tabi]

I think I can use https://bitbucket.org/richardpenman/browsercookie/ to get my script to grab those cookies. The user will then have to log in manually into Amazon vine via his browser, then the script can grab that session data and use it to log in.

It might even be able to avoid the login process altogether.

Hang tight, everyone. I'm working to fix this.

[timur-tabi]

I've added support for the browsercookie package to automatically copy cookies from your browser (currently it supports only Firefox and Chrome) and use those to log in. Maybe one day I'll figure out how to log in without browsercookie, but for now this should work.

It's possible that after a few weeks the cookies will become invalid. If you, you will need to log in again via your browser and restart the script.