namkyup commented 1 year ago

Hi,

using git clone I cloned the repo. After that I installed safeway-coupons using pip install. I created conf file with both required parameters (email and password, no email setting).

After running safeway-coupons -c config_file I get the following error:

Clipping coupons for Safeway account xxxx@xxxx.com
Error: Authentication Failure ("name='SWY_SHARED_SESSION', domain=None, path=None")

I'm running safeway-coupons on Ubuntu:

uname -a
Linux xxxx 5.15.0-69-generic #76~20.04.1-Ubuntu SMP Mon Mar 20 15:54:19 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:    20.04
Codename:   focal

python version:

python3 --version
Python 3.8.10

tpengandrew commented 1 year ago

Same here on Mac.

jmacknet commented 1 year ago

Same here. Looks like they implemented some anti-bot code via Incapsula. Here's the response text I get:

<html style="height:100%"><head><META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"><meta name="format-detection" content="telephone=no"><meta name="viewport" content="initial-scale=1.0"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><script type="text/javascript" src="/_Incapsula_Resource?SWJIYLWA=719d34d31c8e3a6e6fffd425f7e032f3"></script></head><body style="margin:0px;height:100%"><iframe id="main-iframe" src="/_Incapsula_Resource?CWUDNSAI=42&xinfo=10-44996152-0%20SNNN%20RT%281682203502050%201100%29%20q%280%20-1%20-1%20-1%29%20r%280%20-1%29%20B10%2814%2c0%2c0%29%20U24&incident_id=204000110455243564-245269476160570058&edet=10&cinfo=0e00000056f5&rpinfo=0&mth=GET" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px">Request unsuccessful. Incapsula incident ID: 204000110455243564-245269476160570058</iframe></body></html> `

ksebby commented 1 year ago

Has anybody found a workaround yet? I really miss this tool.

jmacknet commented 1 year ago

There's a browser script that works, linked below. Totally different approach that doesn't fix the safeway-coupons tool, but an OK substitute. I read around and couldn't figure out exactly how Incapsula identified bot-like activity, but apparently it was somewhat sophisticated and might take some work to trick (and also might be a whack-a-mole since Incapsula isn't necessarily static).

https://www.reddit.com/r/Frugal/comments/bi5szz/automatic_safeway_coupon_clipper/

tpengandrew commented 1 year ago

There's a browser script that works, linked below. Totally different approach that doesn't fix the safeway-coupons tool, but an OK substitute. I read around and couldn't figure out exactly how Incapsula identified bot-like activity, but apparently it was somewhat sophisticated and might take some work to trick (and also might be a whack-a-mole since Incapsula isn't necessarily static).

https://www.reddit.com/r/Frugal/comments/bi5szz/automatic_safeway_coupon_clipper/

Thanks. I have no knowledge about Javascript -- anyone knows how to run this with a cron job?

jmacknet commented 1 year ago

@tpengandrew you have to run it from your browser as a bookmark, while the Safeway coupons page is open. No way to run it unattended via cron job. Right now, it needs human interaction via a browser to bypass the bot check.

jamesderlin commented 1 year ago

Safeway's website has a "Keep Me Signed In" checkbox when logging in, so it stores an authentication cookie. I'd be satisfied with a mechanism that allowed safeway-coupons to be given a file that contains an existing authentication cookie.

namkyup commented 1 year ago

Thanks for sharing the alternative. The whole idea was that cron job takes care about the coupons. If user has to login (using browser) coupons can be clipped manually as well.

I hope somebody finds the solution for Incapsula.

tpengandrew commented 1 year ago

There's a browser script that works, linked below. Totally different approach that doesn't fix the safeway-coupons tool, but an OK substitute. I read around and couldn't figure out exactly how Incapsula identified bot-like activity, but apparently it was somewhat sophisticated and might take some work to trick (and also might be a whack-a-mole since Incapsula isn't necessarily static). https://www.reddit.com/r/Frugal/comments/bi5szz/automatic_safeway_coupon_clipper/

Thanks. I have no knowledge about Javascript -- anyone knows how to run this with a cron job?

My JavaScript expert friend recommend Automa. I have not tried yet but seems it would work.

jamesderlin commented 1 year ago

Safeway's website has a "Keep Me Signed In" checkbox when logging in, so it stores an authentication cookie. I'd be satisfied with a mechanism that allowed safeway-coupons to be given a file that contains an existing authentication cookie.

I looked into using a session cookie from a browser, but apparently even with the "Keep Me Signed In" checkbox enabled, Safeway's session cookie has an expiration time of about 44 minutes, which is way too short to be useful.

The Safeway Android app seems to stay logged in, however, so it seems like it ought to be possible. I'm not sure what the app is doing or how to find out, though.

MilesBreslin commented 1 year ago

81 "automates" log-in to get past this error. The user still needs to manually pass the reCAPTCHA check though.

Foxtrod89 commented 1 year ago

same here Authentication Failure ("name='SWY_SHARED_SESSION', domain=None, path=None") You can try this in tampermonkey as workaround but this is semi-automatic approach to problem.

// ==UserScript==
// @name     Just For U REBATES, REWARDS Clipper
// @version  1.0
// @grant    SirLoras89
// @include https://www.safeway.com/foru/coupons-deals.html*
// @description Clip only REBATE and REWARD offers on the current (as of 05/08/2023) Safeway Just For U coupon system.
// @namespace https://greasyfork.org/users/22981
// ==/UserScript==

/// While the load more button exists, load more

function loadUntilDone() {
  let buttons = document.getElementsByClassName('load-more')

  if (buttons.length > 0) {
    // Still a load more button. Click until it goes away

    console.log("Loading more coupons...")

    try {
      buttons[0].click()
    } catch (e) {
      console.error(e)
    }

    // Give it some time to load.
      setTimeout(loadUntilDone, 1000)
  } else {
    // Now find and click all the coupons
    console.log("Clicking all coupons...")
    clickAllUnclicked(document.querySelectorAll('[aria-labelledby*="REBATE"]'))
    clickAllUnclicked(document.querySelectorAll('[aria-labelledby*="Reward"]'))
    clickAllUnclicked(document.querySelectorAll('[aria-labelledby*="Points"]'))
  }
}

/// Resolve after the given delay
async function sleep(delay) {
  return new Promise((resolve, reject) => {
    setTimeout(resolve, delay)
  })
}

/// Click on every element in the given collection, at a sensible pace, unless alredy clicked
async function clickAllUnclicked(elems) {
    for (let i = 0; i < elems.length; i++) {
    let elem = elems[i].children[0].children[2].children[1].children[0].children[0];
    if (!elem.classList.contains('btn grid-coupon-btn btn-default')) {
      console.log("Click element " + i + ": " + elem)
      elem.click()
      await sleep(100)
    }
  }
  console.log("All coupons clicked!")
}

// Wait for the page to load and then start collecting coupons
console.log("Waiting to load coupons")
setTimeout(loadUntilDone, 4000)

jamesderlin commented 1 year ago

I looked into using a session cookie from a browser, but apparently even with the "Keep Me Signed In" checkbox enabled, Safeway's session cookie has an expiration time of about 44 minutes, which is way too short to be useful.

I just noticed that with an expired session cookie, Safeway's website requires re-authenticating to access Rewards but apparently does not for accessing the "Safeway for U" page or "My List". I seem to be able to clip coupons from the "Safeway for U" page even with an expired session cookie, so presumably safeway-coupons could work too?

smkent commented 1 year ago

Hi all, thanks for the report and information! I found some time to work on this today. I've merged https://github.com/smkent/safeway-coupons/pull/82 and released v0.2.1 which updates the authentication flow to use undetected-chromedriver, similar in concept to https://github.com/smkent/safeway-coupons/pull/81 (thanks @MilesBreslin!).

As Imperva (aka Incapsula) can produce different results based on a number of factors including requesting IP, this solution won't work if you would be prompted with a CAPTCHA (or denied authentication altogether). Please let me know whether safeway-coupons v0.2.1 works for you.

smkent / safeway-coupons

Authentication Failure #80

81 "automates" log-in to get past this error. The user still needs to manually pass the reCAPTCHA check though.