Getting error 429 during fare checks and logins

[StevenMassaro]

Version

Auto-Southwest Check-In v7.1

Browser Version

Docker image

Description

Recently, I've started getting these 429 errors when checking the prices for upcoming flights. I am not sure if they are truly a 429 error, caused by actually sending too many requests, or a different error being reported as a 429.

To Reproduce

1. Add flights to the app
2. Check for price differences

Expected Behavior

No 429 error

Relevant logs and program output

2024-01-09 13:54:00 DEBUG Process-3[fare_checker:30]: Checking current price for flight
2024-01-09 13:54:00 DEBUG Process-3[fare_checker:88]: Fetching reservation information
2024-01-09 13:54:02 DEBUG Process-3[utils:34]: Successfully made request after 1 attempts
2024-01-09 13:54:02 DEBUG Process-3[fare_checker:103]: Retrieving search information for the current flight
2024-01-09 13:54:03 DEBUG Process-3[utils:34]: Successfully made request after 1 attempts
2024-01-09 13:54:03 DEBUG Process-3[fare_checker:82]: Retrieving matching flights
2024-01-09 13:54:08 DEBUG Process-3[utils:41]: Failed to make request after 7 attempts: Too Many Requests 429
2024-01-09 13:54:08 DEBUG Process-3[utils:44]: Response body: {
    "code": 429999999,

    "message": "Error.",

    "messageKey": "ERROR",

    "httpStatusCode": "BAD_REQUEST",

    "requestId": "",

    "infoList": []
   }
2024-01-09 13:54:08 ERROR Process-3[reservation_monitor:102]: Requesting error during fare check. Too Many Requests 429. Skipping...

Additional context

No response

[jdholtz]

I'm unable to try this myself at this time, but if anyone wants to play around, a next step to explore is to try to login to southwest using: https://github.com/ultrafunkamsterdam/undetected-chromedriver

@Royal2000H this script used to use that project. However, in v6.0 I moved it to use undetected mode in SeleniumBase. SeleniumBase is much better maintained, has many less issues, is much less detectable in undetected mode. You can try the v5.1 Docker image, but last time I tried I received the same 429 error.

[Royal2000H]

@jdholtz Sorry, I did miss that you used to use that project! I reverted to v5.1. Indeed I still get "Encountered a Too Many Requests error while logging in. Skipping reservation retrieval" on v5.1. So like you said, that is not the solution.

[dmytrokoren]

Guy and @jdholtz, I was able to successfully login with Docker on Synology :) YES!!!!

What I did, I gave full privileges to run container, BUT I started getting 400 error wrong credentials. I double checked my credentials they are correct.... next what I did was set check-price to "false" --- Voila successfully logged in.

I'm using version 5.1

In a bit I will test the latest images. Enjoy and hope this can help you.

[dmytrokoren]

Latest version 7.2 also works for me with Check Fare enabled.

I only need to enable MAC_ADMIN -- I do not need to give full privileges.

Everything works for me now in Docker for Synology - Hope this help everyone with notorious 429 error.

For Docker: The --cap-add and --cap-drop flags accept capabilities to be specified with a CAP_ prefix. The following examples are therefore equivalent:

$ docker run --cap-add=MAC_ADMIN ... $ docker run --cap-add=SYS_ADMIN ...

you can try either MAC_ADMIN or SYS_ADMIN whichever works for you.

[eiiot]

7.2 is working for me!

[lukeallister]

I made no change and my local docker deployment just notified me 'scheduled for check in'

I think that means something changed on the Southwest side, since I haven't upgraded since most recently having an issue.

On Thu, Feb 15, 2024, 1:23 PM Eliot Hertenstein @.***> wrote:

7.2 is working for me!

— Reply to this email directly, view it on GitHub https://github.com/jdholtz/auto-southwest-check-in/issues/201#issuecomment-1947362446, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFC3ZGOYAZ4AKLAPSRQC4JLYTZ4EJAVCNFSM6AAAAABBTESKY6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBXGM3DENBUGY . You are receiving this because you commented.Message ID: @.***>

[dmytrokoren]

Southwest did fix it on their end. I thought I was able to resolve it with adjusting the privileges. I guess 429 error was resolved today Feb 15th @ 1:50pm CST. :)

[Royal2000H]

Interesting. I just restarted my Docker container. For me, it's partially fixed. I have it set to login to 3 accounts. The first account I got an exception for:

Element {.login-button-box} was not present after 7 seconds!

The other two accounts after it logged in. Interesting because I thought this exception or a similar one was resolved in 7.2.

I restarted the container again. This time I got the

Encountered a Too Many Requests error while logging in. Skipping reservation retrieval

error for the first account. But then the 2 other accounts logged in after it. Not sure what Southwest does not like about the first account.

[dmytrokoren]

Interesting. I just restarted my Docker container. For me, it's partially fixed. I have it set to login to 3 accounts. The first account I got an exception for:

Element {.login-button-box} was not present after 7 seconds!

The other two accounts after it logged in. Interesting because I thought this exception or a similar one was resolved in 7.2.

I restarted the container again. This time I got the

Encountered a Too Many Requests error while logging in. Skipping reservation retrieval

error for the first account. But then the 2 other accounts logged in after it. Not sure what Southwest does not like about the first account.

I just tried to login to 3 different accounts and 2 were successful and 1 failed with 400 error (wrong creds) - I know the issue is because my password contains special character im not sure yet how to enter special character in config.json to pass correctly to southwest. But overall everything is back to normal and working.

[borski]

For now, just make your password alphanumeric.

[dmytrokoren]

For now, just make your password alphanumeric.

that's the workaround

[Royal2000H]

For now, just make your password alphanumeric.

The password on that account is alphanumeric for this reason. I'm mentioning still getting the 429 error.

@dmytrokoren I suggest making a separate issue for the 400 error as it can be confusing to try to cram two separate issues into this topic.

[jdholtz]

The "Invalid Credentials" 400 error was filed in #223. It is fixed in the develop branch/image.

[jdholtz]

@Royal2000H can you send the traceback you get of the Element not clickable...? Also, just to make sure, try the latest develop Docker image so the newest changes are used (Southwest has been changing their mobile API).

I just logged in successfully using the latest Docker develop image as well. I don't have any flights to test fare checking/check-in right now, but it seems like other people have been avoiding the 429 too. Southwest may have a rolling release cycle where they make their new website changes available to some people--which explains why some are still getting the issue and some are not (it also explains how some people had this 429 issue way earlier than others back in January).

[Royal2000H]

@jdholtz

This is the traceback (occurred on master branch). I'll switch to develop now.

Process Process-1: Traceback (most recent call last): File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/app/lib/reservation_monitor.py", line 42, in monitor self._monitor() File "/app/lib/reservation_monitor.py", line 157, in _monitor reservations, skip_scheduling = self._get_reservations() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/lib/reservation_monitor.py", line 182, in _get_reservations reservations = webdriver.get_reservations(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/lib/webdriver.py", line 84, in get_reservations driver.click(".login-button--box") File "/usr/local/lib/python3.12/site-packages/seleniumbase/core/sb_driver.py", line 59, in click page_actions.click(self.driver, args, **kwargs) File "/usr/local/lib/python3.12/site-packages/seleniumbase/fixtures/page_actions.py", line 1523, in click element = wait_for_element_clickable( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/seleniumbase/fixtures/page_actions.py", line 902, in wait_for_element_clickable timeout_exception(NoSuchElementException, message) File "/usr/local/lib/python3.12/site-packages/seleniumbase/fixtures/page_actions.py", line 266, in timeout_exception raise exc(msg) seleniumbase.common.exceptions.NoSuchElementException: Message: Element {.login-button--box} was not present after 7 seconds!

[Royal2000H]

Logged in to all 3 accounts successfully on develop. I guess I could have tried latest again prior to switching to develop, in case it was just a slower rollout to my account, similar to what you're suggesting. Regardless, seems to login fine!

[jdholtz]

This is the traceback (occurred on master branch). I'll switch to develop now.

Interesting, this error is actually different than what was fixed in v7.2. Seems like the check-in page wasn't successfully loading (possibly also due to more changes on the Southwest side). If you (or anyone else) encounters this again, you can file a separate issue for it.

[dmytrokoren]

The "Invalid Credentials" 400 error was filed in #223. It is fixed in the develop branch/image.

Thank @jdholtz I'll try the dev branch on my end. When you think you can push out v7.3? Since dev has the special character fix.

[jdholtz]

When you think you can push out v7.3?

I want to try and merge #203 first, so hopefully within the next few hours.

Edit: It may be tomorrow. I do want to do a few small things before making the next release.

[dmytrokoren]

Im getting this error right now in development branch for docker. note: the logs are in descending order

2024/02/15 23:39:48 | stdout | (Session info: chrome=121.0.6167.184)
-- | -- | --
2024/02/15 23:39:48 | stdout | selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <span class="login-button--box">...</span> is not clickable at point (1369, 29). Other element would receive the click: <div class="backdrop visible active"></div>
2024/02/15 23:39:48 | stdout | raise exception_class(message, screen, stacktrace)
2024/02/15 23:39:48 | stdout | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
2024/02/15 23:39:48 | stdout | self.error_handler.check_response(response)
2024/02/15 23:39:48 | stdout | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 348, in execute
2024/02/15 23:39:48 | stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024/02/15 23:39:48 | stdout | return self._parent.execute(command, params)
2024/02/15 23:39:48 | stdout | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webelement.py", line 395, in _execute
2024/02/15 23:39:48 | stdout | self._execute(Command.CLICK_ELEMENT)
2024/02/15 23:39:48 | stdout | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webelement.py", line 94, in click
2024/02/15 23:39:48 | stdout | element.click()
2024/02/15 23:39:48 | stdout | File "/usr/local/lib/python3.12/site-packages/seleniumbase/fixtures/page_actions.py", line 1526, in click
2024/02/15 23:39:48 | stdout | page_actions.click(self.driver, *args, **kwargs)
2024/02/15 23:39:48 | stdout | File "/usr/local/lib/python3.12/site-packages/seleniumbase/core/sb_driver.py", line 59, in click
2024/02/15 23:39:48 | stdout | driver.click(".login-button--box")
2024/02/15 23:39:48 | stdout | File "/app/lib/webdriver.py", line 83, in get_reservations
2024/02/15 23:39:48 | stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024/02/15 23:39:48 | stdout | reservations = webdriver.get_reservations(self)
2024/02/15 23:39:48 | stdout | File "/app/lib/reservation_monitor.py", line 182, in _get_reservations
2024/02/15 23:39:48 | stdout | ^^^^^^^^^^^^^^^^^^^^^^^^
2024/02/15 23:39:48 | stdout | reservations, skip_scheduling = self._get_reservations()
2024/02/15 23:39:48 | stdout | File "/app/lib/reservation_monitor.py", line 157, in _monitor
2024/02/15 23:39:48 | stdout | self._monitor()
2024/02/15 23:39:48 | stdout | File "/app/lib/reservation_monitor.py", line 42, in monitor
2024/02/15 23:39:48 | stdout | self._target(*self._args, **self._kwargs)
2024/02/15 23:39:48 | stdout | File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run
2024/02/15 23:39:48 | stdout | self.run()
2024/02/15 23:39:48 | stdout | File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
2024/02/15 23:39:48 | stdout | Traceback (most recent call last):
2024/02/15 23:39:47 | stdout | Process Process-3:
2024/02/15 23:39:46 | stdout | 2024-02-16 05:39:46 DEBUG Process-3[webdriver:79]: Logging into account to get a list of reservations and valid headers
2024/02/15 23:39:11 | stdout | 2024-02-16 05:39:10 DEBUG Process-3[webdriver:121]: Loading Southwest Check-In page
2024/02/15 23:39:10 | stdout | 2024-02-16 05:39:10 DEBUG Process-3[webdriver:117]: Using browser version: 121.0.6167.184

[jdholtz]

@dmytrokoren can you file a separate issue for this?

[dmytrokoren]

@dmytrokoren can you file a separate issue for this?

Yeah I'll do that.

[jdholtz]

Is anyone still running into the 429 issue when using the latest develop image? There is another issue with the Docker image (a 403, not a 429) being tracked in #225, but this is specifically for the 429 error.

[bradzab0623]

Is anyone still running into the 429 issue when using the latest develop image? There is another issue with the Docker image (a 403, not a 429) being tracked in #225, but this is specifically for the 429 error.

I still am as well... :(. I have it set up for several accounts. Some are getting the too many requests and some are making it past that and then getting 403s getting the reservations

2024-02-16 20:47:56 DEBUG Process-1[webdriver:159]: login_request_id set successfully
2024-02-16 20:47:56 DEBUG Process-1[webdriver:215]: Logging in failed for an unknown reason
2024-02-16 20:47:56 WARNING Process-1[reservation_monitor:199]: Encountered a Too Many Requests error while logging in. Skipping reservation retrieval
2024-02-16 20:47:56 DEBUG Process-1[reservation_monitor:179]: Lock released
2024-02-16 20:47:56 DEBUG Process-1[reservation_monitor:128]: Sleeping for 3590 seconds

[bradzab0623]

Is anyone still running into the 429 issue when using the latest develop image? There is another issue with the Docker image (a 403, not a 429) being tracked in #225, but this is specifically for the 429 error.

I still am as well... :(. I have it set up for several accounts. Some are getting the too many requests and some are making it past that and then getting 403s getting the reservations

2024-02-16 20:47:56 DEBUG Process-1[webdriver:159]: login_request_id set successfully
2024-02-16 20:47:56 DEBUG Process-1[webdriver:215]: Logging in failed for an unknown reason
2024-02-16 20:47:56 WARNING Process-1[reservation_monitor:199]: Encountered a Too Many Requests error while logging in. Skipping reservation retrieval
2024-02-16 20:47:56 DEBUG Process-1[reservation_monitor:179]: Lock released
2024-02-16 20:47:56 DEBUG Process-1[reservation_monitor:128]: Sleeping for 3590 seconds

Unfortunately restarting them all, different accounts successfully logged in and others that did previously got 429s. I think that indicates it's not a southwest slow rollout?

[bradzab0623]

Is anyone still running into the 429 issue when using the latest develop image? There is another issue with the Docker image (a 403, not a 429) being tracked in #225, but this is specifically for the 429 error.

I still am as well... :(. I have it set up for several accounts. Some are getting the too many requests and some are making it past that and then getting 403s getting the reservations

2024-02-16 20:47:56 DEBUG Process-1[webdriver:159]: login_request_id set successfully
2024-02-16 20:47:56 DEBUG Process-1[webdriver:215]: Logging in failed for an unknown reason
2024-02-16 20:47:56 WARNING Process-1[reservation_monitor:199]: Encountered a Too Many Requests error while logging in. Skipping reservation retrieval
2024-02-16 20:47:56 DEBUG Process-1[reservation_monitor:179]: Lock released
2024-02-16 20:47:56 DEBUG Process-1[reservation_monitor:128]: Sleeping for 3590 seconds

Unfortunately restarting them all, different accounts successfully logged in and others that did previously got 429s. I think that indicates it's not a southwest slow rollout?

I also randomly saw a few 400s...

2024-02-16 20:56:07 DEBUG Process-1[webdriver:159]: login_request_id set successfully
2024-02-16 20:56:07 DEBUG Process-1[webdriver:215]: Logging in failed for an unknown reason
2024-02-16 20:56:07 DEBUG Process-1[reservation_monitor:205]: Error logging in. Reason: Unknown. Status code: 400. Exiting
2024-02-16 20:56:07 DEBUG Process-1[notification_handler:84]: Sending failed login notification...
Failed to log in to account with username <username>. Reason: Unknown. Status code: 400.

[hildebrau]

I just did a docker-compose pull and restarted my docker image.. it was already pegged to the :develop tag. I saw some 403's for fare checks, and a 429 for a fare check. Requesting error during fare check. Too Many Requests 429. Skipping...

Remind me how to get the DEBUG logs again..

[dmytrokoren]

@bradzab0623 I haven't got the 429 since yesterday when the rollout happened. 400's I got yesterday but the develop branch fixed that since special characters was not being past correctly. Now 403 is a different issue seems like something is forbidden. I have no way of checking it since I have no reservations.

I tried to login with 3 accounts at same time via config.json and all accounts logged in without an issue.

I can try on my end with your reservations... idk if you want to.

[joewesch]

Remind me how to get the DEBUG logs again..

You can add this to your compose file for debug logs:

command: ["--verbose"]

[jdholtz]

I have no way of checking it since I have no reservations.

For @dmytrokoren and anyone else that doesn't have reservations (I also don't right now), the 403 error can still be reached by inputting fake reservation information.

I think that indicates it's not a southwest slow rollout?

@bradzab0623 possibly. I'm not exactly sure how rollouts are done (and it can be done many ways), but it could still be a staggered rollout because your computer could be routed to some of their servers that have the new changes and routed to others with the old changes still (you wouldn't be in control of where you are routed). If this were the case, it could explain why you receive different results for the same accounts

[dmytrokoren]

I have no way of checking it since I have no reservations.

For @dmytrokoren and anyone else that doesn't have reservations (I also don't right now), the 403 error can still be reached by inputting fake reservation information.

@jdholtz but he was login in to his account and from there the reservation was giving 403. So to be clear the login through account just grabs the reservation number and name and run though the same process as if you just give the name and reservation number?

Sorry I'm just trying to understand.

[jdholtz]

So to be clear the login through account just grabs the reservation number and name and run though the same process as if you just give the name and reservation number?

Exactly. Just add arguments like 4htgy6 <random first name> <random last name> and you can hit the 403 then. I'm guessing this is still the same as the 429 issue but just sent differently by the website.

[jdholtz]

Is anyone still getting a 429 error? If not, I will close this issue in favor of #230 which is now tracking a 403 error (which I believe is just an adjustment made by Southwest, but it is still the same underlying issue as this 429).

[bradzab0623]

nope. just the 403

[dmytrokoren]

No issue with 403 on the start of container. All works as expected. After 2-3 fair checks it starts giving 403 error and no reservations found. It might be that the token/key expires after certain period. That's the only way I can see why 403 is coming up. But if some are facing on container start than I don't know...

[mtnrun]

I got a 429 yesterday when I tried to check-in but that's probably a different issue compared to fare checks. First time using this, docker, latest build.

[joewesch]

Is anyone still getting a 429 error?

I'm still getting sporadic 403/429 while running the docker version in k3s when checking fares:

2024-02-24 21:00:33 DEBUG Process-1[reservation_monitor:93]: Checking fares for 2 flights
2024-02-24 21:00:33 DEBUG Process-1[fare_checker:30]: Checking current price for flight
2024-02-24 21:00:33 DEBUG Process-1[fare_checker:88]: Retrieving reservation information
2024-02-24 21:00:38 DEBUG Process-1[utils:39]: Failed to make request after 7 attempts: Forbidden 403
2024-02-24 21:00:38 DEBUG Process-1[utils:42]: Response body: {
  "code": 403050700
}
2024-02-24 21:00:38 ERROR Process-1[reservation_monitor:105]: Requesting error during fare check. Forbidden 403. Skipping...
2024-02-24 21:00:38 DEBUG Process-1[fare_checker:30]: Checking current price for flight
2024-02-24 21:00:38 DEBUG Process-1[fare_checker:88]: Retrieving reservation information
2024-02-24 21:00:40 DEBUG Process-1[utils:32]: Successfully made request after 1 attempts
2024-02-24 21:00:40 DEBUG Process-1[fare_checker:103]: Retrieving search information for the current flight
2024-02-24 21:00:44 DEBUG Process-1[utils:32]: Successfully made request after 4 attempts
2024-02-24 21:00:44 DEBUG Process-1[fare_checker:82]: Retrieving matching flights
2024-02-24 21:00:49 DEBUG Process-1[utils:39]: Failed to make request after 7 attempts: Too Many Requests 429
2024-02-24 21:00:49 DEBUG Process-1[utils:42]: Response body: {
    "code": 429999999,

    "message": "Error.",

    "messageKey": "ERROR",

    "httpStatusCode": "BAD_REQUEST",

    "requestId": "",

    "infoList": []
   }
2024-02-24 21:00:49 ERROR Process-1[reservation_monitor:105]: Requesting error during fare check. Too Many Requests 429. Skipping...

But, the most recent check worked just fine:

2024-02-26 21:00:27 DEBUG Process-1[reservation_monitor:93]: Checking fares for 2 flights
2024-02-26 21:00:27 DEBUG Process-1[fare_checker:30]: Checking current price for flight
2024-02-26 21:00:27 DEBUG Process-1[fare_checker:88]: Retrieving reservation information
2024-02-26 21:00:29 DEBUG Process-1[utils:32]: Successfully made request after 1 attempts
2024-02-26 21:00:29 DEBUG Process-1[fare_checker:103]: Retrieving search information for the current flight
2024-02-26 21:00:31 DEBUG Process-1[utils:32]: Successfully made request after 1 attempts
2024-02-26 21:00:31 DEBUG Process-1[fare_checker:82]: Retrieving matching flights
2024-02-26 21:00:33 DEBUG Process-1[utils:32]: Successfully made request after 1 attempts
2024-02-26 21:00:33 DEBUG Process-1[fare_checker:50]: Found 10 matching flights
2024-02-26 21:00:33 DEBUG Process-1[fare_checker:37]: Flight price change found for +1 USD
2024-02-26 21:00:33 DEBUG Process-1[fare_checker:30]: Checking current price for flight
2024-02-26 21:00:33 DEBUG Process-1[fare_checker:88]: Retrieving reservation information
2024-02-26 21:00:35 DEBUG Process-1[utils:32]: Successfully made request after 1 attempts
2024-02-26 21:00:35 DEBUG Process-1[fare_checker:103]: Retrieving search information for the current flight
2024-02-26 21:00:37 DEBUG Process-1[utils:32]: Successfully made request after 1 attempts
2024-02-26 21:00:37 DEBUG Process-1[fare_checker:82]: Retrieving matching flights
2024-02-26 21:00:39 DEBUG Process-1[utils:32]: Successfully made request after 1 attempts
2024-02-26 21:00:39 DEBUG Process-1[fare_checker:50]: Found 10 matching flights
2024-02-26 21:00:39 DEBUG Process-1[fare_checker:37]: Flight price change found for +29 USD

[dmytrokoren]

I'm getting 403 right on the start of container. I think southwest is making changes as we speak....

[jenke15]

Is anyone still getting a 429 error? If not, I will close this issue in favor of #230 which is now tracking a 403 error (which I believe is just an adjustment made by Southwest, but it is still the same underlying issue as this 429).

I experienced a 429 error on my two attempts to use this script, once on 2/20 and again on 2/27. Here is the verbose log for the most recent attempt:

2024-02-27 17:51:01 DEBUG Process-2:2[webdriver:119]: Using browser version: 122.0.6261.69
2024-02-27 17:51:01 DEBUG Process-2:2[webdriver:123]: Loading Southwest Check-In page
2024-02-27 17:51:06 DEBUG Process-2:2[webdriver:65]: Waiting for valid headers
2024-02-27 17:51:06 DEBUG Process-2:2[webdriver:152]: Waiting for headers_set to be set
2024-02-27 17:51:07 DEBUG Process-2:2[webdriver:156]: headers_set set successfully
2024-02-27 17:51:07 DEBUG Process-2:2[checkin_handler:99]: Lock released
2024-02-27 17:51:07 DEBUG Process-2:2[checkin_handler:103]: Sleeping until check-in: 1727 seconds...
2024-02-27 17:51:07 DEBUG Process-1:2[checkin_handler:96]: Lock acquired
2024-02-27 17:51:07 DEBUG Process-1:2[checkin_scheduler:50]: Refreshing headers for current session
2024-02-27 17:51:07 DEBUG Process-1:2[webdriver:103]: Starting webdriver for current session
2024-02-27 17:52:08 DEBUG Process-1:2[webdriver:119]: Using browser version: 122.0.6261.69
2024-02-27 17:52:08 DEBUG Process-1:2[webdriver:123]: Loading Southwest Check-In page
2024-02-27 17:52:13 DEBUG Process-1:2[webdriver:65]: Waiting for valid headers
2024-02-27 17:52:13 DEBUG Process-1:2[webdriver:152]: Waiting for headers_set to be set
2024-02-27 17:52:13 DEBUG Process-1:2[webdriver:156]: headers_set set successfully
2024-02-27 17:52:14 DEBUG Process-1:2[checkin_handler:99]: Lock released
2024-02-27 17:52:14 DEBUG Process-1:2[checkin_handler:103]: Sleeping until check-in: 1660 seconds...
2024-02-27 18:19:55 DEBUG Process-1:2[checkin_handler:124]: Attempting to check in
2024-02-27 18:19:55 DEBUG Process-2:2[checkin_handler:124]: Attempting to check in
2024-02-27 18:19:55 DEBUG Process-1:2[checkin_handler:138]: Making GET request to check in
Checking in to flight from 'Honolulu (Oahu)' to 'Las Vegas' for [Redacted]

2024-02-27 18:19:55 DEBUG Process-2:2[checkin_handler:138]: Making GET request to check in
2024-02-27 18:20:10 DEBUG Process-2:2[utils:39]: Failed to make request after 20 attempts: Too Many Requests 429
2024-02-27 18:20:10 DEBUG Process-2:2[utils:42]: Response body: {
    "code": 429999999,

    "message": "Error.",

    "messageKey": "ERROR",

    "httpStatusCode": "BAD_REQUEST",

    "requestId": "",

    "infoList": []
   }
2024-02-27 18:20:10 DEBUG Process-2:2[checkin_handler:147]: Failed to check in. Error: Too Many Requests 429. Exiting
2024-02-27 18:20:10 DEBUG Process-2:2[notification_handler:109]: Sending failed check-in notification...
Failed to check in to flight [Redacted] for [Redacted]. Reason: Too Many Requests 429.
Check in at this url: https://mobile.southwest.com/check-in

Let me know if any additional information is helpful here!

[tjs198]

not sure if this is helpful, I was get 403 error. I turned on VPM and was able to run a check-in using docker which it saying 'successfully scheduled the following flights to check in . . . ." I then tried to add a 2nd reservation and got the 403 error message. I switched the VPN to a new location and the 2nd reservation has been successfully scheduled . . .

[hildebrau]

I'm not sure what's happening with the comments, but some discussion on March 20th on other related github projects had mentioned an apiguard3 being the culprit.. I linked to that comment on here.. and now I can't find any mention of either comment.

Later, I saw someone point the finger at F5's acquisition of Shape Security. It's believed that is the software that is causing all the headaches for automation such as this. Read about it here: https://community.f5.com/kb/technicalarticles/what-is-shape-security/284359

Save this comment if you see it, because who knows how long it'll be here!?

I glanced through that article. I'm a network engineer, and, well, I wouldn't want to be on either side of this battle, as it's going to be tough for either side to "win". The bot security side has more money to throw at the problem, I fear.

I think the routine price checking, and likely the click rate and lack of javascript being triggered by human mouse movements is likely easily setting off some red flags. If you can emulate an erratic human using an actual browser, then this should still be possible. But, that may defeat the purpose of being first to check in, if you have to emulate slow typing and mouse movements, etc. ;)

[jdholtz]

I switched the VPN to a new location and the 2nd reservation has been successfully scheduled . . .

@tjs198 Thanks for the info. I don't think Southwest is blocking due to the location of the ISP because I have been able to hit a 403 on a server from the same ISP as my laptop which doesn't hit the 403. The inconsistency in the 403s could be explained by the article I quoted below which explains that AI is used to help detect bots (which isn't 100% effective).

Read about it here: https://community.f5.com/kb/technicalarticles/what-is-shape-security/284359

@hildebrau really interesting article, thanks for sharing. I actually think that's a really cool system and definitely am for it. I am not very familiar with bot detection--all that work is done by SeleniumBase--and trying to evade this system is something I don't want to spend a lot of time doing. I wanted to see how far I can take this project. Eventually, though, I know the cat-and-mouse game will end and with a system like this and 403s being hit by many people using this script, that end may be near.

[bradzab0623]

@jdholtz I think I speak for everyone when I say Don't give up!

[babehboi]

Although I do wonder, instead of checking the fares all at once maybe spread out each check within a timeframe? What I've noticed is that if we have several flights already booked, we're probably more likely hit the Too Many Requests 429 error since it would seem like we're spamming the site.

(It's happened to me more recently as I've booked more flights for my wife and I in the year). Since she probably ends up flying more than I and our itineraries at least for mine will align with hers, I just took my info off to reduce an additional check for X number of flights. There may be a magic number as to how many we should be concurrently checking.

Although, a though in improving, but may make it more complex is the ability to first parse the reservation #'s, and determine if you have any duplicate reservation numbers to reduce the # of responses we need to check for and/or processes that should occur?

Or even schedule out some of the processes so they're not necessarily immediately after +5m-10m after each process or even possibly add a randomization like discussed in some other previous post?

[jdholtz]

@jdholtz I think I speak for everyone when I say Don't give up!

Don't worry, I definitely won't. I'll keep focusing on making the project the best I can.

Although I do wonder, instead of checking the fares all at once maybe spread out each check within a timeframe?

@babehboi that probably would be an improvement and wouldn't be hard to implement. Can definitely look into this.

Although, a though in improving, but may make it more complex is the ability to first parse the reservation #'s, and determine if you have any duplicate reservation numbers to reduce the # of responses we need to check for and/or processes that should occur?

With the way the script is currently structured (a separate process for each account), that would be difficult to check for duplicates between reservations. Although this could help a little bit, the situation is quite rare and probably not worth it compared to the complexity it brings in the script. I think the simpler solution would be to do what you recommended above--spread out checking fares.

(It's happened to me more recently as I've booked more flights for my wife and I in the year)

Have you been getting a 429 only when checking fares? Or also while scheduling reservations or checking in?

[babehboi]

@jdholtz What I've noticed is that on the initial load and it's logging into my wife's account, it finds and schedules all the reservations. However shortly after that when it does try to check for fares, that's when I'm getting the errors. However, after 24h, it seems to go about its merry way. I'd want to say this is scheduling about 9 flights if I recall. Interestingly, by the time it gets to a friend's account it was able to check-in and check fares okay.

I should note it is a combination of error 429 & 403 as we've probably discovered earlier.

If it were up to me, I'd probably add 30s or some random value between 30-60s each of the fare checks? And possibly adding the confirmation number into the log too? I think it would help us clearly troubleshoot where/when it might be occurring? Because as of now, I won't know which reservation failed the checks versus which ones went through.

I can only make assume that after the 9 were scheduled the first 3 fare checks failed (429, 429, 403) then 2 passed then 4 failed again (403 x 4) (The time from which it failed to completed was about 20s from each other) next failure happened roughly 20s later.

[netwavetech]

As the fare prices normally does not change for a day or two, why not even spread them out more than that. I do not know how often the script actually runs stuff like this now, but why not create a routine that creates a random time between 5min and 15mins per say... Then the next check would run say at 6:18, the next one at 11:34, the next one at 7:02, etc...

[babehboi]

@netwavetech I believe it runs every 24h today, not sure how often fare prices change or if it's related to moon cycles, but I don't disagree with spreading it out a little more, as much as I'd like things "instantaneously" adding the human element of being slow doesn't hurt a bit in these instances. For what it can do is already great =)

[jdholtz]

If it were up to me, I'd probably add 30s or some random value between 30-60s each of the fare checks?

@babehboi I think it would be good to add this as a configurable value so its up to the user. If you have more flights, you can set this to a longer value, but it allows for setting it to 0 seconds if you only have 1 or 2 flights which 1. probably won't run into the 403/429 that you see due to the volume of requests and 2. not have to wait a long time to get the script's initial results.

As the fare prices normally does not change for a day or two, why not even spread them out more than that.

@netwavetech you can actually adjust how often the script checks for fares using the retrieval_interval which denotes how often to query for reservations and fare checks. Currently, there isn't a way to separate these two to use different intervals, but I can add it if its desired and there's enough reason to.

[babehboi]

@jdholtz I thought I'd share a thought since I implemented it recently to test my theory of the waiting bit because I noticed I was still consistently 429 or 403, both upon initial run of the script and the subsequent days. Some observations I have which lead me to believe it may be some number game/limitation as I didn't have these issues prior when I had roughly 6 or less legs to check, maybe even 5?

So I thought I'd add in a time.sleep(30) in the script just before it attempts to retrieve the reservations. Hopefully to slow things down a bit and see what the outcome was. Although it does seem to initially work, as I wasn't getting the errors immediately following the scheduling of the reservations, but eventually will lead to the same failures. I'll give it a day to see how it goes with subsequent checks, but considering that it goes in and rechecks all reservations and then leads to farecheck errors leads me to believe that their anti-bot detection is at play and works randomly or maybe lengthening the time for which it sleeps before the next check.

Which now makes me think, if I put them all as reservations rather than logging into the account if it makes a difference?

TL;DR: Attempted to add a delay in checking fares, worked for a bit but still fails. What other things can be tried.