429 error despite using throttle

[KevinLinghu]

(I also posted this issue here: https://github.com/xur17/southwest-alerts/issues/17#issue-523915427)

I constantly got the following 429 error while executing this line:

                        available = southwest.get_available_flights_dollars(
                            departure_date,
                            origin_airport,
                            destination_airport
                        )

Error:

{
    "code": 429999999,

    "message": "Error.",

    "messageKey": "ERROR",

    "httpStatusCode": "BAD_REQUEST",

    "requestId": "",

    "infoList": []

}

Even with 30s throttle this still happen to every single query, so I'm thinking maybe it's caused by incorrect header, rather than too frequent calls?

[Fffrank]

@KevinLinghu I've updated the test branch with what I believe is (finally!) a fix. Care to try it out?

[hoopsbwc34]

Traceback (most recent call last):
  File "app.py", line 222, in <module>
    check_for_price_drops(user.username, user.password, user.email, user.headers, user.cookies)
  File "app.py", line 94, in check_for_price_drops
    southwest = Southwest(username, password, headers, cookies)
  File "/home/user/southwest-alerts/southwestalerts/southwest.py", line 18, in __init__
    self._session = _SouthwestSession(username, password, headers, cookies)
TypeError: __init__() takes 4 positional arguments but 5 were given

Fixed by adding cookies in southwest.py to both the session and object. Everything worked great after that.

But there is still something tripping 429 errors. I have about 10 confirmation numbers, the first time through it worked great, but starting with the second run of the script I immediately get a 429 when it tries to get the current price of my first itinerary. It successfully logs in and gets the value of my trip, but when it makes the call for the current price, it crashes with a 429

[Fffrank]

Try again -- I bunged up and didn't fully commit all of my changes to the test branch. It should be good now.

[hoopsbwc34]

Worked the first time through again.... every time after I get this:

`[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:60953/devtools/browser/01e2fb64-23e3-4809-84de-cf92681aeab7
[I:pyppeteer.launcher] terminate chrome process...
{"code":500599320,"message":"An unknown error has occurred.","httpStatusCode":"INTERNAL_SERVER_ERROR","requestId":"UJ5NlQCrSV6mFwDDMRpWNA-API"}
Traceback (most recent call last):
  File "app.py", line 220, in <module>
    check_for_price_drops(user.username, user.password, user.email, user.headers, user.cookies)
  File "app.py", line 93, in check_for_price_drops
    for trip in southwest.get_upcoming_trips()['trips']:
  File "/home/user/southwest-alerts/southwestalerts/southwest.py", line 16, in get_upcoming_trips
    '/api/customer/v1/accounts/account-number/{}/upcoming-trips'.format(self._session.account_number))
  File "/home/user/southwest-alerts/southwestalerts/southwest.py", line 101, in getb
    return self._parsed_response(resp, success_codes=success_codes)
  File "/home/user/southwest-alerts/southwestalerts/southwest.py", line 155, in _parsed_response
    'Invalid status code received. Expected {}. Received {}.'.format(success_codes, response.status_code))
Exception: Invalid status code received. Expected [200]. Received 500.`
``'
Maybe it's not clearing the cookies or something and using the previous session?

[Fffrank]

That's a weird one. I've never received a 500 error before and it indicates that they were having a server error. Possibly just bad luck and they had an outage?

[hoopsbwc34]

I ran it about 10 times... 9/10 were 500, one was 429. I then connected directly via firefox to see if I had my IP blocked or something (that's happened before) and it was fine. Could be something in my configuration but it's strange that it worked the first time again, then all resulting requests fail. Makes me think there is something with a mismatch in header config or something....

On Fri, Jan 10, 2020 at 9:09 AM Fffrank - notifications@github.com github.fsn.5649b70de0.notifications#reply@reply.github.com wrote:

That's a weird one. I've never received a 500 error before and it indicates that they were having a server error. Possibly just bad luck and they had an outage?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Fffrank/southwest-alerts/issues/2?email_source=notifications&email_token=ABZ4Z3U45LGDROTSUIJ4XI3Q5CMTHA5CNFSM4JOHSDE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIUMTAA#issuecomment-573098368, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZ4Z3R6WTOCXGKOPAUA43LQ5CMTHANCNFSM4JOHSDEQ .

[hoopsbwc34]

Worked the first time I ran it today again. Then all 429. Interesting to note that it always still pulls my first confirmation number so it's definitely connecting. The problem starts with "get_available_flights"

[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:44251/devtools/browser/224e3726-6bcc-4992-acb9-40b95db24cb0
[I:pyppeteer.launcher] terminate chrome process...
INFO:root:Processing: XXXXXX
INFO:root:itinerary original total price: 2879
{
    "code": 429999999,

    "message": "Error.",

    "messageKey": "ERROR",

    "httpStatusCode": "BAD_REQUEST",

    "requestId": "",

    "infoList": []
   }
Traceback (most recent call last):
  File "app.py", line 220, in <module>
    check_for_price_drops(user.username, user.password, user.email, user.headers, user.cookies)
  File "app.py", line 125, in check_for_price_drops
    destination_airport
  File "/home/matt/southwest-alerts/southwestalerts/southwest.py", line 66, in get_available_flights
    return self._session.get(url)
  File "/home/matt/southwest-alerts/southwestalerts/southwest.py", line 97, in get
    return self._parsed_response(resp, success_codes=success_codes)
  File "/home/matt/southwest-alerts/southwestalerts/southwest.py", line 155, in _parsed_response
    'Invalid status code received. Expected {}. Received {}.'.format(success_codes, response.status_code))
Exception: Invalid status code received. Expected [200]. Received 429.

[hoopsbwc34]

Update. I think as long as I wait for the login cookie to expire it runs fine. Trying to run the script back to back brings 429 errors immediately. I still get random 429 errors but it’s once in every 5 or 10 times I run it now. Might be worth some retry logic in the future on 429 rather than killing the script.

Also added texting through twilio and config file based setting of variables instead of using environment variables (just a personal preference).

Thanks!

On Fri, Jan 10, 2020 at 9:43 AM Matt Brown github.fsn@neverbox.com wrote:

I ran it about 10 times... 9/10 were 500, one was 429. I then connected directly via firefox to see if I had my IP blocked or something (that's happened before) and it was fine. Could be something in my configuration but it's strange that it worked the first time again, then all resulting requests fail. Makes me think there is something with a mismatch in header config or something....

On Fri, Jan 10, 2020 at 9:09 AM Fffrank - notifications@github.com github.fsn.5649b70de0.notifications#reply@reply.github.com wrote:

That's a weird one. I've never received a 500 error before and it indicates that they were having a server error. Possibly just bad luck and they had an outage?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Fffrank/southwest-alerts/issues/2?email_source=notifications&email_token=ABZ4Z3U45LGDROTSUIJ4XI3Q5CMTHA5CNFSM4JOHSDE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIUMTAA#issuecomment-573098368, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZ4Z3R6WTOCXGKOPAUA43LQ5CMTHANCNFSM4JOHSDEQ .

[Fffrank]

I'm pretty stumped on the errors because it's running flawlessly for me -- I have a paid booking and a points booking in my account and they both return the correct values and I can run the script 10 times in a row without errors.

If someone wants to add 'print(tempheaders)' above the return in "_get_headers_all" in southwest.py and post up what it results in, I'm happy to take a peek and see if there is something obvious going on that I can fix.

[hoopsbwc34]

New errors popped up today, I just checked in this confirmation number today and now I get this error. Might be a different format that gets returned after checkin

[I:pyppeteer.launcher] Browser listening on:
ws://127.0.0.1:54539/devtools/browser/db75e459-4f6b-487f-9015-23bd94639e37
[I:pyppeteer.launcher] terminate chrome process...
INFO:root:Processing: XXXXXX
Traceback (most recent call last):
  File "/home/user/southwest-alerts/southwestalerts/app.py", line 229, in
    check_for_price_drops(user.username, user.password, user.email,
user.headers, user.cookies)
  File "/home/user/southwest-alerts/southwestalerts/app.py", line 102,
in check_for_price_drops
    itinerary_price = cancellation_details['pointsRefund']['amountPoints']
KeyErr: 'pointsRefund'

Also finding I get this error a lot recently. Not sure what is going on with selenium.

 File "/usr/lib/python3.6/asyncio/base_events.py", line 488, in
run_until_complete
    return future.result()
  File "app.py", line 37, in login_get_headers
    await page.click(selector)
  File "/home/user/.local/lib/python3.6/site-packages/pyppeteer/page.py",
line 1507, in click
    await frame.click(selector, options, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/pyppeteer/frame_manager.py",
line 570, in click
    await handle.click(options)
  File "/home/user/.local/lib/python3.6/site-packages/pyppeteer/element_handle.py",
line 172, in click
    await self._page.mouse.click(x, y, options)
  File "/home/matt/.local/lib/python3.6/site-packages/pyppeteer/input.py",
line 306, in click
    await self.up(options)
  File "/home/user/.local/lib/python3.6/site-packages/pyppeteer/input.py",
line 345, in up
    'clickCount': options.get('clickCount') or 1,
pyppeteer.errors.NetworkError: Protocol error
Input.dispatchMouseEvent: Target closed.

[Fffrank]

@hoopsbwc34 I think you're having chromium version issues (not uncommon.) I've pushed an updated docker image for fffrank/southwest-alerts-test -- use that and report back. It's working for me at least!

[hoopsbwc34]

I'm not getting the chromium version issues... everything is fairly stable now. Only seeing a 429 error every once in a while. I did look at the temp headers and they are identical when it works and when I get a 429 error in the same session...so it's just some strange thing at the server side is my guess.

[hoopsbwc34]

@Fffrank are you planning to merge these changes into your master branch? I've made some changes for my own version to include retry on 429s and Twilio texting instead of email... I'd like to fork of your master branch rather than this test branch to store those in my own repository. Thanks again!

[hoopsbwc34]

Getting 429 every time after successfully getting the value of my first flight. Not a header issue must be something else. @Fffrank is it working for you?

[krazos]

@hoopsbwc34 @Fffrank - I've been having the same issue the past few days. I suspect that the "ee30zvqlwf-a" header is being truncated. In "tempheaders", the "ee30zvqlwf-a" header value always seems to be 4,542 characters in length, whereas the values of this header I am seeing in Chrome Dev Tools vary in length, but typically are 100-200 characters longer than that.

[Fffrank]

@hoopsbwc34 @krazos Since they've broken it so quickly the last few times that I've publicly updated I have to believe they are monitoring this.

There is no limit to the length of a string in python and I'm really not performing any operations on the header keys that could cause it to be truncated. I do see in my chrome dev that that particular key is usually longer but I have no idea why that would be (and obviously there's some math going on behind the scenes to try and prevent exactly what we are doing.) Seems likely they are detecting the pyppeteer instance somehow (and clearly they are better devs than I am.) At this point it's just totally broken.

If anyone wants to experiment with it -- you can copy the tempheaders and compare with chrome dev headers. You can also play with Insomnia (right click on the network request in chrome dev, copy as curl (cmd) and then paste into insomnia.) It's a really helpful tool to see what works and what breaks the request. I'm dropping the headers that are unnecessary already but it's that tricky *-a request header that really seems critical to match "something" at this point.

[Fffrank]

WOW.

Been racking my brain for weeks and testing everything in this script trying to find the problem. Was convinced it was headers and then cookies..... but..... it's neither.

My suspicions for why this was broken were completely wrong. The script is failing from the jump -- it's no longer logging in correctly with pyppeteer. I'm trying to figure this out now. I can't tell if they can somehow tell it's a pyppeteer instance (same error whether headless or not) but it shouldn't be impossible to solve.

[hoopsbwc34]

My guess... they are now detecting pyppeteer like they do with selenium.

On Thu, Mar 12, 2020 at 12:54 PM Frank - notifications@github.com github.fsn.5649b70de0.notifications#reply@reply.github.com wrote:

WOW.

Been racking my brain for weeks and testing everything in this script trying to find the problem. Was convinced it was headers and then cookies..... but..... it's neither.

My suspicions for why this was broken were completely wrong. The script is failing from the jump -- it's no longer logging in correctly with pyppeteer. I'm trying to figure this out now. I can't tell if they can somehow tell it's a pyppeteer instance (same error whether headless or not) but it shouldn't be impossible to solve.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Fffrank/southwest-alerts/issues/2#issuecomment-598384407, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZ4Z3WNKQLE26TSNW6E5E3RHE4YFANCNFSM4JOHSDEQ .

[Fffrank]

Pretty much dumb luck and trial and error but I do have it working again. Test branch is updated and will be pushing docker update (as well as merging with master shortly.)

[joshjohanning]

@Fffrank It works great! I have like 6 itineraries so I do eventually get a 429 - I wonder if we could add some retry logic to try again in that case.

Or do we have to go through the whole retrieve header process in that case?

[Fffrank]

Added some retry logic and fixed another error with flight credits causing it to error out. I've got 5 itens on my profile and it completes them every time now (occasionally times out waiting for headers but that's a bigger issue than I believe I'll be able to solve.) Going to close this issue but feel free to open another if/when southwest breaks it again.