redfern314 / southwest-search

Search, sort, and filter Southwest flights based on a number of parameters.
MIT License
13 stars 5 forks source link

no form matching name 'buildItineraryForm' #1

Closed jipis closed 6 years ago

jipis commented 6 years ago

It appears Southwest has changed their form layout. It seems that the form no longer even has a name element. In fact, the form page has completely changed location, even.

I attempted to fix it myself to at least lead you towards a solution, but my python is WAY rusty.

redfern314 commented 6 years ago

Hi @jipis - thanks for bringing this up! Yeah, SW unfortunately did a full rework of their site. Because of the way they're now using javascript to load and render pages, I was only able to get this to work again by using a full browser emulator (Selenium) which is an order of magnitude slower than the original lightweight parser. Check out the branch I just pushed up, feature/selenium. It's slow, but it does work. Make sure to follow the new install instructions in the README.

Let me know what you think, and I'll keep looking for other ways to do this.

jipis commented 6 years ago

Yep, it indeed works. Slowly. But, I should point out, STILL faster than using their website! :) And, on the plus side, the code is way more readable and intuitive, actually reading things from the DOM rather than the regexes you needed in the old code.

Any reason in particular you upped the sleep between calls to 5 seconds from 2? When I did a search with one destination airport, two departure airports, and two dates (4 searches), 20 of the 70 seconds it took was just sleeping. I almost feel like -- if using selenium, which has such a long setup time on its own -- you might not need to sleep at all any more. Of course, my machine is old as dirt, so YMMV and something less than 8 years old might not take nearly as much time to set up selenium before each call as mine does.

Thanks!

redfern314 commented 6 years ago

Mostly because I couldn't get the implicit wait functionality to work. 5 seconds was just the time it took on average to let the page load all the elements. Someone more familiar with Selenium might be able to fix that.

I actually did some digging into how the site works on the backend and think I have a better solution. Raw POST calls instead of anything to do with the DOM. Will post back when I have it pushed.

redfern314 commented 6 years ago

Pushed a new commit into master which is much faster and more readable. Curious to see how long it'll be until they break it again, but should be good for now!