This no longer seems to work on Heroku...

[flyize]

As travel is going to start spinning back up, I'd love to use this again. Any idea if you'll have the time @samyun to take another look at this project?

[iloveitaly]

@flyize feel free to submit a PR fixing any errors with the app! I don't have time to investigate right now, but I'm sure we could get the PRs merged.

[flyize]

I totally wish I could.

[samyun]

What errors are you seeing @flyize?

[flyize]

Unfortunately, the error seem pretty useless, but here it is.

2021-03-23T20:18:11.173579+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=xyz.herokuapp.com request_id=e6f0169d-4f0a-4791-ae9b-e23ac5a1e245 fwd="47.x.y.z" dyno= connect= service= status=503 bytes= protocol=https

[flyize]

@samyun Is there an easy way for me to deploy this locally and test?

[iloveitaly]

@flyize npm run start should do the trick https://github.com/samyun/southwest-price-drop-bot/blob/47fb4125befa954d4d9d33db7e97f676fa61d799/package.json#L20

[flyize]

@iloveitaly It seems I need to install dependencies first. Clearly I have no idea what I'm doing. Is there an easy way to do that?

edit: Google says 'npm install' so I did that. But now I get this error:

> southwest-price-drop-bot@3.5.0 start /home/ubuntu/southwest-price-drop-bot
> node --trace-warnings index.js

app started on 34103
error connecting to mongo: MongooseError: The `uri` parameter to `openUri()` must be a string, got "undefined". Make sure the first parameter to `mongoose.connect()` or `mongoose.createConnection()` is a string.

[brussrus]

For me, I'm struggling to get this to consistently get prices. When running it headless, the browser seems to fail to pull in prices, but if I set the chrome debug (making it no longer headless), the browser windows open and I can consistently render the page and see prices. I'm using the latest puppeteer stealth plugin, and I don't see anything wrong when going to a headless detection site. That is, it appears to not be detected when in headless mode.

Anyone consistently getting this to work? I'm running this outside of Heroku on a local server, so I'm not contending with any blocking in that regard.

[iloveitaly]

@brussrus my hunch is SW's bot detection tool picks up chrome headless. I would try updating the stealth plugin and see if that helps. If it does, submit a PR!

[brussrus]

Tried that already. I'm running the latest versions of of the stealth plugin (2.7.8) and it still has issues. Basically, what I'm seeing is when I capture an image of the page when it fails, it is sitting at the SW site saying Sorry, we found some errors.... On a few occasions the same URL for fare lookup works flawlessly, but then the next time I run a check it will fail. I've even taking my code and put it on a separate Windows machine and same behavior. It's the same URL I would use if I were just on a normal browser and doing a search outside of this program.

One of the occasions when I was running the code with chrome_debug set to true, which forces the browser to appear, I watched as it loaded the page and had the Sorry error. Before the browser closed, I clicked the Search button on that page and it displayed the fares page and the application was happy. The samyun application isn't smart enough to click the Search button automatically if it has to, and I don't know if that is something that can be done.

That is why I was asking if anyone has this working without issue. Curious if they're running the current samyun code (which uses older modules), or if they tweaked theirs and it works because of modifications they made. Are they running it on Windows? What version of puppeteer are they using (which in turn will determine the version of Chromium).

I've had this working great for periods of time, but then it starts to fail quite regularly. In the past, I kept updating the stealth code to try and rectify it. Right now, I'm out of things to try.

[iloveitaly]

I don't have it running on my end. Curious how they are detecting headless—they must have some advanced fingerprinting going on.

[brussrus]

I'm not proficient with puppeteer at the moment, but I have hobbled together something that may work. I retooled the section that samyun had labelled TODO in the get-prices.js file because he apparently was having similar issues with the Sorry error popping up. I inspected the page more when I had that error, and like I mentioned their was a button on the page to submit your search. While it may not be pretty, I'm now searching for that selector first so that if found I can have it click the button, and then my assumption will be that the page will load fine. I don't know... like I said it's worked so far in the limited testing I've done, so fingers crossed. Lot of console messages and screenshots and commented out stuff from the original that could be cleaned up. Eventually, I could get some of that code up here if anyone is interested. For now, here is that little retooled section.

try {
  console.debug('Trying to find prices.');
await page.waitForSelector('#form-mixin--submit-button');
console.debug('Button found!! Click it!');
await Promise.all([
    page.waitForNavigation({waitUntil: 'networkidle0'}),
    page.click('#form-mixin--submit-button')
]);
  if (await page.waitForSelector('.price-matrix--details-titles', {timeout: 10000}) !== null) {

// await page.waitForSelector('.flight-stops'); await page.screenshot({ path: 'success-btn.png', fullPage: true }); console.debug('Got flight page!'); } else { console.debug('Trying to find button.'); await page.waitForSelector('#form-mixin--submit-button'); console.debug('Button found!! Click it!'); await Promise.all([ page.waitForNavigation({waitUntil: 'networkidle0'}), page.click('#form-mixin--submit-button') ]); await page.screenshot({ path: 'success-2.png', fullPage: true }); console.debug('Got flight page on second try!'); }

} catch (err) {
  // TODO assert that  "Sorry, we found some errors" does not exist
  if (await page.waitForSelector('.price-matrix--details-titles', {timeout: 10000}) !== null) {
    await page.screenshot({ path: 'success-init.png', fullPage: true });
    console.debug('Got flight page!');
  } else {
console.log('Unable to get flights - trying again');
await page.screenshot({ path: 'failed.png', fullPage: true })
console.log(await page.content());
console.log(response.headers());
console.log(response.status());

// try { // await page.goto(url); // await page.waitForSelector('.price-matrix--details-titles'); // await page.waitForSelector('.flight-stops'); // } catch (err) { // const currentUrl = page.url(); // const errHtml = await page.evaluate(() => document.body.outerHTML); // await page.goto('about:blank'); // await page.close();

// if (errHtml.includes('Access Denied')) { // throw new Error('ERROR! Access Denied Error! Unable to find flight information on page: ' + currentUrl + '\nhtml: ' + errHtml); // } else { // throw new Error('ERROR! Unknown error! Unable to find flight information on page: ' + currentUrl + '\nhtml: ' + errHtml); // } // } } }

[flyize]

Well I think I figured out why it doesn't work on Heroku anymore. It seems it requires mongolab, which has been discontinued.

https://devcenter.heroku.com/changelog-items/1823

[flyize]

Anyone got any ideas?

[iloveitaly]

@flyize mongodb has a free db you can signup for: https://www.mongodb.com/pricing (which is probably why the heroku-native dbs went out of business).

[flyize]

Thanks! Seems I've got it running locally now. But alas, its not pulling any data from SW.

[brussrus]

I've got it working locally, but I'm using my own fork as I've modified the get-prices code (and a few other files and some updated versions of puppeteer, etc.) to account for some of the errors I was getting when trying to go to Southwest's pages. I also added some code in because I wanted to be able to specify number of passengers as well. Sometimes the prices for a single ticket are lower than if you need to buy multiples and the original code always assumed 1 passenger. Don't know if that will help you with your locally running version.

I gave up on Heroku a couple years ago, so can't really assist with that.

[flyize]

Oh, I see the error now. Any ideas how to fix this?

/home/ubuntu/southwest-price-drop-bot/node_modules/puppeteer/.local-chromium/linux-686378/chrome-linux/chrome: error while loading shared libraries: libXcomposite.so.1: cannot open shared object file: No such file or directory

[brussrus]

Not running linux myself, but what package version of Puppeteer do you have installed? Perhaps try updating it to the latest version will get you a newer node_module for it, and a more current version of chromium. npm install puppeteer@latest, and while you're at it look at updating your puppeteer-extra-plugin-stealth package.

[flyize]

@brussrus omg I got yours to work!

[iloveitaly]

@brussrus Do you have things working locally on the branch you have a separate PR?

[brussrus]

Yes. I'm running the code from my fork, and currently have 6 flights that I've been watching with different passenger counts, etc. I run it locally on a Windows machine, with a Task Scheduler task that kicks off every 30 minutes to check prices. Been working well for me.

[flyize]

@brussrus Does your version not run in the background and check automatically? Is there something else I need to do to make it check? I'm running it as a service in Ubuntu.

[brussrus]

Running the main program as a service (I'm guessing npm start) will spawn the web server piece and connect into mongodb. So if you go to your local web page you'll be able to add flights, etc. and that service will do an initial pull of prices, but it is not intended for recurring checks. For that, given you are Ubuntu, I would assume you'd set up some kind of cron job to periodically kick off and run the command: npm run task:check

It's that command that will go and check the prices. For me, since I'm on a Windows platform, I've created a scheduled task (equivalent of cron job) to run every 30 minutes. The command it runs is the one I mentioned above, and it runs from the main directory of this application.

If that is not what you are referring to, then perhaps I'm missing something.

[flyize]

Ahhh! No I don't have anything running a task check. That's an important piece that I missed. This was handled automatically somehow by Heroku?

edit: Also, any idea of the format for the Plivo number? I seem to recall when you add a new reservation that it sends a 'welcome' text. I'm not getting that, so I'm assuming something is wrong.

[brussrus]

That is correct. If you look at the readme for this project you'll see after deploying to Heroku you: Open up the Heroku Scheduler from your app's dashboard Add an hourly task that runs npm run task:check

So you need to do the equivalent in Ubuntu of creating something on a schedule. In your case, the reason I suggested a cron job.

[razzamatazm]

@brussrus - I had your branch working, but am now getting javascript errors. Is puppeteer still working for you, or is it throwing errors for everyone?

Thanks for you work on this project- i thought it was dead!

[flyize]

Is there any way to test Plivo?