PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.
MIT License
1.78k stars 210 forks source link

New CI with Github Actions. #127

Closed thebigG closed 3 years ago

thebigG commented 3 years ago

New CI with Github Actions

Hi everyone, hope you are all doing well.

Description

As we discussed on this issue #126, TravisCI is no longer an option as it has a time limit not just on individual builds, but every account only gets an X amount of minutes it can use for its lifetime. After that, you'll have to pay. This is why I have setup GitHub actions for our CI needs. The only issue I ran across with Github Actions was the fact that if the max_listing_days was set to something as high as 35, JobFunel will error out for some reason. This is an issue that needs further investigation; I don't know the exact reason why this happens. I'll document this odd issue at some point in the future.

In the meantime, Github Actions seems to work without a time limit. I have updated the trigger branches (push and pull_request keys on github actions configuration), so it should get triggered when this PR gets merged. Fingers crossed :sunglasses: . The .travis.yml file has been deleted. Check commit history for details.

After this gets merged, and if all goes well, I'll submit a PR to update the readme's CI badge to report Github Actions status instead of TravisCI.

Context of change

Please add options that are relevant and mark any boxes that apply.

Type of change

Please mark any boxes that apply.

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration.

Checklist:

Please mark any boxes that have been completed.

thebigG commented 3 years ago

It got triggered. We are in business :+1:.

thebigG commented 3 years ago

@PaulMcInnis Will merge :+1: . If it fails, I have a feeling Indeed might be showing a popup message when we send the request and so we are not getting the HTML because there is some javascript executing before we get the HTML page. Something similar has happened to me when I try to scrape Indeed without being signed in, which would make sense given that Github server does not "log in". In any case, if it fails, I'll post an issue and will investigate further. At the very least now we don't have the time limit we had on TravisCI anymore.