Removal of inactive job listings

[dsychin]

Currently implementation adds new job listings but does not remove old ones.

2 ways to do this.

1. If the job page is no longer accessible when it is no longer valid, then just checking the page regularly and marking it appropriately should be fine.
2. During the scraper job, scrape all job listings and compare with all entries in the database and mark missing ones as inactive.

[syahnur197]

to implement number 2, maybe I can do it like this

1. when running cmd/jobbuzz-scraper.go, mark all jobs as inactive
2. Fetch all jobs
3. Loop all jobs, for each job, if exist, mark back as active, else insert

[syahnur197]

@dsychin what do you think of this approach? Is there any better approach?

[dsychin]

@syahnur197 I am leaning towards solution 1 at the moment. It has a few advantages:

• more scalable because we don't have to scrape everything all the time
• more resilient because we even if some pages fail it is still fine.
• similar flow can be used for both providers. e.g. if page is unavailable for jobcentre, then it is inactive. if entry is older than x days in bruneida, then mark deleted.

[dsychin]

Implement expiry date for job listings.

Use "last date to apply" for jobcentre listings, and an arbitrary time period for bruneida.