Handling duplicated job

[syahnur197]

Since the worker will be running every night, there will be a situation where the worker might have scraped the existing job in the db. How do we address this?

[dsychin]

Using the job provider's primary key #38

The logic for the scraper would work something like this:

• Fetch html
• Parse html
• Loop through each job and parse the job Id and company Id from the url using regex
• Pass it to the JobService.UpdateJobs() method
• Service will perform the logic to update
  • Loop through jobs
  • If job id already exist, update entry
  • If not exist, insert new job
  • Copy company to company hash map
  • end job loop
  • Loop through company hash map
  • If company exist, update entry
  • If not exist, insert entry
  • finish

[dsychin]

Job and Company object includes the provider's job and company ID.

Still need to implement JobService to handle update/insert logic.