search

rentea-tw / rentea-crawler

A crawler that provides timely response to data change on public rental house platform
MIT License
5 stars 2 forks source link

Verify crawler performance and find best parameter #4

Open ddio opened 5 years ago

ddio commented 5 years ago

As a crawler engineer, I want to know appropreciate minuteago setting, so

  1. Crawler can perform all its work in minuteago 24 hours a day, 7 days a week.
  2. Reduce target website #request at best
  3. The higher update frequency the better

Acceptance Criteria

  1. All of goal mentioned above
  2. Provide a mechanism to ensure crawler won't be executed simultaneously even when it take longer than minuteago
ddio commented 5 years ago

Action Items

  1. [x] Detail test plan
  2. [ ] Implement job collision avoidance alg.