Sotera / watchman

Watchman: An open-source social-media event-detection system
GNU General Public License v2.0
20 stars 7 forks source link

Add JobSet entity for better JobMonitor mgmt #57

Closed lukewendling closed 7 years ago

lukewendling commented 7 years ago

Purpose

Let's revisit job running and scheduling for a more robust solution to processing historical data from QCR. We previously identified the need for a parent entity of JobMonitors, and the new entity is potentially a good candidate to drive the improved job scheduler.

Process Flow

max_retries = job interval * multiplier / loop interval

job set exists?
  N
    create
    (a) posts count > MIN_POSTS?
      N
        retry count == max_retries
          N
            increment retry counter
            exit
          Y
            set state = 'skip'
            exit
      Y
        set state = 'running'
        create monitors
  Y
    job set running?
      N
        goto (a)    
      Y
        exit