Development roadmap

[datnguyenzzz]

Consist of 2 main component

Scheduler

Understand job definition from ENV variables

• Reader which read from ENV and convert to a CronJob object

Implementation of Quartz (library for scheduling)

• Scheduler
• Job
• Trigger
• Listener
• Concurrency

AWS SDK

[datnguyenzzz]

• [x] Need CronJobConfiguration Component, where load ENV properties

Must be provided env:

• CRON_JOB_PROVIDER: local/s3
• CRON_JOB_LOCATION: yaml file define the cron job trigger and job's content
• HEALTH_CHECK_TIME_WINDOW
• CONSECUTIVE_ERROR_POSTPONE_THRESHOLD: const - postpone job if consecutive error exceed threhold
• PERCENTAGE_ERROR_POSTPONE_THRESHOLD: const - postpone job if percentage's error exceed threhold
• AWS_REGION
• AWS_ACCESS_KEY
• AWS_SECRET_KEY
• AWS_CREDENTIALS_TARGET: file, in which provide aws credentials

[datnguyenzzz]

CronJob file format:

jobList:
  - name: "Upload to SQS #1"
    cronTrigger: "0 0 0 1 2 2/3"
    usedService: "SQS"
    messages:
      - key: "key-1"
        value: "value-1"
      - key: "key-2"
        value: "value-2"

  - name: "Upload to SQS #2"
    usedService: "SQS"
    afterJobDone: "Upload to SQS #1"
    lambdaActionFile: "abc/def/xyz.java"

  - name: "Upload to SQS #3"
    cronTrigger: "0 12 2 1 2/3 * *"
    usedService: "SQS"
    lambdaActionFile: "xyz/ghi/dfc.java"

[datnguyenzzz]

CronJob Factory contain

• [x] Read Job definition from local system
• [ ] Read Job definition from cloud storage
• [x] Yaml file reader
• [x] Tree execution job
• [ ] Execution logic is violated exception

[datnguyenzzz]

Start quartz scheduler

• [x] Job trigger by defined BFS order
• [x] Each job create a instance of Quartz Job with PublisherFactory property

JobFactory.newJob: Called by the scheduler at the time of the trigger firing, in order to produce a Job instance on which to call execute.

• [ ] Multiple scheduler (have distributed manner)
• [ ] Equally distribute job across all Scheduler (consistency hashing)

[datnguyenzzz]

AWSPublisherFactory - define with method publisher will be used (SQS, Kinesis, Custom)

• [ ] Add file path of custom java job
• [ ] Compile it at runtime
• [ ] Load class byte code at runtime

[datnguyenzzz]

PublishingJob implement Job

• [x] autowired AWSPublisher

• [x] Because PublishJob is Bean -> need get instance of Bean in when sheduler call executer

• [x] Pass usedService paramenter into Job description, in order to get corresponding Bean ....AWSPublisher

• [x] Disallow concurrent functionality of PublishingJob (Example, if job is trigger every 5s, but a job need 10s to complete)

• [ ] Handle misfired job

• [x] Persist number of job executed in data map

• [ ] REST API for adding new job

[datnguyenzzz]

PublishigTrigger

[datnguyenzzz]

Health check service: - quartz schedule health check job base periodically

• [x] Persist job data with status after fired
• [x] Periodically yelling statuses
• [ ] REST API with json return

[datnguyenzzz]

AWS SDK using