SoftwareDesignLab / nvip-crawler

Crawler project for National Vulnerability Integlligence Platform
MIT License
0 stars 0 forks source link

PNE/Patchfinder Job Streaming #163

Closed dylan-mulligan closed 1 year ago

dylan-mulligan commented 1 year ago

Modify the PNE/PF to be able to process 1 job at a time and send out 1 job at a time. We want to move to a system where we can gracefully shut down these components in case of updates, or if they're unexpectedly shut down at least jobs aren't lost. For graceful shutdown, that's going to have to involve sending "stop" messages through the rabbit queues. Fortunately, rabbit supports priority queues, so if we give those STOP messages the highest priority and jobs are 1 CVE each, then components can finish the job they're on, get the STOP message and shut down. The remaining unfinished jobs will remain in the queue until the system is started back up so those jobs aren't lost.