[discussion] Deployment on production server

[alvinlys]

Hi, anyone would like to share on the CICD design of cron using bree in production as reference because i am concerning on the possibility of

• job runs twice due to restarting the server
• job fail to run as it passed over the specific date due to slow deployment

[shadowgate15]

job runs twice due to restarting the server

I would either use specific dates to ensure that doesn't happen or make sure your pre-defined jobs can handle being run mutliple times. This could be done by either using some form of tracking when a job has been run or using db fields to element running over a document multiple times. See this example.

job fail to run as it passed over the specific date due to slow deployment

Currently this is an active issue. In v8, there will be a warning printed when this occurs, but I still haven't come up with a good way to handle this situation. As we do not want jobs to run if the date is past, however we also don't want that to happen silently. I have been toying with the idea of adding a worker skipped event that would fire under this condition then the setup code can handle the situation.

[alvinlys]

Thanks @shadowgate15 for your informative reply.

job runs twice due to restarting the server

if this the case, then how the other solve this issue? Are they using microservice with other language to perform such task? Beside that, due to async await in Nodejs, I noticed those interval jobs are not performed on time, is there a way to solve this issue too? Thanks

[shadowgate15]

if this the case, then how the other solve this issue? Are they using microservice with other language to perform such task?

I suggest using some form of external storage that changes when the job has been run, for example changing a database field or Redis storage. We do have a plan to eventually add adaptor plugins for easier connection to databases or pub/sub systems (Redis). However, that will happen when I have time since I don't get to work on Bree full-time ☹️.

If you are running into an issue with jobs not starting at the correct times I would suggest running the Bree setup on a separate server/process. There will always be some minor discrepancy due to NodeJS's event loop, however this discrepancy should be pretty small, like a few milliseconds max. If you are seeing large discrepancies, it probably indicates that there is too much happening in that process or the code is blocking the event loop for extended periods of time. Bree's logic to start jobs is pretty fast and shouldn't cause huge delay's in the event loop.

[alvinlys]

Ya. After considerations, I had to dockerize at another server to prevent slow deployment. And for external storage, currently I am using redis to prevent job running twice.

Really hope anyone would share your opinion or practice of using it in production whenever come across this discussion

[zanechua]

Would really love to understand more about how to use a db backed layer

I'm thinking of a scenario where if you have a dockerized node application that runs bree and you deploy it to a k8s or something like ecs fargate that spins up the containers as and when required.

What would be the way to solve the following problem if we have e.g. a jobs table in a MySQL database

First container retrieves and writes is_lock property Second container retrieves the job BEFORE the first container writes the is_lock property.

The above scenario would result in the job being processed twice right?

[titanism]

We have several examples in the Forward Email code base for jobs with persistence. Further discussion on this topic belongs in https://github.com/breejs/bree/discussions. Bree handles your workers and is agnostic to how you configure your setup. Some folks may not need a persistence layer, while others may require something more complex. Bree makes you do manage state with your own DB or own architecture. We did provide several examples in the README and in our Forward Email codebase.