P1 - Separate Cron Tasks From Impact-Graph

[mhmdksh]

In an attempt to make impact-graph replicable and running in a high availability environment. Making this a reality concerns running multiple impact-graph instances at the same time which are all connected to the same DB and doing the same operations depending on the load and the traffic.

One thing that is preventing that from happening is embedding a lot of cronjobs inside impact-graph, making it impossible to run in a replicable manner.

Something that can help achieve that is separating these crons from impact-graph and make them run as a separate service that is talking to and endpoint on impact-graph which is called from somewhere else.

This will make us one step closer to run impact-graph in high-availability infra (Like Kubernetes or others)

@jainkrati @aminlatifi @mohammadranjbarz @CarlosQ96 I would appreciate your opinion on this

[mhmdksh]

@mohammadranjbarz I'll need your help to group all the cronjobs that we are currently embedding in impact-graph that we can detach. My current list if the cronjobs is the below:

CHECK_PROJECT_VERIFICATION_STATUS_CRONJOB_EXPRESSION
CHECK_USERS_SUPER_TOKEN_BALANCES_CRONJOB_TIME
DONATION_SAVE_BACKUP_CRONJOB_EXPRESSION
FILL_BLOCK_NUMBERS_OF_SNAPSHOTS_CRONJOB_EXPRESSION
FILL_POWER_SNAPSHOT_BALANCE_CRONJOB_EXPRESSION
IMPORT_LOST_DONATIONS_CRONJOB_EXPRESSION
INSTANT_BOOSTING_UPDATE_CRONJOB_EXPRESSION
MAKE_UNREVIEWED_PROJECT_LISTED_CRONJOB_EXPRESSION
MATCH_DRAFT_DONATION_CRONJOB_EXPRESSION
REVIEW_OLD_GIV_PRICES_CRONJOB_EXPRESSION
SYNC_GIVING_BLOCKS_CRONJOB_EXPRESSION
SYNC_IDRISS_TWITTER_DONATIONS_CRONJOB_EXPRESSION
SYNC_POIGN_ART_CRONJOB_EXPRESSION
UPDATE_POWER_ROUND_CRONJOB_EXPRESSION
VERIFY_DONATION_CRONJOB_EXPRESSION
VERIFY_RECURRING_DONATION_CRONJOB_EXPRESSION

[mohammadranjbarz]

@mohammadranjbarz I'll need your help to group all the cronjobs that we are currently embedding in impact-graph that we can detach. My current list if the cronjobs is the below:

CHECK_PROJECT_VERIFICATION_STATUS_CRONJOB_EXPRESSION
CHECK_USERS_SUPER_TOKEN_BALANCES_CRONJOB_TIME
DONATION_SAVE_BACKUP_CRONJOB_EXPRESSION
FILL_BLOCK_NUMBERS_OF_SNAPSHOTS_CRONJOB_EXPRESSION
FILL_POWER_SNAPSHOT_BALANCE_CRONJOB_EXPRESSION
IMPORT_LOST_DONATIONS_CRONJOB_EXPRESSION
INSTANT_BOOSTING_UPDATE_CRONJOB_EXPRESSION
MAKE_UNREVIEWED_PROJECT_LISTED_CRONJOB_EXPRESSION
MATCH_DRAFT_DONATION_CRONJOB_EXPRESSION
REVIEW_OLD_GIV_PRICES_CRONJOB_EXPRESSION
SYNC_GIVING_BLOCKS_CRONJOB_EXPRESSION
SYNC_IDRISS_TWITTER_DONATIONS_CRONJOB_EXPRESSION
SYNC_POIGN_ART_CRONJOB_EXPRESSION
UPDATE_POWER_ROUND_CRONJOB_EXPRESSION
VERIFY_DONATION_CRONJOB_EXPRESSION
VERIFY_RECURRING_DONATION_CRONJOB_EXPRESSION

What do you mean by separating them? Do you want to create another repo and move to there? if you want to do it it's very hard because they use lots of common functions and entities, My suggestion is to bring up multiple instance of impact-graph, in one of them the cronjobs are enabled in other instances the jobs are disabled, it's easier to manage them

[mhmdksh]

@mohammadranjbarz I'll need your help to group all the cronjobs that we are currently embedding in impact-graph that we can detach. My current list if the cronjobs is the below:

CHECK_PROJECT_VERIFICATION_STATUS_CRONJOB_EXPRESSION
CHECK_USERS_SUPER_TOKEN_BALANCES_CRONJOB_TIME
DONATION_SAVE_BACKUP_CRONJOB_EXPRESSION
FILL_BLOCK_NUMBERS_OF_SNAPSHOTS_CRONJOB_EXPRESSION
FILL_POWER_SNAPSHOT_BALANCE_CRONJOB_EXPRESSION
IMPORT_LOST_DONATIONS_CRONJOB_EXPRESSION
INSTANT_BOOSTING_UPDATE_CRONJOB_EXPRESSION
MAKE_UNREVIEWED_PROJECT_LISTED_CRONJOB_EXPRESSION
MATCH_DRAFT_DONATION_CRONJOB_EXPRESSION
REVIEW_OLD_GIV_PRICES_CRONJOB_EXPRESSION
SYNC_GIVING_BLOCKS_CRONJOB_EXPRESSION
SYNC_IDRISS_TWITTER_DONATIONS_CRONJOB_EXPRESSION
SYNC_POIGN_ART_CRONJOB_EXPRESSION
UPDATE_POWER_ROUND_CRONJOB_EXPRESSION
VERIFY_DONATION_CRONJOB_EXPRESSION
VERIFY_RECURRING_DONATION_CRONJOB_EXPRESSION

What do you mean by separating them? Do you want to create another repo and move to there? if you want to do it it's very hard because they use lots of common functions and entities, My suggestion is to bring up multiple instance of impact-graph, in one of them the cronjobs are enabled in other instances the jobs are disabled, it's easier to manage them

@mohammadranjbarz Thanks for the suggestion, but I would say that making one version of impact-graph different than the others makes this whole idea of replicability worthless.

If we want to worry which version on impact-graph is deciding what cronjobs are run, then it is an additional complication that is in the way of making it ready for scaling.

@geleeroyale WDYT?

[mhmdksh]

Mentioning @jainkrati @mohammadranjbarz @divine-comedian @aminlatifi @geleeroyale @Rolazo for more engagement.

[divine-comedian]

I think it sounds good, we do maintain a lot of cron jobs and we are planning to add more!

I don't have a good grasp on the amount of work this requires or what kind of work needs to happen.

What would be the definitive PROs and CONs of this change? How many dev hours do we estimate this work would need?

[geleeroyale]

Look what I found - I will use this to move givfarm-notify jobs from an outdated server

https://github.com/mcuadros/ofelia

Edit: Its a bit of a tough system to set up, but I still like the possibilities. Its more geared towards running cron jobs in docker containers (which was not my use case - I wanted to run cron jobs on the host)

so for refactoring impact-graph jobs out of impact-graph but still use the container this seems to be perfect