Caching resources for recipe and csi addons

abhijeet219 commented 4 months ago

Currently the csi-addon fetch kustomization resources whenever started. While, recipe addon do not use kustomization, this change introduces that. Also, this change will start using cache for those resources, so starting addons can directly use the cached resources.

Changes:

update csi-addon kustomization to start using static kustomization now.
cache csi-addon kustomization resources
recipe start using kustomization for its resources
cache recipe kustomization resources

Working:

drenv fetch can be used to fetch resources anytime.
Starting an addon will first try to fetch resources, then apply the fetched resources. If there is no change, fetch won't do anything, so takes very less time. Hence, starting an addon with cached resources is faster.

Example demonstrating how the cache works:

Fetching of resources for envs/regional-dr.yaml:

$ drenv fetch envs/regional-dr.yaml -v
2024-05-02 14:38:38,743 INFO    [rdr] Fetching
2024-05-02 14:38:38,745 DEBUG   [main] Add executor fetch
2024-05-02 14:38:38,746 INFO    [rdr] Running addons/rook-operator/fetch
2024-05-02 14:38:38,752 INFO    [rdr] Running addons/rook-cluster/fetch
2024-05-02 14:38:38,753 INFO    [rdr] Running addons/rook-toolbox/fetch
2024-05-02 14:38:38,759 INFO    [rdr] Running addons/csi-addons/fetch
2024-05-02 14:38:38,760 INFO    [rdr] Running addons/recipe/fetch
2024-05-02 14:38:38,764 INFO    [rdr] Running addons/ocm-controller/fetch
2024-05-02 14:38:38,872 DEBUG   [rdr] Fetching /home/abshakya/.cache/drenv/addons/rook-operator.yaml
2024-05-02 14:38:38,872 DEBUG   [rdr] Fetching /home/abshakya/.cache/drenv/addons/rook-cluster.yaml
2024-05-02 14:38:38,873 DEBUG   [rdr] Fetching /home/abshakya/.cache/drenv/addons/rook-toolbox.yaml
2024-05-02 14:38:38,881 DEBUG   [rdr] Fetching /home/abshakya/.cache/drenv/addons/ocm-controller.yaml
2024-05-02 14:38:38,885 DEBUG   [rdr] Fetching /home/abshakya/.cache/drenv/addons/csi-addons.yaml
2024-05-02 14:38:38,900 DEBUG   [rdr] Fetching /home/abshakya/.cache/drenv/addons/recipe.yaml
2024-05-02 14:38:39,115 INFO    [rdr] addons/rook-cluster/fetch completed in 0.36 seconds
2024-05-02 14:38:39,181 INFO    [rdr] addons/csi-addons/fetch completed in 0.42 seconds
2024-05-02 14:38:39,367 INFO    [rdr] addons/rook-toolbox/fetch completed in 0.61 seconds
2024-05-02 14:38:39,766 INFO    [rdr] addons/rook-operator/fetch completed in 1.02 seconds
2024-05-02 14:38:41,792 INFO    [rdr] addons/recipe/fetch completed in 3.03 seconds
2024-05-02 14:38:48,042 INFO    [rdr] addons/ocm-controller/fetch completed in 9.28 seconds
2024-05-02 14:38:48,043 INFO    [rdr] Fetching finishied in 9.30 seconds
2024-05-02 14:38:48,043 DEBUG   [main] Starting shutdown
2024-05-02 14:38:48,044 DEBUG   [main] Shutting down executor fetch
2024-05-02 14:38:48,044 DEBUG   [main] Terminating process group 131907

Fetching resources may take different amount of time for different runs. And once fetched, addons can directly use these resources to get started, saving time and escaping the flaky network situation.

nirs commented 3 months ago

@abhijeet219 the PR you based on was merged, you can rebase this PR.

nirs commented 3 months ago

@abhijeet219 The examples runs for csi-addons in the PR message are not helpful, lets remove them. This PR is about changing 3 addson, not sure why you show in detail changes in one addon?

abhijeet219 commented 3 months ago

Fixes: #1330

nirs commented 3 months ago

Fixes: #1330

See https://github.com/RamenDR/ramen/pull/1351/commits/ddb7226088cbd0421aaace461b6ba9bb85a75b87#r1588167619

RamenDR / ramen

Caching resources for recipe and csi addons #1351