search

opendevstack / ods-pipeline

Alternative ODS CI/CD pipeline based on Tekton / OpenShift Pipelines
Apache License 2.0
13 stars 5 forks source link

Implement caching without PVC? #535

Closed michaelsauter closed 1 year ago

michaelsauter commented 2 years ago

Currently the caching mechanism works by using one PVC for all pipelines of one repository. This has consequences on how many pipelines can run in parallel. We may want to rethink the caching approach. One alternative may be to create a gzipped tarball, upload it somewhere (e.g. pipeline manager) and download it at the beginning of the pipeline when appropriate. This could work for both dependency cache and build cache.

See also https://github.com/tektoncd/pipeline/issues/3097#issuecomment-673626583.

henrjk commented 2 years ago

I am doubtful that having the caches' content gzipped uploaded/downloaded would be viable from a performance angle. At the moment my working assumption is that the lack of parallelism is mitigated by build skipping and dependency caching. What scenarios make you want to increase parallelism? On the other hand one could perhaps enable experimentation towards this, by enabling pipelines without a PVC at all if that even is feasible where the build file system would be on local file storage.

michaelsauter commented 2 years ago

@henrjk I actually share your doubts regarding performance. Also, there is no use case right now that has reported a need for more parallelism. That said, in a repo with very long builds (think 20min+ or so), we could run into issues pretty quickly. Therefore I wanted to note this alternative down here. I do not see an immediate need to action on it, but it would be nice to collect thoughts and potentially run an experiment at some point.

michaelsauter commented 1 year ago

Closing this for now. We can revisit this when problems arise with the PVC approach currently in use. When that time comes, there may be new solutions in Tekton / OpenShift Pipelines that we can make use of then.