search

multicloud-ops / build-harness-extensions

1 stars 8 forks source link

add change image back and forth #35

Open zshearin opened 4 years ago

zshearin commented 4 years ago

This is what I am currently using to test bvt of a feature branch. This one is not as clear-cut as to what we want to do (feedback to potentially change would be greatly appreciated), but I have found this useful and have been using it locally.

Two assumptions this makes:

  1. the APP variable is defined in your BuildHarnessConfig.local file and it is the same as your current repo operator name (or operand repo name)

  2. your operator yaml file lives in directory deploy/operator.yaml - this I believe will change in future versions of the operator-sdk, that's why I thought it makes sense to have it defined above instead of in-line

This make target would also just be used for feature branch builds prior to promoting to master. What I am currently doing is this (in my feature dev test):

build/push change-image deploy test (k8s job runs container and completes if successful, errors if not) change-image-back delete (deployment and fvt job - unless it failed then leave it to be able to debug results)

This is just another common function that would also make scratch deployments a little easier (specifically for monitoring squads)

hassenius commented 4 years ago

If I understand you right, it kindof tries to achieve the same as https://github.com/multicloud-ops/build-harness-extensions/blob/master/modules/operatorcatalog/Makefile, however, operatorcatalog/local/createrelease creates a single item catalog entry based on merging CSV info with image overrides generated from operatorcatalog/local/overridetemplate

However, the single-item-catalog idea turns out to be flawed with OLM, because the package will exist in two different catalogs (original and dev), and there will be a bit of a flip-flop effect between the two catalogs, so you never know which one you get (the original, or the one with the dev version). This means that sometimes OLM will upgrade to the new local versions you're creating, sometimes not....

I was also contemplating doing some experiments with configmap-server type from https://github.com/operator-framework/operator-registry to see whether that might offer a possible approach to "testing upgrades of one operator in the context of a bunch of other related operators".

Maybe this approach here is more straight forward?

zshearin commented 4 years ago

Right - that was what we were trying to mimic.

We have several tests that we would run prior to pushing to master (i.e. validation steps in addition to unit testing to ensure functionality is maintained - a form of integration testing). This was also intended to be used in isolation from other components - deploy our dependencies and have them live on the system we are testing, then deploy our operator and create the cr that our operator handles and run a kubernetes job that runs our integration tests. The desired output is either good or bad - we wouldn't need to keep anything created by the build - we would just want to indicate whether the changes made still maintain our desired functionality in an integrated environment.