All custom test commands seem to always be executed on file changes for a given image

oversizedhat commented 2 years ago

Expected behavior

When declaring a list of custom test commands in skaffold yml its expected that commands only run if the changes are in files defined in their respective dependencies block.

Actual behavior

When 2 or more commands in a custom block for a image are defined in the skaffold yml it seems that ALL commands are executed regardless if the file changes is defined in the specific commands dependency block or not.

This seems to happen regardless if the command has a dependencies block defined or not. I.e. even if command 1 has a dependencies block containing a different path than command 2 both commands are executed if a change is made in any of the paths.

Information

Skaffold version: 1.36.1, 1.35.2
Operating system: Ubuntu 18.04.6 LTS
Installed via: skaffold.dev
Contents of skaffold.yaml:

apiVersion: skaffold/v2beta27
kind: Config
build:
  artifacts:
  - image: custom-test-example
test:
  - image: custom-test-example
    custom:
      - command: ./test.sh
        timeoutSeconds: 60
        dependencies:
          paths:
          -  "*_test.go"
          -  "test.sh"
      - command: echo Hello world!!
        #dependencies:
        #  command: echo [\"main_test.go\"] 
deploy:
  kubectl:
    manifests:
      - k8s-*

Steps to reproduce the behavior

It seems easy to reproduce in the example project.

clone this project git@github.com:GoogleContainerTools/skaffold.git
cd examples/custom-tests
Comment out the dependencies block for the Hello world command
skaffold dev
Make any change in main_test.go and observe how both commands are executed in the console.

Same issue if you comment out the dependencies block for the ./test.sh command. Ie both commands still execute on a change in main_test.go.

tejal29 commented 2 years ago

Thanks you for the issue. We would appreciate any help from the community on this issue.

kadern0 commented 2 years ago

Hi @tejal29, I've been playing around with this and got skaffold to run as described here. Personally, I don't think this is an issue but the intended behavior. Dependencies are "associated" to images, therefore all tests will be run if any of the dependencies change, otherwise the dependency model for custom tests would need to be more complex (mandatory dependencies, default deps, etc...). Also the documentation says:

dependencies | additional test-specific file dependencies; changes to these files will re-run this test.

It's specifically mentioned "re-run this test", not re-run this command.

What do you think?

oversizedhat commented 2 years ago

@tejal29 @kadern0 Thanks for looking into this! I would however consider it an issue. If you have a few different scopes of testing to be executed it becomes very time consuming to run everything whenever some unrelated test is changed. Also the setup of being able to configure dependency paths for certain test commands does not make sense if the config is disregarded and all tests are executed anyway. Then the dependencies block should be moved upwards instead of being per command.

But maybe I am looking at this the wrong way? As this is not working I get the feeling that no-one is running unit level tests using skaffold. Is that not the purpose?

The GoogleCLoudPlatform microservices sample app is for example running unit level tests in a very custom fashion instead of relying on the skaffold test phase. https://github.com/GoogleCloudPlatform/microservices-demo/blob/main/.github/workflows/ci-master.yaml

ericzzzzzzz commented 2 years ago

Hi @oversizedhat . Thank you for the feedback. I think you're right. Dependencies are owned by command, if a dependency change, only the command owns should be run. The expected behavior you describe makes a lot of sense for TDD. I think this is something we should fix. @tejal29 @aaron-prindle

GoogleContainerTools / skaffold