search

mozilla / bedrock

Making mozilla.org awesome, one pebble at a time
https://www.mozilla.org
Mozilla Public License 2.0
1.18k stars 918 forks source link

Test files are persisting between branches #7866

Closed jgmize closed 5 years ago

jgmize commented 5 years ago

Description

Files created in the jenkins workspace during a test run triggered by a push to the run-integration-tests branch are persisting between runs, causing test failures.

The following failure, from https://ci.vpn1.moz.works/blue/organizations/jenkins/bedrock_multibranch_pipeline/detail/run-integration-tests/380/pipeline#step-163-log-531 is FAILED tests/functional/firefox/privacy/test_products.py::test_download_button_displayed while running commit b707c247e but as can be seen in https://github.com/mozilla/bedrock/tree/b707c247e/tests/functional/firefox this directory and file do not exist in that commit.

Looking at the working directory, /var/lib/jenkins/jobs/bedrock_multibranch_pipeline/branches/run-integration-tests/workspace@5, I found that there were a large number of files from previous commits (including the culprit above) that should have been cleaned out during the make clean build step of the pipeline, which runs a git clean -f command. Running this command manually did in fact remove those files from the workspace (and then I iterated through the other workspaces to do the same) but the docker images previously built from this working directory still contain the files. To address the specific failure above, I ran docker rmi mozorg/bedrock_test:b707c247e32f7274d703c2b5a5c064c51f30dee2 -f.

jgmize commented 5 years ago

I was mistaken when I thought that git clean -f was sufficient to remove the files; after further research I found that we needed the -d flag to remove untracked directories.

jgmize commented 5 years ago

@alexgibson after running git clean -d -f in each of the run-integration-tests/workspace* dirs I reran the last failed pipeline, and while it didn’t pass it is at least not failing on tests that shouldn’t exist: https://ci.vpn1.moz.works/blue/organizations/jenkins/bedrock_multibranch_pipeline/detail/run-integration-tests/384/pipeline