search

RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.25k stars 1.25k forks source link

Mysterious CI Error - could not lock config file #4034

Closed liangfok closed 6 years ago

liangfok commented 7 years ago

The Error

15:10:33 Cloning into 'drake/doc/textbook'...
15:10:33 Submodule path 'drake/doc/textbook': checked out 'e051917c15a513e1924465060134164c85fe5141'
15:10:33 error: could not lock config file /home/ubuntu/workspace/linux-gcc-ninja-continuous-matlab-debug/.git/modules/drake/doc/textbook/config: File exists
15:10:33 Failed to register url for submodule path 'HTMLBook'
15:10:33 Failed to recurse into submodule path 'drake/doc/textbook'

Example Console Logs

jwnimmer-tri commented 7 years ago

A prior example: https://drake-cdash.csail.mit.edu/viewBuildError.php?buildid=169296

Cloning into 'externals/xfoil'...
error: could not lock config file /home/ubuntu/workspace/linux-gcc-ninja-experimental/.git/modules/externals/xfoil/config: File exists
jwnimmer-tri commented 7 years ago

This happens up to a few times per week or more. Let get it onto Kitware's radar.

david-german-tri commented 7 years ago

I have yet to be inconvenienced myself, but @jamiesnape @BetsyMcPhail, PTAL?

liangfok commented 7 years ago

@hongkai-dai experienced this here: https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental-open-source-debug/804/console

jwnimmer-tri commented 7 years ago

Again https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-continuous-matlab-debug/494/

jamiesnape commented 7 years ago

Has anyone observed this on a non-Ninja build?

liangfok commented 7 years ago

I just experienced this issue here: https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental/1170/

Lucy-tri commented 7 years ago

Failing with snopt: linux-clang-ninja-continuous-debug-pre-drake

Cloning into 'externals/snopt'... error: could not lock config file /home/ubuntu/workspace/linux-clang-ninja-continuous-debug/.git/modules/externals/snopt/config: File exists

https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-clang-ninja-continuous-debug/1061/

jwnimmer-tri commented 7 years ago

Again https://drake-jenkins.csail.mit.edu/job/linux-clang-ninja-experimental-open-source/1166/

liangfok commented 7 years ago

Again: https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental/1365/console

liangfok commented 7 years ago

Again: https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental/1368/console

liangfok commented 7 years ago

Again: https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental/1372/

liangfok commented 7 years ago

Again: https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental/1431/

jamiesnape commented 7 years ago

We have enough information now and are trying to reproduce. Post if there is a failure that is not Linux and Ninja.

mwoehlke-kitware commented 7 years ago

RobotLocomotion/drake-ci#62 will hopefully give us some information if it happens again, and should also work around the problem by retrying the failed git command. Builds that have git errors should show as 'unstable'. (For build-cop purposes, these can be treated as "clean", but the strace output should be inspected for clues.)

SeanCurtis-TRI commented 7 years ago

This cropped up again. This time affecting libccd. Per @jamiesnape's request, this is in a mac build.

liangfok commented 7 years ago

@SeanCurtis-TRI can you provide a link to the console log or CI build page?

SeanCurtis-TRI commented 7 years ago

Sorry, here you go: https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/mac-clang-ninja-nightly-minimal-debug/160/

jamiesnape commented 7 years ago

(Aside: CDash links persist longer than Jenkins links)

SeanCurtis-TRI commented 7 years ago

I suspected that might be the case, which is why the original posting used a cdash link.

mwoehlke-kitware commented 7 years ago

The build should not have failed outright, however. Looking closer, there are issues other than the "could not lock config file" problem:

[9/46] cd /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/CMakeFiles/submodule-sync --  /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/ci/tools/git-wrapper.bash submodule --quiet sync --
fatal: Not a git repository: /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/.git/modules/externals/ccd
fatal: Not a git repository: /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/.git/modules/externals/ccd
[10/46] cd /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download --  /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/ci/tools/git-wrapper.bash submodule update --init --recursive -- externals/ccd && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download --  /usr/local/Cellar/cmake/HEAD-ce71bd9/bin/cmake -E touch /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download
FAILED: ccd-prefix/src/ccd-stamp/ccd-download 
cd /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download --  /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/ci/tools/git-wrapper.bash submodule update --init --recursive -- externals/ccd && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download --  /usr/local/Cellar/cmake/HEAD-ce71bd9/bin/cmake -E touch /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download
Cloning into 'externals/ccd'...
error: could not lock config file /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/.git/modules/externals/ccd/config: File exists
fatal: Could not set 'remote.origin.url' to 'https://github.com/danfis/libccd.git'
fatal: clone of 'https://github.com/danfis/libccd.git' into submodule path 'externals/ccd' failed
Cloning into 'externals/ccd'...
Submodule path 'externals/ccd': checked out '16b9379fb6e8610566fe5e1396166daf7106f165'

The failure despite succeeding on retry is due to a stupid logic error in the wrapper, fixed by RobotLocomotion/drake-ci#70. What is interesting however are the errors from the submodule sync attempt, which makes me wonder if all the "could not lock config file" errors are merely symptomatic of something having gone wrong earlier...

liangfok commented 7 years ago

Great job sleuthing!

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental-open-source-debug/2463

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental/2824/

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-xenial-clang-ninja-experimental-open-source/308/

jwnimmer-tri commented 7 years ago

Demoting high -> medium, because the git retry logic is helping, and because in BazelDrake land we won't be shelling out to git commands for submodules anymore.

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-xenial-cpplint-experimental/1038/

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-xenial-clang-ninja-experimental-open-source/1609/

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-xenial-gcc-ninja-experimental-open-source-debug/1711/

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-xenial-gcc-ninja-experimental/1982/

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-xenial-gcc-ninja-continuous-release/393/

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-xenial-gcc-ninja-experimental-open-source-debug/2314/

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-xenial-gcc-ninja-experimental/2351/

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/job/linux-xenial-clang-ninja-experimental-minimal/2836/

liangfok commented 7 years ago

https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-xenial-gcc-ninja-continuous-release/759/

liangfok commented 6 years ago

https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-xenial-clang-ninja-continuous-release/1336/