search

crystal-lang / shards

Dependency manager for the Crystal language
Other
758 stars 99 forks source link

Use of 'git checkout' causes intermittent failure due to git index.lock contention #547

Open russdill opened 2 years ago

russdill commented 2 years ago

When doing multiple simultaneous builds, the shards install step sometimes fails with the below message:

I: Resolving dependencies
I: Installing crc (1.0.1)
E: Failed git --work-tree=/home/russd/src/foo-service/worktrees/5a1e87e36c16413334b5818f596002c80af1243e/lib/crc checkout refs/tags/v1.0.1 -- . (). Maybe a commit, branch or file doesn't exist?

An examination of strace output shows more detail:

[pid 44168] write(2, "fatal: Unable to create '/home/russd/.cache/shards/github.com/postmodern/crystal-crc.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n", 393) = 393

The changeset here: https://github.com/crystal-lang/shards/pull/435 causes the issue. Shards either needs to figure out how to make git archive work on windows, utilize a git shallow clone, retry git checkout on failures, or have some other way to obtain the git source.

russdill commented 2 years ago

Setting the environmental variable GIT_INDEX_FILE to an lib/.<dependency>.index may also work in this situation.

straight-shoota commented 2 years ago

Yeah, git really isn't supposed to be run in parallel. But shards should be. So we'll have to manage.

What do you mean with "make git archive work on windows"? You're apparently not on Windows, right? (EDIT: Ah I realize it's commenting on the reasoning for #435)

I don't understand what a shallow clone would help here. The error has nothing to do with cloning, it happens when checking out a work tree.

Using GIT_INDEX_FILE looks like a good idea. shards checks out a shared git repo to (potentially) multiple work trees. So every work tree needs its own index. That how git worktree operates as well. There it's not just index because it's also used for making changes, but shards only needs to read.

As a side effect, keeping a local index per worktree checkout should probably improve performance for updates.

russdill commented 2 years ago

A clone would not touch the index of the bare repo so would not run into this issue. Making the clone shallow just avoids copying a whole bunch of unnecessary data.

But yes, I think the GIT_INDEX_FILE is the way to go. I've been testing that locally and things seem to be working. Although I'm not sure if shards using GIT_* variables from the environment when cloning dependencies and copying their work-trees is desired behavior or not. It's at least helpful in this case.