error "package is dirty, bailing" on windows

[JeffBezanson]

I tried to switch to latest master of ZMQ on windows using Pkg.checkout("ZMQ",pull=true). I got "ERROR: ZMQ is dirty, bailing", but git status shows that the working tree is clean, and on a normal released version. Manually going to the directory and doing git checkout master followed by git pull worked.

[staticfloat]

I wonder if these errors aren't red herrings: it looks like Git.dirty() is implemented as:

dirty(; dir="") = !success(`diff --quiet HEAD`, dir=dir)

Which could return true if there was some problem in invoking git diff, e.g. git paths being wrong, the git executable not even being found, etc....

[StefanKarpinski]

The executable not being found should at least throw an error but it's true that other issues might return a false dirty status here. I don't know a better way to test this, unfortunately. Any ideas?

[staticfloat]

It doesn't look like git paths being set incorrectly can cause this error, unfortunately. (Or perhaps fortunately, depending on whether or not you want things to be broken)

I don't know of any better way to test.

[stevengj]

Probably same as #4328

[stevengj]

Shouldn't Pkg.build wrap the build script in Git.transact or similar so that failed build attempts don't leave the ZMQ directory in a dirty state? But I'm not sure why BinDeps should ever make ZMQ dirty.

[quinnj]

I've been seeing this as well: https://groups.google.com/forum/#!topic/julia-users/MGl42Hc2qWc

I've tried uninstalling my msysgit and using the version provided by msys2, reinstalling msysgit, changing all the core.autocrlf settings, etc. I'd dig into it more, but I'm really not sure how the Base.Git, Base.Pkg are implemented.

[quinnj]

p.s. This happens on any and all packages I try it with (not just ones with build dependencies).

[staticfloat]

@karbarcca can I get a versioninfo() from you?

Also, let's try running the git commands manually as Base.Git tries to:

julia> Pkg.dir("BinDeps")
"/home/sabae/.julia/BinDeps"

julia> gitcmd = Base.Git.git(Pkg.dir("BinDeps"))
`git --work-tree=/home/sabae/.julia/BinDeps --git-dir=/home/sabae/.julia/BinDeps/.git`

julia> run(`$gitcmd status`)
# Not currently on any branch.
nothing to commit, working directory clean

julia> success(`$gitcmd diff --quiet HEAD`)
true

If you can show these we should be able to rule out git<-->julia interaction as the problem.

[quinnj]

Sorry for the delay, I've been having some julia build problems. Here's the output of the above commands:

julia> versioninfo()
Julia Version 0.2.0-prerelease+3836
Commit b5c797c 2013-09-24 17:42:42 UTC
Platform Info:
  System: Windows (x86_64-w64-mingw32)
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
  LAPACK: libopenblas
  LIBM: libopenlibm

julia> Pkg.dir("BinDeps")
"C:\\Users\\karbarcca\\AppData\\Roaming\\Julia\\packages\\BinDeps"

julia> gitcmd = Base.Git.git(Pkg.dir("BinDeps"))
`git '--work-tree=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps' '--git-dir=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps\.git'`

julia> run(`$gitcmd status`)
fatal: index file open failed: Invalid argument
ERROR: failed process: Process(`git '--work-tree=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps' '--git-dir=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps\.git' status`, ProcessExited(128)) [128]
 in error at error.jl:22
 in pipeline_error at process.jl:447
 in success at process.jl:439
 in spawn_opts_inherit at process.jl:370
 in run at process.jl:424

julia> success(`$gitcmd diff --quiet HEAD`)
false

[quinnj]

p.s. The fatal: index file open failed error happens each and every time I try to add a package.

[pao]

Possibly related to this reported git bug (if git is sufficiently new)? http://www.spinics.net/lists/git/msg217305.html

[pao]

(That's to @karbarcca's fatal: index file open failed--I'd need to read more closely to see if that could be causing the problem @JeffBezanson reported.)

[quinnj]

@pao, thanks for the tip! Indeed, switching back to git-1.8.3 fixes the fatal: index file open failed error. Unfortunately, still seeing the package is dirty message when trying to checkout packages' master.

[vtjnash]

Do you have a git configuration file? e.g. anything in .gitconfig in your home directory? Or, what does git say:

julia> ;git config -l
core.autocrlf=true
core.filemode=false
core.ignorecase=true

core.safecrlf core.eol core.autocrlf core.ignoreStat

[quinnj]

Mine says

julia> ; git config -l
core.symlinks=false
core.autocrlf=true
color.diff=auto
color.status=auto
color.branch=auto
color.interactive=true
pack.packsizelimit=2g
help.format=html
http.sslcainfo=/bin/curl-ca-bundle.crt
sendemail.smtpserver=/bin/msmtp.exe
diff.astextplain.textconv=astextplain
rebase.autosquash=true
user.email=quinn.jacobd@gmail.com
user.name=Jacob Quinn

[vtjnash]

What about git --version? Is it possible you have a cygwin git in your path? I notice that the msys2 git appears to be badly broken when called from julia.

[quinnj]

That was one of my original problems, but after changing my msys2 git to _git.exe, I finally got past some of the other errors I was seeing, but still see the package dirty bailing error when trying to checkout.

julia> ;git --version
git version 1.8.3.msysgit.0

[vtjnash]

Very strange, I figured that was fine, but just don't see where your configuration is different. Can to post the output of the git commands from earlier?

[quinnj]

julia> Pkg.dir("BinDeps")
"C:\\Users\\karbarcca\\AppData\\Roaming\\Julia\\packages\\BinDeps"

julia> gitcmd = Base.Git.git(Pkg.dir("BinDeps"))
`git '--work-tree=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps' '--git-dir=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps\.git'`

julia> run(`$gitcmd status`)
# HEAD detached at 00bdc84
nothing to commit, working directory clean

julia> success(`$gitcmd diff --quiet HEAD`)
true

julia> Pkg.checkout("Datetime")
INFO: Checking out Datetime master...
ERROR: Datetime is dirty, bailing
 in anonymous at no file:120
 in transact at git.jl:69

julia> Pkg.checkout("BinDeps")
INFO: Checking out BinDeps master...
ERROR: BinDeps is dirty, bailing
 in anonymous at no file:120
 in transact at git.jl:69 (repeats 2 times)

julia> 

[vtjnash]

interesting:

julia> g=Base.Git.git("ZMQ")
`git '--work-tree=C:\Users\julia\AppData\Roaming\Julia\packages\ZMQ' '--git-dir=C:\Users\julia\AppData\Roaming\Julia\packages\ZMQ\.git'`

julia> Base.Git.dirty(dir="ZMQ")
true

julia> success(`$g diff --exit-code HEAD`)
false

julia> success(`$g diff -w --exit-code HEAD`)
true

julia> success(`$g diff --ignore-space-at-eol --exit-code HEAD`)
true

julia> success(`$g diff -a --exit-code HEAD`)
false

[StefanKarpinski]

Woah, man. That's bananas. What's going on here?

[pao]

Could different values for core.autocrlf have something to do with it?

[JeffBezanson]

Hopefully we can keep thinking about this and testing during the RC period, and if we don't solve it in a week or two release 0.2 without a fix.

[vtjnash]

git status fixes / does not have this issue

$ git status || echo 1
# On branch master
nothing to commit, working directory clean
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ touch REQUIRE
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ git diff --quiet || echo 1
1
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ git status
# On branch master
nothing to commit, working directory clean
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ git diff --quiet || echo 1
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ touch REQUIRE
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ git diff --quiet || echo 1
1
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$

@StefanKarpinski Could we replace success(git diff --quiet) with isempty(readall(git status --porcelain))

[vtjnash]

Alternatively, git update-index --really-refresh also "fixes" the call to git diff

[StefanKarpinski]

This only happens on Windows, right? Should this be considered an upstream bug? How does git update-index --really-refresh help?

[pao]

Should this be considered an upstream bug?

Maybe, but I think @vtjnash's earlier suggestion of git status --porcelain makes good sense--it's in a guaranteed stable format, doesn't rely on git's seemingly random choices for exit codes, and if there are changes to the working copy it should be faster than computing the diff (if only slightly).

[vtjnash]

I assume this is an issue with the file system on Windows being absurdly slow. So cheap operations on linux (like lstat and listing the files in a directory) are instead being cached on Windows, to make is usable.

[StefanKarpinski]

@vtjnash, can you please elaborate on how git update-index --really-refresh helps? Does calling that right before calling git diff --quiet make this work?

[vtjnash]

Yes

[StefanKarpinski]

Let me know if that fixes the problem – hopefully it does.

[stevengj]

Can someone please file an issue for this upstream with the git folks?

[quinnj]

Starting from no .julia directory.

julia> Pkg.init()
INFO: Initializing package repository C:\Users\karbarcca\.julia
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl

julia> Pkg.add("Datetime")
INFO: Cloning cache of Datetime from git://github.com/karbarcca/Datetime.jl.git
fatal: This operation must be run in a work tree
ERROR: failed process: Process(`git '--git-dir=C:\Users\karbarcca\.julia\.cache\Datetime' update-index --really-refresh`, ProcessExited(128)) [128]
 in pipeline_error at process.jl:449
 in run at process.jl:426
WARNING: backtraces on your platform are often misleading or partially incorrect

julia> Pkg.clone("Datetime")
INFO: Cloning Datetime from git://github.com/karbarcca/Datetime.jl.git
Cloning into 'Datetime'...

julia> Pkg.checkout("Datetime")
INFO: Checking out Datetime master...
INFO: Pulling Datetime latest master...
INFO: No packages to install, update or remove.

julia> Pkg.add("Datetime")
INFO: No packages to install, update or remove.
INFO: REQUIRE updated.

julia> Pkg.add("DataFrames")
INFO: Cloning cache of Blocks from git://github.com/tanmaykm/Blocks.jl.git
fatal: This operation must be run in a work tree
ERROR: failed process: Process(`git '--git-dir=C:\Users\karbarcca\.julia\.cache\Blocks' update-index --really-refresh`, ProcessExited(128)) [128]
 in pipeline_error at process.jl:449
 in success at process.jl:441
 in run at process.jl:426

julia> 

checkout seems to work, but add is now wonky.

[StefanKarpinski]

Sigh. But it's a bit closer, I guess.

[StefanKarpinski]

Ah, ok, I think I know what the deal with this is.

[StefanKarpinski]

Hopefully this is now fully fixed...

[quinnj]

Again starting from a new .julia

julia> Pkg.init()
INFO: Initializing package repository C:\Users\karbarcca\.julia
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl

julia> Pkg.add("Datetime")
INFO: Cloning cache of Datetime from git://github.com/karbarcca/Datetime.jl.git
INFO: Installing Datetime v0.1.2
INFO: REQUIRE updated.

julia> Pkg.checkout("Datetime")
INFO: Checking out Datetime master...
ERROR: Datetime is dirty, bailing
 in anonymous at no file:133
 in transact at git.jl:81
WARNING: backtraces on your platform are often misleading or partially incorrect

julia> Pkg.clone("DateFrames")
ERROR: can't determine package name from URL: DateFrames
 in clone at pkg/entry.jl:125

julia> Pkg.clone("DataFrames")
INFO: Cloning DataFrames from git://github.com/JuliaStats/DataFrames.jl.git
Cloning into 'DataFrames'...
INFO: Computing changes...
INFO: Cloning cache of Blocks from git://github.com/tanmaykm/Blocks.jl.git
INFO: Cloning cache of GZip from git://github.com/kmsquire/GZip.jl.git
INFO: Cloning cache of SortingAlgorithms from git://github.com/JuliaLang/SortingAlgorithms.jl.git
INFO: Cloning cache of Stats from git://github.com/JuliaStats/Stats.jl.git
INFO: Installing Blocks v0.0.0
INFO: Installing GZip v0.2.5
INFO: Installing SortingAlgorithms v0.0.0
INFO: Installing Stats v0.2.8

julia> Pkg.checkout("DataFrames")
INFO: Checking out DataFrames master...
ERROR: DataFrames is dirty, bailing
 in anonymous at no file:133
 in transact at git.jl:81
 in write at stream.jl:734

julia> 

[StefanKarpinski]

Plz try one more time...

[staticfloat]

@karbarcca just out of curiosity, what's your versioninfo()?

[quinnj]

Winner winner!

julia> Pkg.init()
INFO: Initializing package repository C:\Users\karbarcca\.julia
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl

julia> Pkg.add("Datetime")
INFO: Cloning cache of Datetime from git://github.com/karbarcca/Datetime.jl.git
INFO: Installing Datetime v0.1.2
INFO: REQUIRE updated.

julia> Pkg.checkout("Datetime")
INFO: Checking out Datetime master...
INFO: Pulling Datetime latest master...
INFO: No packages to install, update or remove.

julia> Pkg.clone("DataFrames")
INFO: Cloning DataFrames from git://github.com/JuliaStats/DataFrames.jl.git
Cloning into 'DataFrames'...
INFO: Computing changes...
INFO: Cloning cache of Blocks from git://github.com/tanmaykm/Blocks.jl.git
INFO: Cloning cache of GZip from git://github.com/kmsquire/GZip.jl.git
INFO: Cloning cache of SortingAlgorithms from git://github.com/JuliaLang/SortingAlgorithms.jl.git
INFO: Cloning cache of Stats from git://github.com/JuliaStats/Stats.jl.git
INFO: Installing Blocks v0.0.0
INFO: Installing GZip v0.2.5
INFO: Installing SortingAlgorithms v0.0.0
INFO: Installing Stats v0.2.8

julia> versioninfo()
Julia Version 0.2.0-rc1+141
Commit 8d9fdf4 2013-10-24 20:18:27 UTC
Platform Info:
  System: Windows (x86_64-w64-mingw32)
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
  LAPACK: libopenblas
  LIBM: libopenlibm

julia> 

[StefanKarpinski]

Phew. What a relief!

[quinnj]

Thanks for the work everyone. This is definitely great to have for the rc.

[vtjnash]

@StefanKarpinski this problem exists on Linux too (it's a configuration option) -- the defaults are just different:

$ git config  diff.autorefreshindex false
$ git diff --quiet HEAD || echo 1
$ touch README.md
$ git diff --quiet HEAD || echo 1
1
$ git status > /dev/null
$ git diff --quiet HEAD || echo 1
$

[StefanKarpinski]

Well then can we just set the option? Please test and tell me if that works. Why is there a default option to be broken?

[vtjnash]

well, the user could change the option: this option makes git much faster on windows, or on large repositories.

[StefanKarpinski]

I don't really care. If this breaks things we will change the option every time. I can't respect options that break everything in random ways.

[vtjnash]

Or we could apply the same fix on linux?

[StefanKarpinski]

If this fix is the fault of an idiotic config option, we should revert that fix and just force the option to something that isn't broken. It's not like this option changes how anything reasonable works, it is literally just an option to sacrifice correct behavior for dubious performance improvement.

[StefanKarpinski]

Seriously. The existence of the option is why we can't have nice things.

[vtjnash]

The config documentation for that option seems to suggests that you should be calling git update-index --refresh anytime you are using git diff from a script. That option is apparently just there to make life a bit easier on platforms that have fast file I/O.

       diff.autorefreshindex
           When using git diff to compare with work tree files, do not consider stat-only change as changed.
           Instead, silently run git update-index --refresh to update the cached stat information for paths
           whose contents in the work tree match the contents in the index. This option defaults to true.
           Note that this affects only git diff Porcelain, and not lower level diff commands such as git
           diff-files.

Oh, and this option appears to be ignored on Windows: changing it doesn't resolve this issue. I would guess that is because the performance improvement is not dubious: git diff on julia on linux takes < 0.1 seconds. On my windows virtual machine, it takes between 0.5 and 1 second. Or roughly 10x slower for file operations. While the penalty usually isn't quite as bad when I use a native Windows host, it is still very noticeable.