Closed JeffBezanson closed 11 years ago
I wonder if these errors aren't red herrings: it looks like Git.dirty()
is implemented as:
dirty(; dir="") = !success(`diff --quiet HEAD`, dir=dir)
Which could return true
if there was some problem in invoking git diff
, e.g. git
paths being wrong, the git
executable not even being found, etc....
The executable not being found should at least throw an error but it's true that other issues might return a false dirty status here. I don't know a better way to test this, unfortunately. Any ideas?
It doesn't look like git paths being set incorrectly can cause this error, unfortunately. (Or perhaps fortunately, depending on whether or not you want things to be broken)
I don't know of any better way to test.
Probably same as #4328
Shouldn't Pkg.build
wrap the build script in Git.transact
or similar so that failed build attempts don't leave the ZMQ directory in a dirty state? But I'm not sure why BinDeps should ever make ZMQ dirty.
I've been seeing this as well: https://groups.google.com/forum/#!topic/julia-users/MGl42Hc2qWc
I've tried uninstalling my msysgit and using the version provided by msys2, reinstalling msysgit, changing all the core.autocrlf
settings, etc. I'd dig into it more, but I'm really not sure how the Base.Git, Base.Pkg are implemented.
p.s. This happens on any and all packages I try it with (not just ones with build dependencies).
@karbarcca can I get a versioninfo()
from you?
Also, let's try running the git commands manually as Base.Git
tries to:
julia> Pkg.dir("BinDeps")
"/home/sabae/.julia/BinDeps"
julia> gitcmd = Base.Git.git(Pkg.dir("BinDeps"))
`git --work-tree=/home/sabae/.julia/BinDeps --git-dir=/home/sabae/.julia/BinDeps/.git`
julia> run(`$gitcmd status`)
# Not currently on any branch.
nothing to commit, working directory clean
julia> success(`$gitcmd diff --quiet HEAD`)
true
If you can show these we should be able to rule out git
<-->julia
interaction as the problem.
Sorry for the delay, I've been having some julia build problems. Here's the output of the above commands:
julia> versioninfo()
Julia Version 0.2.0-prerelease+3836
Commit b5c797c 2013-09-24 17:42:42 UTC
Platform Info:
System: Windows (x86_64-w64-mingw32)
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
LAPACK: libopenblas
LIBM: libopenlibm
julia> Pkg.dir("BinDeps")
"C:\\Users\\karbarcca\\AppData\\Roaming\\Julia\\packages\\BinDeps"
julia> gitcmd = Base.Git.git(Pkg.dir("BinDeps"))
`git '--work-tree=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps' '--git-dir=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps\.git'`
julia> run(`$gitcmd status`)
fatal: index file open failed: Invalid argument
ERROR: failed process: Process(`git '--work-tree=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps' '--git-dir=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps\.git' status`, ProcessExited(128)) [128]
in error at error.jl:22
in pipeline_error at process.jl:447
in success at process.jl:439
in spawn_opts_inherit at process.jl:370
in run at process.jl:424
julia> success(`$gitcmd diff --quiet HEAD`)
false
p.s. The fatal: index file open failed
error happens each and every time I try to add a package.
Possibly related to this reported git bug (if git is sufficiently new)? http://www.spinics.net/lists/git/msg217305.html
(That's to @karbarcca's fatal: index file open failed
--I'd need to read more closely to see if that could be causing the problem @JeffBezanson reported.)
@pao, thanks for the tip! Indeed, switching back to git-1.8.3 fixes the fatal: index file open failed
error. Unfortunately, still seeing the package is dirty message when trying to checkout packages' master.
Do you have a git configuration file? e.g. anything in .gitconfig
in your home directory?
Or, what does git say:
julia> ;git config -l
core.autocrlf=true
core.filemode=false
core.ignorecase=true
core.safecrlf core.eol core.autocrlf core.ignoreStat
Mine says
julia> ; git config -l
core.symlinks=false
core.autocrlf=true
color.diff=auto
color.status=auto
color.branch=auto
color.interactive=true
pack.packsizelimit=2g
help.format=html
http.sslcainfo=/bin/curl-ca-bundle.crt
sendemail.smtpserver=/bin/msmtp.exe
diff.astextplain.textconv=astextplain
rebase.autosquash=true
user.email=quinn.jacobd@gmail.com
user.name=Jacob Quinn
What about git --version
? Is it possible you have a cygwin git in your path? I notice that the msys2 git appears to be badly broken when called from julia.
That was one of my original problems, but after changing my msys2 git to _git.exe
, I finally got past some of the other errors I was seeing, but still see the package dirty bailing error when trying to checkout.
julia> ;git --version
git version 1.8.3.msysgit.0
Very strange, I figured that was fine, but just don't see where your configuration is different. Can to post the output of the git commands from earlier?
julia> Pkg.dir("BinDeps")
"C:\\Users\\karbarcca\\AppData\\Roaming\\Julia\\packages\\BinDeps"
julia> gitcmd = Base.Git.git(Pkg.dir("BinDeps"))
`git '--work-tree=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps' '--git-dir=C:\Users\karbarcca\AppData\Roaming\Julia\packages\BinDeps\.git'`
julia> run(`$gitcmd status`)
# HEAD detached at 00bdc84
nothing to commit, working directory clean
julia> success(`$gitcmd diff --quiet HEAD`)
true
julia> Pkg.checkout("Datetime")
INFO: Checking out Datetime master...
ERROR: Datetime is dirty, bailing
in anonymous at no file:120
in transact at git.jl:69
julia> Pkg.checkout("BinDeps")
INFO: Checking out BinDeps master...
ERROR: BinDeps is dirty, bailing
in anonymous at no file:120
in transact at git.jl:69 (repeats 2 times)
julia>
interesting:
julia> g=Base.Git.git("ZMQ")
`git '--work-tree=C:\Users\julia\AppData\Roaming\Julia\packages\ZMQ' '--git-dir=C:\Users\julia\AppData\Roaming\Julia\packages\ZMQ\.git'`
julia> Base.Git.dirty(dir="ZMQ")
true
julia> success(`$g diff --exit-code HEAD`)
false
julia> success(`$g diff -w --exit-code HEAD`)
true
julia> success(`$g diff --ignore-space-at-eol --exit-code HEAD`)
true
julia> success(`$g diff -a --exit-code HEAD`)
false
Woah, man. That's bananas. What's going on here?
Could different values for core.autocrlf
have something to do with it?
Hopefully we can keep thinking about this and testing during the RC period, and if we don't solve it in a week or two release 0.2 without a fix.
git status
fixes / does not have this issue
$ git status || echo 1
# On branch master
nothing to commit, working directory clean
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ touch REQUIRE
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ git diff --quiet || echo 1
1
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ git status
# On branch master
nothing to commit, working directory clean
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ git diff --quiet || echo 1
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ touch REQUIRE
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$ git diff --quiet || echo 1
1
julia@JULIA-PC ~/AppData/Roaming/julia/packages/ZMQ (master)
$
@StefanKarpinski Could we replace success(
git diff --quiet)
with isempty(readall(
git status --porcelain))
Alternatively, git update-index --really-refresh
also "fixes" the call to git diff
This only happens on Windows, right? Should this be considered an upstream bug? How does git update-index --really-refresh
help?
Should this be considered an upstream bug?
Maybe, but I think @vtjnash's earlier suggestion of git status --porcelain
makes good sense--it's in a guaranteed stable format, doesn't rely on git's seemingly random choices for exit codes, and if there are changes to the working copy it should be faster than computing the diff (if only slightly).
I assume this is an issue with the file system on Windows being absurdly slow. So cheap operations on linux (like lstat and listing the files in a directory) are instead being cached on Windows, to make is usable.
@vtjnash, can you please elaborate on how git update-index --really-refresh
helps? Does calling that right before calling git diff --quiet
make this work?
Yes
Let me know if that fixes the problem – hopefully it does.
Can someone please file an issue for this upstream with the git folks?
Starting from no .julia
directory.
julia> Pkg.init()
INFO: Initializing package repository C:\Users\karbarcca\.julia
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
julia> Pkg.add("Datetime")
INFO: Cloning cache of Datetime from git://github.com/karbarcca/Datetime.jl.git
fatal: This operation must be run in a work tree
ERROR: failed process: Process(`git '--git-dir=C:\Users\karbarcca\.julia\.cache\Datetime' update-index --really-refresh`, ProcessExited(128)) [128]
in pipeline_error at process.jl:449
in run at process.jl:426
WARNING: backtraces on your platform are often misleading or partially incorrect
julia> Pkg.clone("Datetime")
INFO: Cloning Datetime from git://github.com/karbarcca/Datetime.jl.git
Cloning into 'Datetime'...
julia> Pkg.checkout("Datetime")
INFO: Checking out Datetime master...
INFO: Pulling Datetime latest master...
INFO: No packages to install, update or remove.
julia> Pkg.add("Datetime")
INFO: No packages to install, update or remove.
INFO: REQUIRE updated.
julia> Pkg.add("DataFrames")
INFO: Cloning cache of Blocks from git://github.com/tanmaykm/Blocks.jl.git
fatal: This operation must be run in a work tree
ERROR: failed process: Process(`git '--git-dir=C:\Users\karbarcca\.julia\.cache\Blocks' update-index --really-refresh`, ProcessExited(128)) [128]
in pipeline_error at process.jl:449
in success at process.jl:441
in run at process.jl:426
julia>
checkout
seems to work, but add
is now wonky.
Sigh. But it's a bit closer, I guess.
Ah, ok, I think I know what the deal with this is.
Hopefully this is now fully fixed...
Again starting from a new .julia
julia> Pkg.init()
INFO: Initializing package repository C:\Users\karbarcca\.julia
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
julia> Pkg.add("Datetime")
INFO: Cloning cache of Datetime from git://github.com/karbarcca/Datetime.jl.git
INFO: Installing Datetime v0.1.2
INFO: REQUIRE updated.
julia> Pkg.checkout("Datetime")
INFO: Checking out Datetime master...
ERROR: Datetime is dirty, bailing
in anonymous at no file:133
in transact at git.jl:81
WARNING: backtraces on your platform are often misleading or partially incorrect
julia> Pkg.clone("DateFrames")
ERROR: can't determine package name from URL: DateFrames
in clone at pkg/entry.jl:125
julia> Pkg.clone("DataFrames")
INFO: Cloning DataFrames from git://github.com/JuliaStats/DataFrames.jl.git
Cloning into 'DataFrames'...
INFO: Computing changes...
INFO: Cloning cache of Blocks from git://github.com/tanmaykm/Blocks.jl.git
INFO: Cloning cache of GZip from git://github.com/kmsquire/GZip.jl.git
INFO: Cloning cache of SortingAlgorithms from git://github.com/JuliaLang/SortingAlgorithms.jl.git
INFO: Cloning cache of Stats from git://github.com/JuliaStats/Stats.jl.git
INFO: Installing Blocks v0.0.0
INFO: Installing GZip v0.2.5
INFO: Installing SortingAlgorithms v0.0.0
INFO: Installing Stats v0.2.8
julia> Pkg.checkout("DataFrames")
INFO: Checking out DataFrames master...
ERROR: DataFrames is dirty, bailing
in anonymous at no file:133
in transact at git.jl:81
in write at stream.jl:734
julia>
Plz try one more time...
@karbarcca just out of curiosity, what's your versioninfo()
?
Winner winner!
julia> Pkg.init()
INFO: Initializing package repository C:\Users\karbarcca\.julia
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
julia> Pkg.add("Datetime")
INFO: Cloning cache of Datetime from git://github.com/karbarcca/Datetime.jl.git
INFO: Installing Datetime v0.1.2
INFO: REQUIRE updated.
julia> Pkg.checkout("Datetime")
INFO: Checking out Datetime master...
INFO: Pulling Datetime latest master...
INFO: No packages to install, update or remove.
julia> Pkg.clone("DataFrames")
INFO: Cloning DataFrames from git://github.com/JuliaStats/DataFrames.jl.git
Cloning into 'DataFrames'...
INFO: Computing changes...
INFO: Cloning cache of Blocks from git://github.com/tanmaykm/Blocks.jl.git
INFO: Cloning cache of GZip from git://github.com/kmsquire/GZip.jl.git
INFO: Cloning cache of SortingAlgorithms from git://github.com/JuliaLang/SortingAlgorithms.jl.git
INFO: Cloning cache of Stats from git://github.com/JuliaStats/Stats.jl.git
INFO: Installing Blocks v0.0.0
INFO: Installing GZip v0.2.5
INFO: Installing SortingAlgorithms v0.0.0
INFO: Installing Stats v0.2.8
julia> versioninfo()
Julia Version 0.2.0-rc1+141
Commit 8d9fdf4 2013-10-24 20:18:27 UTC
Platform Info:
System: Windows (x86_64-w64-mingw32)
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
LAPACK: libopenblas
LIBM: libopenlibm
julia>
Phew. What a relief!
Thanks for the work everyone. This is definitely great to have for the rc.
@StefanKarpinski this problem exists on Linux too (it's a configuration option) -- the defaults are just different:
$ git config diff.autorefreshindex false
$ git diff --quiet HEAD || echo 1
$ touch README.md
$ git diff --quiet HEAD || echo 1
1
$ git status > /dev/null
$ git diff --quiet HEAD || echo 1
$
Well then can we just set the option? Please test and tell me if that works. Why is there a default option to be broken?
well, the user could change the option: this option makes git much faster on windows, or on large repositories.
I don't really care. If this breaks things we will change the option every time. I can't respect options that break everything in random ways.
Or we could apply the same fix on linux?
If this fix is the fault of an idiotic config option, we should revert that fix and just force the option to something that isn't broken. It's not like this option changes how anything reasonable works, it is literally just an option to sacrifice correct behavior for dubious performance improvement.
Seriously. The existence of the option is why we can't have nice things.
The config documentation for that option seems to suggests that you should be calling git update-index --refresh
anytime you are using git diff
from a script. That option is apparently just there to make life a bit easier on platforms that have fast file I/O.
diff.autorefreshindex
When using git diff to compare with work tree files, do not consider stat-only change as changed.
Instead, silently run git update-index --refresh to update the cached stat information for paths
whose contents in the work tree match the contents in the index. This option defaults to true.
Note that this affects only git diff Porcelain, and not lower level diff commands such as git
diff-files.
Oh, and this option appears to be ignored on Windows: changing it doesn't resolve this issue. I would guess that is because the performance improvement is not dubious: git diff
on julia on linux takes < 0.1 seconds. On my windows virtual machine, it takes between 0.5 and 1 second. Or roughly 10x slower for file operations. While the penalty usually isn't quite as bad when I use a native Windows host, it is still very noticeable.
I tried to switch to latest master of ZMQ on windows using
Pkg.checkout("ZMQ",pull=true)
. I got "ERROR: ZMQ is dirty, bailing", but git status shows that the working tree is clean, and on a normal released version. Manually going to the directory and doinggit checkout master
followed bygit pull
worked.