Can Not Pull on Windows 7/TortoiseHG 2

[JoseRaffucci]

Hello Everyone, first let me thank you for this great project having interoperability with git from mercurial is absolutely great.

Now to my issue, on windows(this does not happen under linux) I am able to clone a repository just fine. But when I want to pull again from the master repo I am greeted with this error bellow.

C:\Users\x\Dev\ProcessWire>hg pull pulling from git+ssh://git@github.com:jraffucci/ProcessWire.git importing Hg objects into Git ["git-upload-pack 'jraffucci/ProcessWire.git'"] * unknown exception encountered, please report by visiting * http://mercurial.selenic.com/wiki/BugTracker * Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] * Mercurial Distributed SCM (version 1.8.2) \ Extensions loaded: fixfrozenexts, hggit Traceback (most recent call last): File "hg", line 36, in File "mercurial\dispatch.pyo", line 16, in run File "mercurial\dispatch.pyo", line 36, in dispatch File "mercurial\dispatch.pyo", line 58, in _runcatch File "mercurial\dispatch.pyo", line 601, in _dispatch File "mercurial\dispatch.pyo", line 406, in runcommand File "mercurial\dispatch.pyo", line 655, in _runcommand File "mercurial\dispatch.pyo", line 609, in checkargs File "mercurial\dispatch.pyo", line 598, in File "mercurial\util.pyo", line 433, in check File "mercurial\commands.pyo", line 2940, in pull File "C:\Users\x\Dev\hg-git\hggit\hgrepo.py", line 14, in pull git.fetch(remote.path, heads) File "C:\Users\x\Dev\hg-git\hggit\git_handler.py", line 101, in fetch refs = self.fetch_pack(remote, heads) File "C:\Users\x\Dev\hg-git\hggit\git_handler.py", line 666, in fetch_pack f.write, self.ui.status) File "dulwich\client.pyo", line 232, in fetch_pack File "dulwich\client.pyo", line 295, in can_read File "dulwich\client.pyo", line 46, in _fileno_can_read select.error: (10038, 'An operation was attempted on something that is not a socket')

I am not very well versed with the internals of either project, but it seems to me that If I can clone the process in pulling should not be an issue but it still fails. Any and all help with this would be appreciated. Thank you very much for your help and a great project. Again this does not happen on linux, everything works just fine there.

[jhawk28]

This may be related to this dulwich issue: https://bugs.launchpad.net/dulwich/+bug/670035

[bobschi]

I'm also having the some problem. Here's my log.

D:\Blog>hg pull
pulling from git+ssh://git@github.com/bobschi/bobschi.github.com.git
importing Hg objects into Git
["git-upload-pack '/bobschi/bobschi.github.com.git'"]
** unknown exception encountered, please report by visiting
**  http://mercurial.selenic.com/wiki/BugTracker
** Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)]
** Mercurial Distributed SCM (version 1.7.2)
** Extensions loaded: fixfrozenexts, bookmarks, hggit
Traceback (most recent call last):
  File "hg", line 36, in 
  File "mercurial\dispatch.pyo", line 16, in run
  File "mercurial\dispatch.pyo", line 36, in dispatch
  File "mercurial\dispatch.pyo", line 58, in _runcatch
  File "mercurial\dispatch.pyo", line 590, in _dispatch
  File "mercurial\dispatch.pyo", line 401, in runcommand
  File "mercurial\dispatch.pyo", line 641, in _runcommand
  File "mercurial\dispatch.pyo", line 595, in checkargs
  File "mercurial\dispatch.pyo", line 588, in 
  File "mercurial\util.pyo", line 427, in check
  File "mercurial\extensions.pyo", line 130, in wrap
  File "mercurial\util.pyo", line 427, in check
  File "hgext\bookmarks.pyo", line 419, in pull
  File "mercurial\util.pyo", line 427, in check
  File "mercurial\commands.pyo", line 2802, in pull
  File "C:\Program Files (x86)\TortoiseHg\hg-git\hggit\hgrepo.py", line 14, in p
ull
    git.fetch(remote.path, heads)
  File "C:\Program Files (x86)\TortoiseHg\hg-git\hggit\git_handler.py", line 101
, in fetch
    refs = self.fetch_pack(remote, heads)
  File "C:\Program Files (x86)\TortoiseHg\hg-git\hggit\git_handler.py", line 666
, in fetch_pack
    f.write, self.ui.status)
  File "dulwich\client.pyo", line 231, in fetch_pack
  File "dulwich\client.pyo", line 303, in can_read
  File "dulwich\client.pyo", line 268, in can_read
select.error: (10038, 'Ein Vorgang bezog sich auf ein Objekt, das kein Socket is
t')

The repository I tried to pull from is publicly viewable under https://github.com/bobschi/bobschi.github.com -- hope this is at least a little helpful! : )

Flo

[derigel23]

The same for me:

pulling from git+ssh://git@github.com:derigel23/dotnetopenid.git
importing Hg objects into Git
["git-upload-pack 'derigel23/dotnetopenid.git'"]
** unknown exception encountered, please report by visiting
**  http://mercurial.selenic.com/wiki/BugTracker
** Python 2.6.6 (r266:84297, Aug 24 2010, 18:13:38) [MSC v.1500 64 bit (AMD64)]
** Mercurial Distributed SCM (version 1.8.3)
** Extensions loaded: fixfrozenexts, transplant, color, graphlog, progress, rebase, mq, hggit
Traceback (most recent call last):
  File "hg", line 36, in <module>
  File "mercurial\dispatch.pyo", line 16, in run
  File "mercurial\dispatch.pyo", line 36, in dispatch
  File "mercurial\dispatch.pyo", line 58, in _runcatch
  File "mercurial\dispatch.pyo", line 601, in _dispatch
  File "mercurial\dispatch.pyo", line 406, in runcommand
  File "mercurial\extensions.pyo", line 178, in wrap
  File "hgext\color.pyo", line 223, in colorcmd
  File "mercurial\dispatch.pyo", line 655, in _runcommand
  File "mercurial\dispatch.pyo", line 609, in checkargs
  File "mercurial\dispatch.pyo", line 598, in <lambda>
  File "mercurial\util.pyo", line 433, in check
  File "mercurial\extensions.pyo", line 133, in wrap
  File "mercurial\util.pyo", line 433, in check
  File "D:\hg\hgext\hgsubversion\hgsubversion\wrappers.py", line 495, in generic return orig(ui, repo, *args, **opts)
  File "mercurial\util.pyo", line 433, in check
  File "mercurial\extensions.pyo", line 133, in wrap
  File "mercurial\util.pyo", line 433, in check
  File "hgext\mq.pyo", line 3049, in mqcommand
  File "mercurial\util.pyo", line 433, in check
  File "mercurial\extensions.pyo", line 133, in wrap
  File "mercurial\util.pyo", line 433, in check
  File "hgext\rebase.pyo", line 546, in pullrebase
  File "mercurial\util.pyo", line 433, in check
  File "mercurial\commands.pyo", line 2940, in pull
  File "D:\hg\hg-git\hggit\hgrepo.py", line 14, in pull
        git.fetch(remote.path, heads)
  File "D:\hg\hg-git\hggit\git_handler.py", line 101, in fetch
    refs = self.fetch_pack(remote, heads)
  File "D:\hg\hg-git\hggit\git_handler.py", line 666, in fetch_pack
    f.write, self.ui.status)
  File "dulwich\client.pyo", line 232, in fetch_pack
  File "dulwich\client.pyo", line 295, in can_read
  File "dulwich\client.pyo", line 46, in _fileno_can_read
select.error: (10038, 'An operation was attempted on something that is not a socket')

[bobschi]

Upgrading to the newer tortoisehg-version with 2.x didn't solve this problem. For the time being, I'm using git ... ;)

[ghost]

I am also having this issue, same error, same circumstances

[marcusrbrown]

I'm using the following hack to get pull and incoming working on Windows:

diff --git a/hggit/_ssh.py b/hggit/_ssh.py
--- a/hggit/_ssh.py
+++ b/hggit/_ssh.py
@@ -12,7 +12,7 @@

     class _Vendor(SSHVendor):
         def connect_ssh(self, host, command, username=None, port=None):
-            from dulwich.client import SubprocessWrapper
+            from dulwich.client import SubprocessWrapper, _fileno_can_read
             from mercurial import util
             import subprocess

@@ -25,6 +25,19 @@
             proc = subprocess.Popen(util.quotecommand(cmd), shell=True,
                                     stdin=subprocess.PIPE,
                                     stdout=subprocess.PIPE)
-            return SubprocessWrapper(proc)
+            class HackSubprocessWrapper(SubprocessWrapper):
+                def __init__(self, proc):
+                    SubprocessWrapper.__init__(self, proc)
+
+                def can_read(self):
+                    if subprocess.mswindows:
+                        from msvcrt import get_osfhandle
+                        from win32pipe import PeekNamedPipe
+                        handle = get_osfhandle(self.proc.stdout.fileno())
+                        return PeekNamedPipe(handle, 0)[2] != 0
+                    else:
+                        return _fileno_can_read(self.proc.stdout.fileno())
+
+            return HackSubprocessWrapper(proc)

     return _Vendor

But the issue needs to be addressed in dulwich, not hg-git (looking at the bzr-git fix, bzrlib's SSH transport knows whether the fd is a socket or a pipe and adjusts its read function accordingly).

[acelent]

igetgames, do you mind if I submit a patch with that solution to can_read to dulwich?

[marcusrbrown]

Nope, I don't mind :)

[pierreh]

Thanks this patch works well for me

[piotr-dobrogost]

acelent, could you please inform us what's the status of the patch you submitted?

[acelent]

It's in dulwich since July 1st 2011: jelmer/dulwich@31e11499b7ea5f2fff1ade49b07ed926e7ffcec3. However, that only solves half of the problem if you're using TortoiseHg Workbench (or any of its sub-interfaces/sub-dialogs). If you don't, you may stop reading here.

Short story: thgw.exe doesn't have valid open files in stdin, stdout and stderr, and subprocess invocations that don't pipe the three will raise an error.

Long story:

Dulwich only explicitly pipes stdin and stdout when running git, i.e. it doesn't explicitly pipe stderr. In python under win32, if you don't explicitly pipe a std, it tries to duplicate the win32 handle of the current respective std variable value.

So, when running thg.exe, there's no problem because it always starts with some std handles, because it's a console application. Console applications that are run by "themselves" (e.g. from a shortcut or CreateProcess without a startup parameter) always get a console. When run inside a console, they share the parent process's console. And native win32 applications that pipe a child's process input and output usually invoke CreateProcess with the startup info flag STARTF_USESTDHANDLES set and, if well behaved, also set the three hStd* fields to inheritable pipe/file/console/etc. handles.

Now thgw.exe is a windowed application, and it never starts with std handles if started by "itself", like through a shortcut or even inside a console (unless someone explicitly invokes CreateProcess as previously stated).

Thgw.exe is created using py2exe, which sets stderr to something that, when accessed (and I mean accessed by python), it creates a file to store error messages. This is done to both have a way to log error messages, but only if they're generated to avoid empty log files. This only works if python code itself writes to stderr, not external code or subprocesses. So, thgw.exe never has valid stdin and stdout files, and it starts with an invalid stderr file until something writes to it. The stdin and stdout haven't been a problem because dulwich explicitly pipes them when running subprocesses, as previously stated, but stderr is something that most of the times doesn't have a (valid) win32 handle to duplicate.

The reason I was told why dulwich doesn't pipe stderr is that it shouldn't, so that the subprocess inherits the current stderr to allow the subprocess error output be directly shown. I think it's a good reason, although maybe dulwich could instead pipe stderr to capture it and then rewrite when the subprocess finished. Anyway, this option would only mask the bigger problem.

Note that not having valid std values means that any subprocess invocation may fail unless it pipes the three std, which means that the problem is not limited to dulwich.

I don't know if it's better to require that any subprocess invocation within thgw.exe be responsible to pipe std* or to make thgw.exe create pipes when started, or to change py2exe, or to ditch thgw.exe and use thg.exe with the small nuance of having a console appear and disappear right after, or something else. I only know that the former is not nice at all, because it means it doesn't tolerate any code or library that doesn't really want to pipe/capture subprocess input/output (i.e. it wants I/O to go through).

I poked around the sources of python to find out how it duplicates handles when starting subprocesses, py2exe to find out if it sets any std, TortoiseHg, Hg-Git and Mercurial for subprocess calls (IIRC these three always pipe the three std), and dulwich for subprocess calls too (it only pipes stdint and stdout). This was 4 months ago, so take it with a grain of salt, as something here described might not be the factual truth of then and/or now.

Meanwhile, I stopped using hg with hg-git for git repositories. I no longer have the personal time and motivation to contact authors or to mess with and test code.

[piotr-dobrogost]

Thank you very much for spending time to investigate the issue and taking time to so precisely describe the problem.

[acelent]

I've duplicated this information at tortoisehg's bitbucket issue 788.

PS: Using pythonw.exe to invoke thg also doesn't give us stdhandles, unless it's called with CreateProcess and stdhandles in the startup info argument. This is for the same reason as for thgw.exe: it's a windowing application instead of a console application. There are reported issues on the python project where this problem is discussed, some closed and others open, with almost philosophical arguments, but most are more concerned with python's own writing to stdout/stderr than with duplicating handles for subprocesses.