hg testing on Windows

[hayd]

The good news is that (I think) this is a testing bug, rather than a bug in the actual Hg implementation...

Tests fails atm, I had a play a short while ago.

On Windows there seems to be an inconsistency in how os.chdir works, it seems to work correctly working in the repl but not in the actual code. I tried wrapping functions with (e.g. I think that successfully_commit is not wrapped probably and changes directory), but even fixing that... no avail.

@contextmanager
def in_temp_dir():
    current_dir = os.getcwd()
    os.chdir(TEMP_DIR)
    yield
    os.chdir(current_dir)

This context manager just doesn't work in Windows for me (in the test script, it works as you'd expect in repl/debugging). Maddening.

Will try and dig up full stack trace error when/if I next use Windows.

[schodge]

No problems with @contextmanager for me run in a script on the command line for a simple script:

context_test.py:

import os
from contextlib import contextmanager
TEMP_DIR = 'c:/hodgework/'
@contextmanager
def in_temp_dir():
    current_dir = os.getcwd()
    os.chdir(TEMP_DIR)
    yield
    os.chdir(current_dir)
with in_temp_dir():
    print '@hayd loves Windows!'

c:\HodgeWork\python
λ python context_test.py
@hayd loves Windows!

[hayd]

... :suspect:

What if you print os.getcwd() or print check_ouput(['chdir']) ?

[schodge]

print os.getcwd() is fine:

import os
from contextlib import contextmanager
from subprocess import check_output
TEMP_DIR = 'c:/hodgework/'
@contextmanager
def in_temp_dir():
    current_dir = os.getcwd()
    print os.getcwd()
    os.chdir(TEMP_DIR)
    print os.getcwd()
    #print check_output(['chdir'])
    yield
    os.chdir(current_dir)
    print os.getcwd()
    #print check_output(['chdir'])

with in_temp_dir():
    print '@hayd loves Windows!'
    print os.getcwd()

c:\HodgeWork\python
λ python context_test.py
c:\HodgeWork\python
c:\hodgework
@hayd loves Windows!
c:\hodgework
c:\HodgeWork\python

[schodge]

Well, now isn't this annoying. With the modified program below for check_output() (trying to kill everything that may be a problem), I can't get past this (wonder if this is related http://bugs.python.org/issue15533 ) no matter what I try.

WindowsError: [Error 2] The system cannot find the file specified

self = <subprocess.Popen object at 0x0000000002B762B0>
args = ['cd', 'c:\\hodgework']
bufsize = 0
executable = None
stdin = None
stdout = -1
stderr = None
preexec_fn = None
close_fds = False
shell = False
cwd = None
env = None
universal_newlines = True
startupinfo = None
creationflags = 0

import os
from contextlib import contextmanager
from subprocess import check_output
TEMP_DIR = os.path.abspath('c:/hodgework/')
@contextmanager
def in_temp_dir():
    current_dir = os.getcwd()
    #print os.getcwd()
    os.chdir(TEMP_DIR)
    #print os.getcwd()
    print check_output(['cd', TEMP_DIR], universal_newlines=True)
    yield
    os.chdir(current_dir)
    print check_output(['chdir', current_dir], universal_newlines=True)
    #print os.getcwd()
    #print check_output(['chdir'])

with in_temp_dir():
    print '@hayd loves Windows!'
    #print os.getcwd()

[hayd]

Did you ever get hg up and running the tests? I can't recall if I had a small repoducable example for this. It could be something weird I'm doing... somewhere.

...Maybe I only saw this in the tests, basically the tests are wrapped manually (not using contextmanager, but change directory to TEMP do stuff then change back) and it seems to work fine in linux/mac but windows just doesn't seem to accept the change in directory - either when debugging or running the actual code (the check_output) the os.getcwd is still the original.

[schodge]

Hg installed, in path, still not working. Not sure what this last error is. I changed my entire python installation (still 2.7, but moved from WinPython to Anaconda), don't know if it's related. Work laptop is still on WinPython so I'll check that tomorrow.

λ nosetests test_pep8radius.py
...EEE.fatal: Not a valid object name random_junk_sha
..F....
======================================================================
ERROR: test_one_line (test_pep8radius.TestRadiusHg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\GitHub\pep8radius\tests\test_pep8radius.py", line 234, in test_one_line
    self.check(original, modified, expected, 'test_one_line')
  File "C:\GitHub\pep8radius\tests\test_pep8radius.py", line 186, in check
    r = Radius.new(vc=self.vc, options=options)
  File "C:\GitHub\pep8radius\pep8radius.py", line 238, in new
    return r(rev=rev, options=options)
  File "C:\GitHub\pep8radius\pep8radius.py", line 219, in __init__
    self.filenames_diff = self.get_filenames_diff()
  File "C:\GitHub\pep8radius\pep8radius.py", line 324, in get_filenames_diff
    diff_files = check_output(cmd, stderr=STDOUT).decode('utf-8')
  File "C:\Anaconda\lib\subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['hg', 'diff', '--stat', '-r', u'default']' returned non-zero exit status 255

======================================================================
ERROR: test_one_line_from_subdirectory (test_pep8radius.TestRadiusHg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\GitHub\pep8radius\tests\test_pep8radius.py", line 241, in test_one_line_from_subdirectory
    directory=SUBTEMP_DIR)
  File "C:\GitHub\pep8radius\tests\test_pep8radius.py", line 186, in check
    r = Radius.new(vc=self.vc, options=options)
  File "C:\GitHub\pep8radius\pep8radius.py", line 238, in new
    return r(rev=rev, options=options)
  File "C:\GitHub\pep8radius\pep8radius.py", line 219, in __init__
    self.filenames_diff = self.get_filenames_diff()
  File "C:\GitHub\pep8radius\pep8radius.py", line 324, in get_filenames_diff
    diff_files = check_output(cmd, stderr=STDOUT).decode('utf-8')
  File "C:\Anaconda\lib\subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['hg', 'diff', '--stat', '-r', u'default']' returned non-zero exit status 255

======================================================================
ERROR: test_with_docformatter (test_pep8radius.TestRadiusHg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\GitHub\pep8radius\tests\test_pep8radius.py", line 247, in test_with_docformatter
    self.check(original, modified, expected, 'test_with_docformatter')
  File "C:\GitHub\pep8radius\tests\test_pep8radius.py", line 186, in check
    r = Radius.new(vc=self.vc, options=options)
  File "C:\GitHub\pep8radius\pep8radius.py", line 238, in new
    return r(rev=rev, options=options)
  File "C:\GitHub\pep8radius\pep8radius.py", line 219, in __init__
    self.filenames_diff = self.get_filenames_diff()
  File "C:\GitHub\pep8radius\pep8radius.py", line 324, in get_filenames_diff
    diff_files = check_output(cmd, stderr=STDOUT).decode('utf-8')
  File "C:\Anaconda\lib\subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['hg', 'diff', '--stat', '-r', u'default']' returned non-zero exit status 255

======================================================================
FAIL: test_list_fixes (test_pep8radius.TestRadiusNoVCS)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\GitHub\pep8radius\tests\test_pep8radius.py", line 136, in test_list_fixes
    self.assertEqual(fixes, afixes)
AssertionError: "E101 - Reindent all lines.\nE112 - <... goes on for a really long time...>

----------------------------------------------------------------------
Ran 14 tests in 6.653s

FAILED (errors=3, failures=1)

[hayd]

(Is the test_line_fixes one could be a new line at the end of one of them? I think I fixed/changed this recently, are you on latest?)

I claim is these CalledProcessErrors are because the hg command is not being run in the TEMP_DIR (even though it should be because we've done a os.chdir(TEMP_DIR) !!!) Hence my continued sulking about the contextmanager...

[schodge]

Git insists our masters are in sync, though git and I have yet to come to any sort of mutual trust with each other.

Andy Hayden wrote:

(Is the |test_line_fixes| one could be a new line at the end of one of them? I think I fixed/changed this recently, are you on latest?)

I /claim/ is these CalledProcessErrors are because the hg command is not being run in the TEMP_DIR (even though it /should/ be because we've done a |os.chdir(TEMP_DIR)| !!!) Hence my continued sulking about the contextmanager...

— Reply to this email directly or view it on GitHub https://github.com/hayd/pep8radius/issues/57#issuecomment-45463090.

[schodge]

Hehe, I thought it might be this. Prepare to curse Windows again.

c:\github\pep8radius\tests (master)
λ nosetests --pdb-failures test_pep8radius.py
...SSS.fatal: Not a valid object name random_junk_sha
..> c:\winpython-64bit-2.7.6.2\python-2.7.6.amd64\lib\unittest\case.py(508)_baseAssertEqual()
-> raise self.failureException(msg)
(Pdb) <going to skip going up the stack a bit>
(Pdb) u
> c:\github\pep8radius\tests\test_pep8radius.py(136)test_list_fixes()
-> self.assertEqual(fixes, afixes)
(Pdb) l
131
132         def test_list_fixes(self):
133             fixes = pep8radius_main(['--list-fixes'])
134             afixes = check_output(['autopep8', '--list-fixes'])\
135                        .decode("utf-8").strip()
136  ->         self.assertEqual(fixes, afixes)
137
138         def test_bad_rev(self):
139             self.assertRaises(CalledProcessError,
140                               lambda x: Radius.new(rev=x),
141                               'random_junk_sha')
(Pdb) p fixes
"E101 - Reindent all lines.\nE112 <continues>
(Pdb) p afixes
u"E101 - Reindent all lines.\r\nE112 <continues>
(see the problem?)
(Pdb) p len(fixes)
2438
(Pdb) p len(afixes)
2493
(Pdb) temp = afixes.replace("\r", "")
(Pdb)
(Pdb) len(temp)
2438
(Pdb) p temp == fixes
True

[schodge]

Pull request sent to fix the universal newline issue. While all the git pulls, statusii, etc. showed our repositories synced there still seemed to be a mess of empty changes that got sucked along with my pull request.

Also, while trying to run pep8radius...:

c:\github\pep8radius (test_windows_newlines)
λ python pep8radius.py c:\github\pep8radius\tests\test_pep8radius.py
fatal: Not a valid object name c:\github\pep8radius\tests\test_pep8radius.py
Traceback (most recent call last):
  File "pep8radius.py", line 555, in <module>
    main()
  File "pep8radius.py", line 92, in main
    output = c.output.splitlines()[0]
IndexError: list index out of range

? (Though I mostly confirmed PEP8 compatibility in Sublime).

[hayd]

Can you post a gist of the source code which caused that error?

[schodge]

I simply ran the pep8radius on its own test file. I'd bet it's related to the error you can see above ...SSS.fatal: Not a valid object name random_junk_sha , both not a valid object errors...?

[hayd]

I have a fix for the fatal things https://github.com/hayd/pep8radius/commit/56cc324de8662dbdf521c55b55c106931a6bd302

but the IndexError: list index out of range thing is not good, would like to see gist of file that's giving this error! It looks like a proper error

[schodge]

Hmmm... something's really weird - I run python -m pdb pep8radius.py ./tests/test_pep8radius.py, step past the initial point, and it just loops this output forever. I think I grabbed an entire loops worth, but as it's continually dumping it into the terminal I'm not sure. It looks like the problem may be with readline, but I guess pdb is restarting the script after debugging and recrashing without my ever getting to do anything?

test_pep8radius.py used: https://gist.github.com/schodge/d51be310ba9a07f8e7c0

fatal: Not a valid object name .\tests\test_pep8radius.py
Traceback (most recent call last):
  File "C:\Anaconda\lib\pdb.py", line 1314, in main
    pdb._runscript(mainpyfile)
  File "C:\Anaconda\lib\pdb.py", line 1233, in _runscript
    self.run(statement)
  File "C:\Anaconda\lib\bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "pep8radius.py", line 1, in <module>
    from __future__ import print_function
  File "pep8radius.py", line 92, in main
    output = c.output.splitlines()[0]
IndexError: list index out of range
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> c:\github\pep8radius\pep8radius.py(92)main()
-> output = c.output.splitlines()[0]
Readline internal error
Traceback (most recent call last):
  File "C:\Anaconda\lib\site-packages\pyreadline\console\console.py", line 768, in hook_wrapper_23
    res = ensure_str(readline_hook(prompt))
  File "C:\Anaconda\lib\site-packages\pyreadline\rlmain.py", line 569, in readline
    self.readline_setup(prompt)
  File "C:\Anaconda\lib\site-packages\pyreadline\rlmain.py", line 565, in readline_setup
    self._print_prompt()
  File "C:\Anaconda\lib\site-packages\pyreadline\rlmain.py", line 466, in _print_prompt
    x, y = c.pos()
  File "C:\Anaconda\lib\site-packages\pyreadline\console\console.py", line 261, in pos
    self.GetConsoleScreenBufferInfo(self.hout, byref(info))
ArgumentError: argument 2: <type 'exceptions.TypeError'>: expected LP_CONSOLE_SCREEN_BUFFER_INFO instance instead of pointer to CONSOLE_SCREEN_BUFFER_INFO
Post mortem debugger finished. The pep8radius.py will be restarted
> c:\github\pep8radius\pep8radius.py(1)<module>()
-> from __future__ import print_function
Readline internal error
Traceback (most recent call last):
  File "C:\Anaconda\lib\site-packages\pyreadline\console\console.py", line 768, in hook_wrapper_23
    res = ensure_str(readline_hook(prompt))
  File "C:\Anaconda\lib\site-packages\pyreadline\rlmain.py", line 569, in readline
    self.readline_setup(prompt)
  File "C:\Anaconda\lib\site-packages\pyreadline\rlmain.py", line 565, in readline_setup
    self._print_prompt()
  File "C:\Anaconda\lib\site-packages\pyreadline\rlmain.py", line 466, in _print_prompt
    x, y = c.pos()
  File "C:\Anaconda\lib\site-packages\pyreadline\console\console.py", line 261, in pos
    self.GetConsoleScreenBufferInfo(self.hout, byref(info))
ArgumentError: argument 2: <type 'exceptions.TypeError'>: expected LP_CONSOLE_SCREEN_BUFFER_INFO instance instead of pointer to CONSOLE_SCREEN_BUFFER_INFO

[schodge]

See the latest in PR #59 . Also, any idea how to handle the

fatal: Not a valid object name random_junk_sha

I'm not sure how to get that to drop into pdb - for some reason I can't find documentation on fatal errors in unittest.

[hayd]

So, that is fixed in one of my PRs (I think the bazaar one), need to merge that (assuming it's not too broken)... the way to fix was to use the context manager for trapping out & err (that is an intentional error in the testing suite if the branch name is bogus).

Bazaar is broken IIRC (mainly as I don't understand how it works for branches/merge-base against a commit) so will have to label it "experimental" somehow.

Sorry about my foot dragging!

[schodge]

In general, how would you debug a fatal? I couldn't find a way to drop into the debugger when it occurred.

Is any project of note using Bazaar? I'd think git + HG would cover 95% of the cases.

[hayd]

It's a fatal in hg (when it shells out to change branch to a non-existing branch), not a fatal in python.

Maybe not, though it's popular for linux gui libraries and such, I thought it wouldn't be too much bother to include.. lol.

[hayd]

This was due to bad config and me not catching it nicely!!