search

collective / mr.bob

Mister Bob (the builder) is filesystem template renderer
http://mrbob.readthedocs.org/en/latest/
Other
69 stars 30 forks source link

Support for Windows #49

Open Annakan opened 11 years ago

Annakan commented 11 years ago

On windows at least, trying to activate mrbob from an url fail with a I/O error 13, permission denied from urlretrieve in configurator.py / parse_template.

The reason is that urlretrieve expect to open in write mode the file name passed in and fail since tempfile.NamedTemporaryFile already did that.

it seems that (temp_file_name, heasers)= urlretrieve(url) is enough to get a tempfile with the downloaded content in it. (from http://docs.python.org/2/library/urllib.html)

Something like this seems to do the job

(temp_file_name, _ )= urlretrieve(url)
if not is_zipfile(temp_file_name):
    raise ConfigurationError("Not a zip file: %s" % temp_file_name)
zf = ZipFile(temp_file_name)
try:
   path = tempfile.mkdtemp()
   zf.extractall(path)
   return os.path.join(path, subpath), True
finally:
    zf.close()
domenkozar commented 11 years ago

Thanks for reporting the issue! Could you try applying that patch and running tests? In documentation there should be instructions how to do that

Annakan commented 11 years ago

You are welcome No time for that tonight, I'll try to look into it tomorrow.

On 2013-06-10 17:44, Domen Kožar wrote:

Thanks for reporting the issue! Could you try applying that patch and running tests? In documentation there should be instructions how to do that

Reply to this email directly or view it on GitHub [1].

Links:

[1] https://github.com/iElectric/mr.bob/issues/49#issuecomment-19207251

Annakan commented 11 years ago

I have a hard time running the tests,

I created a venv with Jinja2==2.6 distribute==0.6.27 mock==1.0.1 nose==1.3.0 pyreadline==2.0 six==1.3.0

Going into mr.bob/test and doing "nosetest" brings heaps of violations and failures, some messages seems to hint toward running nosetest with specific parameters but I did not guess witch ones yet.

I am doing my test on Windows so it might be the problem, I had no time to test under one of my FreeBSD systems yet.

Is there a how-to do tests or should it work "right away" ?

domenkozar commented 11 years ago

http://mrbob.readthedocs.org/en/latest/developer.html#running-tests

On Tue, Jun 11, 2013 at 10:26 PM, Annakan notifications@github.com wrote:

I have a hard time running the tests,

I created a venv with Jinja2==2.6 distribute==0.6.27 mock==1.0.1 nose==1.3.0 pyreadline==2.0 six==1.3.0

Going into mr.bob/test and doing "nosetest" brings heaps of violations and failures, some messages seems to hint toward running nosetest with specific parameters but I did not guess witch ones yet.

I am doing my test on Windows so it might be the problem, I had no time to test under one of my FreeBSD systems yet.

Is there a how-to do tests or should it work "right away" ?

— Reply to this email directly or view it on GitHubhttps://github.com/iElectric/mr.bob/issues/49#issuecomment-19289850 .

Annakan commented 11 years ago

ok shame on me for not seing that but the missing parts where mostly the sphinx and pep compliance part if I am not mistaken and the error number is nearly the same (probably the same in fact)

to be completely sure I redid the whole install from scratch

on windows I add to do (mrbob)> pip install pyreadline in the venv (I named it mrbob because virtualenv . is not working on windows with py2.7 and a third party virtualenv tool for cmd.exe, the base 'windows shell')

without it mrbob --help would fail because readline does not exist on base windows.

Many of the errors seems to be temp file logic related and probably of the same class of error I tried to fix. Others are more puzzling but I need to dig into the code much more to understand what is really happening. (like nosetests-script.py: error: config file does not exist: /notexists nosetests-script.py: error: You must specify what template to use.

here is the complete test result FYI


(mrbob) Z:\DocRoot\ProjetsX\mr.bob>make test
"==== Running nosetests ===="
.EEEEEusage: nosetests-script.py [-h] [-O TARGET_DIRECTORY] [-v] [-c CONFIG] [-V]
                           [-l] [-w] [-n] [-q]
                           [template]
nosetests-script.py: error: config file does not exist: /notexists
.usage: nosetests-script.py [-h] [-O TARGET_DIRECTORY] [-v] [-c CONFIG] [-V]
                           [-l] [-w] [-n] [-q]
                           [template]
nosetests-script.py: error: ConfigurationError: Template directory does not exist: Z:\DocRoot\ProjetsX\mr.bob\foo
.usage: nosetests-script.py [-h] [-O TARGET_DIRECTORY] [-v] [-c CONFIG] [-V]
                           [-l] [-w] [-n] [-q]
                           [template]
nosetests-script.py: error: You must specify what template to use.
..usage: nosetests-script.py [-h] [-O TARGET_DIRECTORY] [-v] [-c CONFIG] [-V]
                           [-l] [-w] [-n] [-q]
                           [template]
nosetests-script.py: error: ConfigurationError: Template directory does not exist: Z:\DocRoot\ProjetsX\mr.bob\foobar.zip
..............F....................E...EEE...................E.....EEEFE......E..FFF.FEFFFFEE........FFE
======================================================================
ERROR: test_configs_override_each_other (mrbob.tests.test_cli.TestCLI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 107, in test_configs_override_each_other
    template_dir)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 19, in call_FUT
    return main(args)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\cli.py", line 162, in main
    defaults=defaults)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 138, in __init__
    self.template_dir, self.is_tempdir = parse_template(template)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "C:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named Z
-------------------- >> begin captured stdout << ---------------------
Configuration provided:
[variables] from ~/.mrbob
only_global = glob
overriden_by_file = file
[variables] from --config file
only_file = file
overriden_by_file = file1
[defaults] from ~/.mrbob
only_global_2 = glob2
overriden_by_file_2 = file1
[defaults] from --config file
only_file_2 = file1
overriden_by_file_2 = file2
[mr.bob] from ~/.mrbob
only_global = glob
overriden_by_file = foo
[mr.bob] from --config file
only_file = file
overriden_by_file = file1
[mr.bob] from command line interface
non_interactive = True
quiet = False
remember_answers = False
verbose = True

--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: test_dummy_template (mrbob.tests.test_cli.TestCLI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 34, in test_dummy_template
    self.call_FUT('-O', self.output_dir, template_dir)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 19, in call_FUT
    return main(args)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\cli.py", line 162, in main
    defaults=defaults)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 138, in __init__
    self.template_dir, self.is_tempdir = parse_template(template)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "C:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named Z

======================================================================
ERROR: test_dummy_template_create_target_directory (mrbob.tests.test_cli.TestCLI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 46, in test_dummy_template_create_target_directory
    self.call_FUT('-O', os.path.join(self.output_dir, 'notexist'), template_dir)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 19, in call_FUT
    return main(args)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\cli.py", line 162, in main
    defaults=defaults)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 138, in __init__
    self.template_dir, self.is_tempdir = parse_template(template)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "C:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named Z

======================================================================
ERROR: test_list_questions (mrbob.tests.test_cli.TestCLI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 51, in test_list_questions
    self.call_FUT('--list-questions', template_dir)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 19, in call_FUT
    return main(args)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\cli.py", line 162, in main
    defaults=defaults)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 138, in __init__
    self.template_dir, self.is_tempdir = parse_template(template)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "C:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named Z

======================================================================
ERROR: test_missing_mrbobini_in_template (mrbob.tests.test_cli.TestCLI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 55, in test_missing_mrbobini_in_template
    self.assertRaises(SystemExit, self.call_FUT, '-O', self.output_dir, template_dir)
  File "C:\Python27\Lib\unittest\case.py", line 475, in assertRaises
    callableObj(*args, **kwargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_cli.py", line 19, in call_FUT
    return main(args)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\cli.py", line 162, in main
    defaults=defaults)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 138, in __init__
    self.template_dir, self.is_tempdir = parse_template(template)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "C:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named Z

======================================================================
ERROR: test_absolute (mrbob.tests.test_configurator.parse_templateTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 108, in test_absolute
    abs_path = self.call_FUT(template_dir)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 96, in call_FUT
    return parse_template(name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "C:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named Z

======================================================================
ERROR: test_zipfile (mrbob.tests.test_configurator.parse_templateTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 123, in test_zipfile
    abs_path = self.call_FUT('http://foobar.com/bla.zip')
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 96, in call_FUT
    return parse_template(name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 80, in parse_template
    urlretrieve(url, tmpfile.name)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 955, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1018, in _mock_call
    ret_val = effect(*args, **kwargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 146, in fake_zip
    zf = zipfile.ZipFile(path, 'w')
  File "C:\Python27\Lib\zipfile.py", line 752, in __init__
    self.fp = open(file, modeDict[mode])
IOError: [Errno 13] Permission denied: 'c:\\users\\gryom\\appdata\\local\\temp\\tmpq16azv'

======================================================================
ERROR: test_zipfile_base_path (mrbob.tests.test_configurator.parse_templateTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 130, in test_zipfile_base_path
    abs_path = self.call_FUT('http://foobar.com/bla.zip#some/dir')
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 96, in call_FUT
    return parse_template(name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 80, in parse_template
    urlretrieve(url, tmpfile.name)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 955, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1018, in _mock_call
    ret_val = effect(*args, **kwargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 155, in fake_zip_base_path
    zf = zipfile.ZipFile(path, 'w')
  File "C:\Python27\Lib\zipfile.py", line 752, in __init__
    self.fp = open(file, modeDict[mode])
IOError: [Errno 13] Permission denied: 'c:\\users\\gryom\\appdata\\local\\temp\\tmplzn2nb'

======================================================================
ERROR: test_zipfile_not_zipfile (mrbob.tests.test_configurator.parse_templateTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 138, in test_zipfile_not_zipfile
    self.assertRaises(ConfigurationError, self.call_FUT, 'http://foobar.com/bla.tar#some/dir')
  File "C:\Python27\Lib\unittest\case.py", line 475, in assertRaises
    callableObj(*args, **kwargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 96, in call_FUT
    return parse_template(name)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\configurator.py", line 80, in parse_template
    urlretrieve(url, tmpfile.name)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 955, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1018, in _mock_call
    ret_val = effect(*args, **kwargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 141, in fake_wrong_zip
    with open(path, 'w') as f:
IOError: [Errno 13] Permission denied: 'c:\\users\\gryom\\appdata\\local\\temp\\tmp4azfab'

======================================================================
ERROR: test_parse_remote_config (mrbob.tests.test_parsing.parse_configTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_parsing.py", line 112, in test_parse_remote_config
    c = self.call_FUT(configname='http://nohost/mrbob.ini')
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_parsing.py", line 29, in call_FUT
    return parse_config(f)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\parsing.py", line 46, in parse_config
    urlretrieve(configname, tmpfile.name)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 955, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\mock-1.0.1-py2.7.egg\mock.py", line 1018, in _mock_call
    ret_val = effect(*args, **kwargs)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_parsing.py", line 103, in write
    f = open(filename, 'w')
IOError: [Errno 13] Permission denied: 'c:\\users\\gryom\\appdata\\local\\temp\\tmpr7mula'

======================================================================
ERROR: test_empty (mrbob.tests.test_parsing.write_configTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_parsing.py", line 175, in tearDown
    os.remove(self.tmpfile)
WindowsError: [Error 32] Le processus ne peut pas accÚder au fichier car ce fichier est utilisÚ par un autre processus: 'c:\\users\\gryom\\appdata\\local\\temp\\tmpj4avcd'

======================================================================
ERROR: test_multiple_options (mrbob.tests.test_parsing.write_configTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_parsing.py", line 175, in tearDown
    os.remove(self.tmpfile)
WindowsError: [Error 32] Le processus ne peut pas accÚder au fichier car ce fichier est utilisÚ par un autre processus: 'c:\\users\\gryom\\appdata\\local\\temp\\tmp1kei6a'

======================================================================
ERROR: test_non_str (mrbob.tests.test_parsing.write_configTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_parsing.py", line 175, in tearDown
    os.remove(self.tmpfile)
WindowsError: [Error 32] Le processus ne peut pas accÚder au fichier car ce fichier est utilisÚ par un autre processus: 'c:\\users\\gryom\\appdata\\local\\temp\\tmps3j8hf'

======================================================================
ERROR: test_utf8 (mrbob.tests.test_parsing.write_configTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_parsing.py", line 175, in tearDown
    os.remove(self.tmpfile)
WindowsError: [Error 32] Le processus ne peut pas accÚder au fichier car ce fichier est utilisÚ par un autre processus: 'c:\\users\\gryom\\appdata\\local\\temp\\tmpq4btyp'

======================================================================
ERROR: test_pluses_in_path (mrbob.tests.test_rendering.render_filenameTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 348, in test_pluses_in_path
    dict(bar='em0'))
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 321, in call_FUT
    return render_filename(filename, variables)
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\rendering.py", line 123, in render_filename
    raise KeyError('%s key part of filename %s was not found in variables %s' % (actual_replaceable, filename, variables))
KeyError: "/bla/ key part of filename +/bla/+/+bar+ was not found in variables {'bar': 'em0'}"

======================================================================
ERROR: test_encoding_is_utf8 (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 24, in tearDown
    rmtree(self.fs_tempdir)
  File "C:\Python27\Lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\Python27\Lib\shutil.py", line 250, in rmtree
    os.remove(fullname)
WindowsError: [Error 2] Le fichier spÚcifiÚ est introuvable: 'c:\\users\\gryom\\appdata\\local\\temp\\tmpo3rbed\\mapca'

======================================================================
ERROR: test_string_replacement (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 114, in test_string_replacement
    renderer=python_formatting_renderer,
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 43, in call_FUT
    ignored_files,
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\rendering.py", line 68, in render_structure
    fs_source_root = six.u(fs_source_root)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\six-1.3.0-py2.7.egg\six.py", line 313, in u
    return unicode(s, "unicode_escape")
UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 48-49: truncated \uXXXX escape

======================================================================
ERROR: test_subdirectories_created (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 52, in test_subdirectories_created
    renderer=python_formatting_renderer,
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 43, in call_FUT
    ignored_files,
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\rendering.py", line 68, in render_structure
    fs_source_root = six.u(fs_source_root)
  File "C:\Users\Gryom\Envs\mrbob\lib\site-packages\six-1.3.0-py2.7.egg\six.py", line 313, in u
    return unicode(s, "unicode_escape")
UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 48-49: truncated \uXXXX escape

======================================================================
ERROR: test_rendered_permissions_preserved (mrbob.tests.test_rendering.render_templateTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 185, in tearDown
    rmtree(self.fs_tempdir)
  File "C:\Python27\Lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\Python27\Lib\shutil.py", line 250, in rmtree
    os.remove(fullname)
WindowsError: [Error 5] AccÞs refus: 'c:\\users\\gryom\\appdata\\local\\temp\\tmpk11m7t\\unbound.conf'

======================================================================
FAIL: test_target_directory_inside_template_dir (mrbob.tests.test_configurator.ConfiguratorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_configurator.py", line 181, in test_target_directory_inside_template_dir
    {})
AssertionError: ConfigurationError not raised

======================================================================
FAIL: test_utf8 (mrbob.tests.test_parsing.write_configTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_parsing.py", line 214, in test_utf8
    self.assertEqual(f.read(), six.u("[variables]\nfoo.bar = %s\n\n") % var_)
AssertionError: u'[variables]\r\nfoo.bar = \u010d\r\n\r\n' != u'[variables]\nfoo.bar = \u010d\n\n'
- [variables]
?            -
+ [variables]
- foo.bar = \u010d
?            -
+ foo.bar = \u010d
-
+

======================================================================
FAIL: all of the above edgecases in one fixture
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 172, in test_compount_renaming
    self.assertTrue(os.path.exists(fs_rendered))
AssertionError: False is not true

======================================================================
FAIL: test_copied_file_is_renamed (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 137, in test_copied_file_is_renamed
    self.assertTrue(os.path.exists('%s/%s' % (self.fs_tempdir, '/foo.blubber.rst')))
AssertionError: False is not true

======================================================================
FAIL: test_directory_is_renamed (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 127, in test_directory_is_renamed
    self.assertTrue(os.path.exists('%s/%s' % (self.fs_tempdir, '/partsblubber/part')))
AssertionError: False is not true

======================================================================
FAIL: test_encoding_is_utf8 (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 105, in test_encoding_is_utf8
    self.assertEquals(f.read(), expected)
AssertionError: u'\u0106a\u010da.\r\n' != u'\u0106a\u010da.\n'
- \u0106a\u010da.
?      -
+ \u0106a\u010da.

-------------------- >> begin captured stdout << ---------------------
mkdir c:\users\gryom\appdata\local\temp\tmpo3rbed\map\u010da
Rendering Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\templates\encoding\u010d\map\u010da\\u0107a.bob to c:\users\gryom\appdata\local\temp\tmpo3rbed\map\u010da\\u0107a

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: test_ignored (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 82, in test_ignored
    self.assertTrue(os.path.exists('%s/%s' % (self.fs_tempdir, 'not_ignored')))
AssertionError: False is not true

======================================================================
FAIL: test_rendered_file_is_renamed (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 148, in test_rendered_file_is_renamed
    self.assertTrue(os.path.exists(fs_rendered))
AssertionError: False is not true

======================================================================
FAIL: test_rendered_file_is_renamed_dotted_name (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 160, in test_rendered_file_is_renamed_dotted_name
    '/foo_endpoint.py')))
AssertionError: False is not true

======================================================================
FAIL: test_skip_mrbobini_copying (mrbob.tests.test_rendering.render_structureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 61, in test_skip_mrbobini_copying
    self.assertTrue(os.path.exists('%s/%s' % (self.fs_tempdir, 'test')))
AssertionError: False is not true

======================================================================
FAIL: if the source is a template, it is rendered and the target file drops
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 227, in test_render_template
    self.assertTrue(fs_rendered.endswith('/unbound.conf'))
AssertionError: False is not true

======================================================================
FAIL: test_rendered_permissions_preserved (mrbob.tests.test_rendering.render_templateTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\DocRoot\ProjetsX\mr.bob\mrbob\tests\test_rendering.py", line 238, in test_rendered_permissions_preserved
    self.assertEqual(stat.S_IMODE(os.stat(fs_rendered).st_mode), 771)
AssertionError: 292 != 771

Name                            Stmts   Miss  Cover   Missing
-------------------------------------------------------------
mrbob                               0      0   100%
mrbob.cli                         104      2    98%   165, 180
mrbob.configurator                167     10    94%   81-89, 143, 151
mrbob.exceptions                    5      0   100%
mrbob.hooks                        12      0   100%
mrbob.parsing                      71      2    97%   47, 71
mrbob.rendering                    77      2    97%   73, 109
mrbob.tests                         0      0   100%
mrbob.tests.test_cli               60      5    92%   47, 108-113
mrbob.tests.test_configurator     293     12    96%   109, 124, 131, 142, 147-151, 156-160
mrbob.tests.test_hooks             31      0   100%
mrbob.tests.test_parsing          121      3    98%   104-108, 113
mrbob.tests.test_rendering        178      8    96%   54, 62, 116-117, 149, 173, 228, 349
-------------------------------------------------------------
TOTAL                            1119     44    96%
nose.plugins.cover: ERROR: TOTAL Coverage did not reach minimum required: 100%
make: *** [nosetests] Error 1

(mrbob) Z:\DocRoot\ProjetsX\mr.bob>
domenkozar commented 11 years ago

Thanks. I'll get around eventually and test this on a Windows box. If someone else wants to give it a shot - go for it!

Annakan commented 11 years ago

I started digging on it a bit.

First

you assumed than any ":" in a "pathname" would denote a "python_package" relative path specifier, but it happens than on windows ":" is present in every absolute file names since windows file names are of the form "DriveLetter:/path1/path2/path3/.../filename.extension" I made a quick fix in configurator.py / parse_template

    if ':' in template_name and not os.path.exists(template_name):
        path = resolve_dotted_path(template_name)
    else:
        path = os.path.realpath(template_name)

It is an ugly fix that assume no python package will be a one letter name that correspond (accidentally) to a real path name but ....

Second

We have subtle bug (or documented corner case) in six "u" function used in rendering.py, render_structure(), that happen to be the reason that sometimes on some paths and windows system mr.bob where not generating anything silently for me on other computers (to my dismay). The reason is the following : when passed the string Z:\\DocRoot\\ProjetsX\\mr.bob\\mrbob\\testsemplates\\multiconfig, a valid file path the six.u() returns the unicode string u'Z:\\DocRoot\\ProjetsX\\mr.bob\\mrbob\testsemplates\\multiconfig' that is interpreted as u'Z:\\DocRoot\\ProjetsX\\mr.bob\\mrbobestsemplates\\multiconfig' who does, of course not exist. Thus the for fs_source_dir, local_directories, local_files in os.walk(fs_source_root) iterates over nothing and no template is read and thus rendered.

The documentation here http://pythonhosted.org/six/index.html?highlight=u#six.u contains just the right amount of warning to confuse me about the safe usage of this function (there seem to be none to me since the call can be made upon any kind of data, that could happen not to be ASCII, this is a function with no guarantee and no way of making any) But the probability is very high that \\t is somehow misinterpreted with \t becoming an escaped char in char by char analysis, that happen to be Unicode encoded the wrong way).

I did not see a good way out of this right away so will postpone a fix till tomorrow, my intuition is that this transformation is happening too late anyway, we should move to and from Unicode at the edges of the application (input and output parameters) and not in the middle of the function (where sanity checks are less likely).

I need to sleep on it ;)

Annakan commented 11 years ago

FYI I was able to fix a class of bugs pertaining on managing tempfiles on windows (it seems you can open a file twice on unix with the same process but not on windows, top of my head hypothesis not tested).

I need to test my "unicode fix" with python3 and on unix.

I have trouble in the test with mock (I hate mock as much as I love it ) who seems hellbent on considering urlretrieve must be called with two parameters when only one is possible but it must be confusing what is truly patched (often the case with mock).

I will see if I can solve those remaining tests and test on Python3 on Windows at least.

domenkozar commented 11 years ago

Hi @Annakan, any progress on this?

Annakan commented 11 years ago

Hello sorry for the delay in answering, busy summer busy days. I did not progress much further than what was documented here, mostly I need to go beyond the ugly unicode fix (at the consumption point) I did to be able to fix it at the origin point where the information is gathered (and should be made unicode right away), but to do that I need to have a passing test suite and I did not find a workaround my mock problem. I probably won't have time to give to this before the end of the month, I might then have more time to devote to fixing this or get some help on the aforementioned troubles. I truly hate pushing anything that do not have a proper test suite running (on windows).

iandennismiller commented 6 years ago

I'm curious what the status of this issue is. This has become a blocker for Windows compatibility.

It looks to me like the configurator.py hack is workable in practical application. What is the likelihood that tweak can be mainlined?

pbauer commented 6 years ago

@iandennismiller as far as I know there are no more problems with windows. At least I know that people use the plone-templates in https://github.com/plone/bobtemplates.plone with windows.

iandennismiller commented 6 years ago

I can report that resolve_dotted_path() in mr.bob==0.1.2 still has the same issue with fully-qualified Windows paths that include a colon as part of a disk drive - e.g. C:\. This ultimately throws ModuleNotFoundError: No module named 'C', where C is just the drive letter being parsed by mr.bob as a python module. 

mrbob C:\Users\idm\test-template
Traceback (most recent call last):
  File "C:\Users\idm\Envs\testing\Scripts\mrbob-script.py", line 11, in <module>
    load_entry_point('mr.bob==0.1.2', 'console_scripts', 'mrbob')()
  File "c:\users\idm\envs\testing\lib\site-packages\mr.bob-0.1.2-py3.6.egg\mrbob\cli.py", line 162, in main
    defaults=defaults)
  File "c:\users\idm\envs\testing\lib\site-packages\mr.bob-0.1.2-py3.6.egg\mrbob\configurator.py", line 138, in __init__
    self.template_dir, self.is_tempdir = parse_template(template)
  File "c:\users\idm\envs\testing\lib\site-packages\mr.bob-0.1.2-py3.6.egg\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "c:\users\idm\envs\testing\lib\site-packages\mr.bob-0.1.2-py3.6.egg\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "c:\users\idm\envs\testing\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'C'

We are making this work by using a relative pathname, so it's no longer a blocker on Windows for us. However, I wanted to follow up to let you know the Windows resolve_dotted_path() problem still persists with full Windows paths.

domenkozar commented 6 years ago

I would accept a PR that asserts : is not the second letter in the resolvable name. I'd assume no one has one char modules.