documentation needed: install pre-requisites on a Windows host using Cygwin

iandennismiller commented 9 years ago

How does one install Flask-Diamond on a Windows host? Is Cygwin the best way? Please share your knowledge so it can be incorporated into the System Requirements document.

julianz commented 7 years ago

As far as I can see Flask-Diamond will install on Windows using a regular Windows Python 3.5 install (not Cygwin) apart from the mr.bob prerequisite, which fails when attempting to build a wheel. I'll dig a bit further.

iandennismiller commented 7 years ago

Thanks for the update! Please keep us posted.

julianz commented 7 years ago

OK, here's where I'm up to. I'll build a Scripts/flask-diamond.cmd file next to make the script work on the command line:

Using python 3.5 (from python.org) on Windows 8.1 (it should be the same on W10 though)

pip install Flask-Diamond fails because mr.bob fails building a wheel. It does seem to successfully install all other prereqs though, so do that anyway.
then you can easy_install -U mr.bob
mr.bob needs readline, so on Windows you have to pip install pyreadline
pip install Flask-Diamond still fails as it doesn't seem to see that mr.bob is installed (or is it a version thing? It uninstalls the one we just easy_installed and then blows up building the wheel again, so go back round the easy_install route once more)
pip install --no-deps Flask-Diamond now succeeds. I'm assuming the other prereqs have been satisfied.
the flask-diamond script in the Scripts/ directory has no suffix so it won't execute on Windows.
- best idea is to provide a flask-diamond.cmd that finds and calls the python file.
- if you just add a copy with a .py suffix you run into issues with virtualenv because Windows maps .py to the wrong Python interpreter.
For now, copying the flask-diamond script and adding a .py extension, then executing with the right interpreter does work. It throws a bunch of ExtDeprecationWarnings from Flask about importing extensions the wrong way, but other than that it seems to execute. Maybe the warnings are to do with recent developments in Flask?
Running the scaffold command throws mr.bob errors to do with a missing .mrbob.ini file. I've pasted the whole thing below. (My virtualenv is in venv)

(venv) W:\Python\diamond>python venv\Scripts\flask-diamond.py scaffold tutorial-planets

W:\Python\diamond\venv\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.sqlalchemy is deprecated, use flask_sqlalchemy instead. .format(x=modname), ExtDeprecationWarning

W:\Python\diamond\venv\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.script is deprecated, use flask_script instead. .format(x=modname), ExtDeprecationWarning

W:\Python\diamond\venv\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.admin is deprecated, use flask_admin instead. .format(x=modname), ExtDeprecationWarning

W:\Python\diamond\venv\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.babelex is deprecated, use flask_babelex instead. .format(x=modname), ExtDeprecationWarning

W:\Python\diamond\venv\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.restful is deprecated, use flask_restful instead. .format(x=modname), ExtDeprecationWarning

W:\Python\diamond\venv\lib\site-packages\flask_diamond\skels usage: mrbob [-h] [-O TARGET_DIRECTORY] [-v] [-c CONFIG] [-V] [-l] [-w] [-n] [-q] [template] mrbob: error: config file does not exist: .mrbob.ini

julianz commented 7 years ago

The .mrbob.ini file does exist in the skels/tutorial-planets/ directory so I think it might be a path building/joining thing?

propupul commented 7 years ago

I'm using Python 3.4.4 on Windows 7 at first I got UnicodeEncodeError: 'charmap' codec can't encode character '\u2560' in position 91: character maps to <undefined> which running chcp 65001 fixed it. You can read it https://stackoverflow.com/questions/32382686/unicodeencodeerror-charmap-codec-cant-encode-character-u2010-character-m

As julianz pointed out:

the flask-diamond script in the Scripts/ directory has no suffix so it won't execute on Windows.

is correct. I'll continue to look into this.

Sanjurjo7 commented 7 years ago

I'm having the same trouble with the mr.bob install on Windows 10.

Is there a way this can be done with an alternative to mr.bob? What it's having problems with is this character (at least initially): ╠ which seems trivial in the long run.

I get that it's part of displaying a file structure cleanly, but if that's a big issue in Windows, then there may be a reason to deprecate mr.bob until it's fixed.

EDIT: There's a 0.1.2 version of mr.bob that claims to have fixed the Unicode errors during install, consider updating dependencies?

devsetgo commented 7 years ago

I had the issue with Mr.Bob on Windows and report the issue at the beginning of January. I have not gotten a response. When I install the library by itself I still get the unicode error as @Sanjurjo7 also notes.

Here is the issue and I am to new to python to know what exactly is going on - https://github.com/domenkozar/mr.bob/issues/84

ScriptingDad commented 7 years ago

@StuRyan56 there's a comment from @julianz that explains how to install mr.bob separately when it fails on Windows. I found recently though that flask-diamond still needs a slightly older version and I needed to do easy_install -U mr.bob==0.1.1 which will more than likely succeed and satisfy the requirement.

ScriptingDad commented 7 years ago

@julianz I've managed to get the point of running the python ~/virt/script/scaffold app (note I didn't add the .py not sure if it matter but it seems to run throwing all the import errors mentioned above. I get the following as an error. Seems to be mr.bob as yet another hurdle.

C:\Users\alharris\.virtualenvs\flask-diamond\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.sqlalchemy is deprecated, use flask_sqlalchemy instead.
  .format(x=modname), ExtDeprecationWarning
C:\Users\alharris\.virtualenvs\flask-diamond\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.script is deprecated, use flask_script instead.
  .format(x=modname), ExtDeprecationWarning
C:\Users\alharris\.virtualenvs\flask-diamond\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.admin is deprecated, use flask_admin instead.
  .format(x=modname), ExtDeprecationWarning
C:\Users\alharris\.virtualenvs\flask-diamond\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.babelex is deprecated, use flask_babelex instead.
  .format(x=modname), ExtDeprecationWarning
C:\Users\alharris\.virtualenvs\flask-diamond\lib\site-packages\flask\exthook.py:71: ExtDeprecationWarning: Importing flask.ext.restful is deprecated, use flask_restful instead.
  .format(x=modname), ExtDeprecationWarning
C:\Users\alharris\.virtualenvs\flask-diamond\lib\site-packages\flask_diamond\skels
Traceback (most recent call last):
  File "C:\Users\alharris\.virtualenvs\flask-diamond\Scripts\mrbob-script.py", line 11, in <module>
    load_entry_point('mr.bob==0.1.1', 'console_scripts', 'mrbob')()
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\site-packages\mr.bob-0.1.1-py3.5.egg\mrbob\cli.py", line 162, in main
    defaults=defaults)
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\site-packages\mr.bob-0.1.1-py3.5.egg\mrbob\configurator.py", line 138, in __init__
    self.template_dir, self.is_tempdir = parse_template(template)
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\site-packages\mr.bob-0.1.1-py3.5.egg\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\site-packages\mr.bob-0.1.1-py3.5.egg\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
ImportError: No module named 'C'

(Edit) Should probably mention I'm on Windows 10 using Python 3.5.

iandennismiller commented 7 years ago

Hey @yokai117 - thanks for moving this forward. Sorry for the delay. Can you try running mrbob manually on the command line?

mrbob -w --config .mrbob.ini -O . C:\Users\alharris\.virtualenvs\flask-diamond\lib\site-packages\flask_diamond\skels/app

ScriptingDad commented 7 years ago

@iandennismiller I just gave that command a try and ended up with the exact same error. And tracing through to see what it's choking on in the stack trace yielded an unexpected result. I dropped some print statements at the top of find_and_load_unlocked but got no output.

λ mrbob -w --config .mrbob.ini -O .\app C:\Users\alharris\.virtualenvs\flask-diamond\lib\site-packages\flask_diamond\skels\app
Traceback (most recent call last):
  File "C:\Users\alharris\.virtualenvs\flask-diamond\Scripts\mrbob-script.py", line 11, in <module>
    load_entry_point('mr.bob==0.1.1', 'console_scripts', 'mrbob')()
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\site-packages\mr.bob-0.1.1-py3.5.egg\mrbob\cli.py", line 162, in main
    defaults=defaults)
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\site-packages\mr.bob-0.1.1-py3.5.egg\mrbob\configurator.py", line 138, in __init__
    self.template_dir, self.is_tempdir = parse_template(template)
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\site-packages\mr.bob-0.1.1-py3.5.egg\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\site-packages\mr.bob-0.1.1-py3.5.egg\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "c:\users\alharris\.virtualenvs\flask-diamond\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
ImportError: No module named 'C'

I also pulled down a brand new copy of flask-diamond to make sure everything was as clean as it could be. It seems like it doesn't like the path because the name that is passed into resolve_dotted_path is the path provided in the above command. I'm experimenting with dotted notation to see if I can get it to find the correct path.

ScriptingDad commented 7 years ago

Alright so I narrowed things down to the configurator.py file line 38

def resolve_dotted_path(name):
    module_name, dir_name = name.rsplit(':', 1)
    module = import_module(module_name)

after the rsplit the module_name ended up only being the letter "C" which is absolutely causing my problem but I'm not really sure what this thing is expecting to get.

ScriptingDad commented 7 years ago

Alright finally update for tonight. Seems that the problem stems from the configurator.py file in mr.bob package. The specific function is parse_template on line 69 and the problem point is on line 91.

if ':' in template_name:
        path = resolve_dotted_path(template_name)
    else:
        path = os.path.realpath(template_name)

It seems to assume that if there's a ":" in the path that it's a python type path and tries to resolve it through the dotted path but since Windows paths are usually c:\path to file this short circuits it. I took a page out of @annakan's book and simply change that function to the following:

if ':' in template_name and not os.path.exists(template_name):
        path = resolve_dotted_path(template_name)
    else:
        path = os.path.realpath(template_name)

This seems to allow the skel template to be applied using the cli interface at least.

Referred to: https://github.com/domenkozar/mr.bob/issues/49

Apparently it's been known since 2013 that this causes problems for Windows.

ScriptingDad commented 7 years ago

I guess my next question is. If we want to allow Windows maybe we need to use the dotted path to the skel file? What would that look like?

iandennismiller commented 6 years ago

Hi @yokai117 - thanks for all your investigating on this!

I now have my own Windows 10 machine for experimentation. I am using the following invocation, which looks to me like a good general form for Win10 invocation of the scaffold:

workon my-app
%VIRTUAL_ENV%\scripts\python.exe %VIRTUAL_ENV%\scripts\flask-diamond scaffold tutorial-planets

I'm able to reproduce the issue addressed in domenkozar/mr.bob#49 - thanks for tracking that down!

Traceback (most recent call last):
  File "C:\Users\idm\Envs\testing\Scripts\mrbob-script.py", line 11, in <module>
    load_entry_point('mr.bob==0.1.1', 'console_scripts', 'mrbob')()
  File "c:\users\idm\envs\testing\lib\site-packages\mr.bob-0.1.1-py3.6.egg\mrbob\cli.py", line 162, in main
    defaults=defaults)
  File "c:\users\idm\envs\testing\lib\site-packages\mr.bob-0.1.1-py3.6.egg\mrbob\configurator.py", line 138, in __init__
    self.template_dir, self.is_tempdir = parse_template(template)
  File "c:\users\idm\envs\testing\lib\site-packages\mr.bob-0.1.1-py3.6.egg\mrbob\configurator.py", line 92, in parse_template
    path = resolve_dotted_path(template_name)
  File "c:\users\idm\envs\testing\lib\site-packages\mr.bob-0.1.1-py3.6.egg\mrbob\configurator.py", line 39, in resolve_dotted_path
    module = import_module(module_name)
  File "c:\users\idm\envs\testing\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'C'

I see what you're saying about resolve_dotted_path(). Just so I understand, are you saying you patched mrbob's configurator.py in order to make this work?

ScriptingDad commented 6 years ago

Since the scaffolding command itself that launches mr.bob seems to only handle a fully qualified path the only way to get mrbob to run properly was to tell it to fail if the file path existed at the point that it was making the choice between resolving a dotted path or a file path.

We may not have to go that far if it's possible to provide a relative python style path to the skel file in the scaffold call which would ensure similar functionality on unix and windows.

Just thinking outloud.

PS. just incase I wasn't clear, yes I patched the configurator.py in mr.bob's implementation the same as it was mentioned in that issue I referenced.

iandennismiller commented 6 years ago

Hi @yokai117 - I've implemented a possible solution in the develop branch and I am wondering if it works for you. I got rid of the full path in the flask-diamond invocation of mrbob and made the skel path relative.

I'm going to describe my development environment and the steps I took. I would appreciate your help in determining whether you can reproduce these results.

dev environment

I am using Windows 10, cmd.exe, git, python, and GNU make. I am not using Cygwin, the Windows/Ubuntu Linux subsystem, or any other UNIX-like environment. The tools I'm using are available from the following links:

(flask-diamond) C:\Users\idm\Work\flask-diamond>ver

Microsoft Windows [Version 10.0.15063]

(flask-diamond) C:\Users\idm\Work\flask-diamond>git --version
git version 2.15.0.windows.1

(flask-diamond) C:\Users\idm\Work\flask-diamond>python --version
Python 3.6.3

(flask-diamond) C:\Users\idm\Work\flask-diamond>make --version
GNU Make 3.81

initialization

Using PowerShell, I installed the following python packages as Administrator:

start-process powershell –verb runAs
pip install virtualenvwrapper
pip install virtualenvwrapper-win

I used cmd.exe for all the remaining steps - no longer as Administrator. I set up a workspace in my home directory called Work.

cmd.exe
mkdir %HOMEPATH%\Work
cd %HOMEPATH%\Work

obtain develop branch of flask-diamond

There are a few steps to get the develop branch:

create a python virtualenv called flask-diamond
get the source code from github
switch to the develop branch

mkvirtualenv flask-diamond
workon flask-diamond
git clone https://github.com/diamond-org/flask-diamond.git
cd flask-diamond
git checkout develop

build Flask-diamond from source

install requirements
manually install pyreadline and mr.bob for Windows compatibility
install the flask-diamond project in the virtualenv

make requirements
pip install pyreadline
easy_install -U mr.bob==0.1.2
make install

validate

Make sure you have 0.5.2rc1, which confirms that you are using the latest version from the develop branch. The latest from pypi is 0.5.1 so it shouldn't report this version.

(flask-diamond) C:\Users\idm\Work\flask-diamond>pip show flask-diamond
Name: flask-diamond
Version: 0.5.2rc1

scaffold tutorial-planets

Try to scaffold the planets tutorial. Just call the app planets during the mrbob scaffolding.

mkdir %HOMEPATH%\Work\planets
cd %HOMEPATH%\Work\planets
flask-diamond scaffold app
flask-diamond scaffold tutorial-planets

launch server

I was able to start the planets tutorial server from a new PowerShell window with the following:

cmd.exe
workon flask-diamond
cd Work\planets
set SETTINGS=%cd%\etc\conf\dev-win.conf
%VIRTUAL_ENV%\scripts\python.exe bin\manage.py runserver

Please note the new configuration file called dev-win.conf - this has a setting called LOG that must point to a legal path for the logs to be written to.

feedback

I am now able to use the mr.bob app skel and it produces a working flask-diamond scaffold. The resulting app actually works on a windows host. However, I have found a few ways this doesn't quite work as expected "out of the box" - particularly the Makefile.

It seems to me like the planets tutorial is now working on a Windows 10 host. Are you able to reproduce this result?

ScriptingDad commented 6 years ago

So I didn't use Cygwin which probably caused me more grief than it should have I was able to get it running though with no code changes but I still needed to install mr.bob using the command easy_install mr.bob==0.1.2 and I still needed to install pip install pyreadline.

Got the correct version installed.

pip show flask-diamond
Name: flask-diamond
Version: 0.5.2rc1

And there was two modifications to the dev-win.conf, the Log path and the SQLite DB path need to be changed. The format for the SQLite on Windows is a little different.

sqlite:///C:\\Users\\alharris\\Projects\\Python\\flask-diamond\\Temp\\my-app-dev.db

Only need 3 slashes before the drive letter.

iandennismiller commented 6 years ago

Thanks for running it from the develop branch! Thanks for noting the two changes to the install process - specifically pyreadline (which I added to requirements.txt) and mr.bob. I edited my comment above to reflect this, but here's the important update to the install procedure:

build Flask-diamond from source

install requirements
manually install mr.bob for Windows compatibility
install the flask-diamond project in the virtualenv

make requirements
easy_install -U mr.bob==0.1.2
make install

Based on your notes regarding the DB and LOG config items, I tweaked the template for dev-win, which is located at flask_diamond/skels/app/etc/conf/dev-win.conf.bob in the source code.

The last change I made is to create a Makefile target called server-win and db-win that are windows-compatible versions of the original commands. I'm not sure whether this is a good idea or not, but at a minimum it demonstrates how to accomplish these tasks on a Windows host.

So I've bumped the version to 0.5.2rc2 with commit 24f2418. I think that should address the Windows documentation and compatibility - and we can close this issue soon. I'm going to test this a little more myself, too, but I'm eager to release this so that Windows users can get started!

iandennismiller commented 6 years ago

The documentation isn't fully updated yet, but the develop branch has updated notes regarding Windows:

Now the develop documentation is also visible on readthedocs.

diamond-org / flask-diamond