Python 3 support, Windows installation fix, and others

[uranusjr]

Gigantic bundle of updates.

Unicode

All queries, statements and clauses now output Unicode objects (str on Python 3). This is done by adding

from __future__ import unicode_literals

in util.py, and always using ''.format and % inside format methods. This can be backward-incompatible if you have non-ASCII query strings, and does not use Unicode objects for your database connection (e.g. setting use_unicode=False in MySQL-Python), but should be good enough in most cases on non-Windows platform, and is better in the long run.

Python 3 Support

Python 3 is now supported in all non-deprecated modules, and are test-covered. This is mainly handled by the compat.py module. All print statements are also converted to functions with

from __future__ import print_function

This makes tests in deprecation modules fail, but I just skipped them (via configurations inside runtests.py). They can be converted to work, at least on Python 2, if you really want to, but I deemed it not worth the effort. I added a warning message in those modules (and deprecation.rst) to state that they are not tested anymore.

Many of the tests pass on both Python 2 and 3 after converting the print statements, but tests using dict may fail because hashing behaviours are different on Python 2 and 3, resulting in ordering change. I moved those tests to use unit-testing (using Nose), and use OrderedDict to promise ordering. They are inside tests/. The failing doctests are not removed, but converted to literal code blocks (using :: so they are not picked up by Sphinx’s doctest extension), or skipped explicitly with # doctest: +SKIP (documentation for this syntax).

The original contents inside tests/ are moved to oldtests/. I plan to add them back when I can find a good way to rewrite them. Will issue another pull request when that’s done.

Testing

Tests are now handled by Nose, doctests included (with sphinx-nose). I maintain the sphinx-nose project. Tell me if there are any problems.

Parts of the documentation are slightly modified to make doctests compatible with nose. See commit message of b458cd0 for detailed information.

You will need to install some dependencies to run them:

pip install -r requirements/test.txt

A simple Makefile is added to make things easier. To run the tests for the current Python version:

make test

Tox is also used to provide cross-version testing support. Currently Python 2.7 and 3.4 are tested. I intended to add 2.6 to the support matrix, but found that there are some incompatibilities in util.py. They can be fixed quite easily, but I’m not sure whether this is needed. Drop a word if you do want 2.6 support.

To run Tox tests:

pip install -r requirements/dev.txt
make test-all

You will need both python2.7 and python3.4 somewhere in your PATH.

.travis.yml is also modified to use the new test structure (without tox).

Misc. Bug Fixes

Installation on Windows was failing due to non-ASCII content inside README.rst. Fixed inside setup.py. (34112ad)

changes.rst was generating warnings about duplicate sqlmap references. Fixed with external hyperlink target. (ce449aa)

Some typo fixes in docs.

The _is_iterable_not_str function did not work as expected, and this became a problem on Python 3. This is fixed with other Python 3 fixes (921b031). The new implementation is:

def _is_iterable_not_str(x):
    return (not isinstance(x, compat.class_types)
            and not isinstance(x, compat.string_types + (bytes,))
            and hasattr(x, '__iter__'))

The autoparam feature used a somewhat strange (IMO) way to check whether the parameter is a param class object. I changed the implementation to use proper class identity checking:

# Inside _build_condition
if isinstance(v, compat.class_types) and issubclass(v, param):
    v = v(k)

[uranusjr]

I just added an edge case with Unicode and binary inputs. But I will need to test this on Windows tomorrow first, so please don’t merge this just yet.

[uranusjr]

All tests pass on Windows now.

[uranusjr]

Just added universal (Python 2 and 3) wheel support. To release a new version to PyPI, use

make release

which is equivalent to

python setup.py sdist bdist_wheel upload

One dependency is added to requirements/dev.txt for wheel-building support.

[uranusjr]

Rebased and squashed the commits a bit.

[moskytw]

It's a great PR!

No, I don't think we need to support Python 2.6. I am checking the code.

[moskytw]

looks great. just have some comments. :)