Make Python 2-3 compatible - Githubissues

markgw / pimlico

The Pimlico Processing Toolkit

http://pimlico.readthedocs.org/

GNU Lesser General Public License v3.0

6 stars 1 forks source link

Make Python 2-3 compatible #8

Closed markgw closed 5 years ago

markgw commented 7 years ago

We really need Pimlico to be compatible with both Python 2 and Python 3. Otherwise, people who use Python 3 will be put off from using Pimlico because they'd have to write their own modules in Python 2, or even worse if they're using their previously existing Python 3 code within Pimlico modules. For the same reasons, compatibility is better than just moving to Python 3.

This is a major change that will affect every Python source file in the project. It should be carried out in a branch and probably only after I've implemented much more comprehensive testing (#7), so I can find where it breaks things.

I think that most of the libraries Pimlico uses are 2-3 compatible, but should check this: if there are much-used libraries that don't support Python 3, we'll end up with modules or features that only work if you're using Python 2, which would be nasty.

markgw commented 5 years ago

I plan to use the future library for this.

We will use the 2-stage conversion process. Run futurize --stage1 first on the entire codebase. Then run stage2 and check all of the changes.

Because of the complexity of the codebase and some of the fancy importing stuff we do, it's possible stage2 could break things, so it will be important to ensure that those changes are tested in Py2 and 3.

This change is becoming a high priority, as it's stopping people from using Pimlico.

markgw commented 5 years ago

To be addressed in python3 branch.

Module tests and other unit tests are far from comprehensive, but there's probably enough to get started on this and I need to get started asap, as it's blocking people from using Pimlico.

markgw commented 5 years ago

Steps carried out to get started:

Install future in the default virtualenv
Create another virtualenv using python 3, so we can test converted code: virtualenv -p python3 lib/virtualenv/py3 This can now be used for Pimlico by adding the PIMENV env var: PIMENV=py3 ./pimlico.sh ...
Try using Py3 virtualenv: PIMENV=py3 ./all_test_pipelines.sh This fails, as our code is not yet syntactically valid Python 3, of course.
Activate default virtualenv to make futurize available: . ./lib/virtualenv/default/bin/activate
Run stage 1 conversion of all code using futurize, to make it more Python 3-like: futurize --stage1 -w -n src/python/pimlico/
This seemed to work fine. Run all test pipelines: cd bin ./all_test_pipelines.sh
There were a few more things that needed to be resolved to get all the tests passing. In particular, I added some new features to Python dependency checking and installation that allowed specification of versions for some of the pre-defined dependencies.

The entire codebase is now Python 3-like, but not yet runnable in Python 3.

Next we'll need to run stage 2, which is more dangerous and will require a lot of work to make sure everything is working.

markgw commented 5 years ago

Work now continues on stage 2.

The command futurize --stage2 -w src/python applies potentially dangerous conversions to make the code Python 3 compatible. I'll be reviewing the changes it's made manually and committing them as I go.

After that, I will run pipeline and unit tests and get things working. This means that, until then, the code might not be usable.

markgw commented 5 years ago

Working through all the files now. To-do list:

[x] core/dependencies/base.py
[x] core/dependencies/java.py
[x] core/dependencies/python.py
[x] core/dependencies/versions.py
[x] core/config.py
[x] cli/browser/tools/corpus.py
[x] cli/browser/tools/files.py
[x] cli/browser/tools/formatter.py
[x] cli/clean.py
[x] cli/debug/stepper.py
[x] cli/loaddump.py
[x] cli/locations.py
[x] cli/newmodule.py
[x] cli/pyshell.py
[x] cli/recover.py
[x] cli/reset.py
[x] cli/run.py
[x] cli/shell/base.py
[x] cli/shell/commands.py
[x] cli/status.py
[x] cli/subcommands.py
[x] cli/util.py
[x] core/external/java.py
[x] core/modules/base.py
[x] core/modules/execute.py
[x] core/modules/inputs.py
[x] core/modules/map/init.py
[x] core/modules/map/filter.py
[x] core/modules/map/multiproc.py
[x] core/modules/map/threaded.py
[x] core/modules/multistage.py
[x] core/modules/options.py
[x] datatypes/arrays.py
[x] datatypes/base.py
[x] datatypes/core.py
[x] datatypes/corpora/base.py
[x] datatypes/corpora/data_points.py
[x] datatypes/corpora/floats.py
[x] datatypes/corpora/grouped.py
[x] datatypes/corpora/ints.py
[x] datatypes/corpora/json.py
[x] datatypes/corpora/table.py
[x] datatypes/corpora/tokenized.py
[x] datatypes/dictionary.py
[x] datatypes/embeddings.py
[x] datatypes/features.py
[x] datatypes/files.py
[x] datatypes/gensim.py
[x] datatypes/sklearn.py
[x] modules/candc/execute.py
[x] modules/corenlp/wrapper.py
[x] modules/corpora/concat/info.py
[x] modules/corpora/group/info.py
[x] modules/corpora/interleave/info.py
[x] modules/corpora/shuffle/execute.py
[x] modules/corpora/subset/info.py
[x] modules/corpora/vocab_counter/execute.py
[x] modules/embeddings/dependencies/execute.py
[x] modules/embeddings/store_word2vec/execute.py
[x] modules/embeddings/word2vec/execute.py
[x] modules/features/term_feature_compiler/execute.py
[x] modules/features/term_feature_matrix_builder/execute.py
[x] modules/features/vocab_builder/execute.py
[x] modules/gensim/lda_doc_topics/execute.py
[x] modules/gensim/utils.py
[x] modules/input/embeddings/fasttext/execute.py
[x] modules/input/embeddings/glove/read.py
[x] modules/input/embeddings/word2vec/execute.py
[x] modules/input/text/raw_text_archives/info.py
[x] modules/input/text/raw_text_files/info.py
[x] modules/input/text_annotations/vrt/info.py
[x] modules/opennlp/coreference/execute.py
[x] modules/opennlp/coreference/info.py
[x] modules/opennlp/coreference_pipeline/execute.py
[x] modules/opennlp/coreference_pipeline/info.py
[x] modules/opennlp/ner/execute.py
[x] modules/opennlp/ner/ner.py
[x] modules/opennlp/pos/execute.py
[x] modules/opennlp/pos/tagger.py
[x] modules/opennlp/tokenize/tokenizer.py
[x] modules/output/text_corpus/execute.py
[x] modules/regex/annotated_text/execute.py
[x] modules/sklearn/logistic_regression/execute.py
[x] modules/sklearn/matrix_factorization/execute.py
[x] modules/sklearn/matrix_factorization/info.py
[x] modules/text/simple_tokenize/info.py
[x] modules/text/untokenize/info.py
[x] modules/utility/collect_files/execute.py
[x] modules/utility/collect_files/info.py
[x] modules/visualization/bar_chart/execute.py
[x] modules/visualization/bar_chart/plot_template.py
[x] modules/visualization/embeddings_plot/execute.py
[x] modules/visualization/embeddings_plot/plot_template.py
[x] test/pipeline.py
[x] utils/communicate.py
[x] utils/core.py
[x] utils/docs/commandgen.py
[x] utils/docs/modulegen.py
[x] utils/docs/rest.py
[x] utils/email.py
[x] utils/filesystem.py
[x] utils/format.py
[x] utils/network.py
[x] utils/pipes.py
[x] utils/pos.py
[x] utils/probability.py
[x] utils/progress.py
[x] utils/strings.py
[x] utils/web.py

markgw commented 5 years ago

About halway through the manual checks on stage 2 conversion at 4415c00e70448b9d3b83c92095a4f72bb5664e78.

Now got a lot of pipeline tests running. Some are still failing, of course:

pipelines/corpora/shuffle.conf[shuffle]
pipelines/input/fasttext.conf[vectors]
pipelines/input/glove.conf[vectors]
pipelines/embeddings/word2vec.conf[word2vec]
pipelines/embeddings/store_word2vec.conf[store]
pipelines/embeddings/store_tsv.conf[store]

markgw commented 5 years ago

Finished all manual checks of automatic stage 2 conversion in e0b888e.

Still the same pipeline tests failing given above.

Next steps:

Get all pipeline tests working in Py2
Get all unit tests working in Py2
Try running in Py3 and start addressing problems there

markgw commented 5 years ago

863c43d does stage 1 futurize for unit tests.

44588b4 does stage 2 futurize.

markgw commented 5 years ago

All unit tests and test pipelines passing in Python 2, from d60fc0cbf457706826f94a01bc34360b36764c27.

Next step is to work out how to specify using Python 3 or 2 for virtualenv creation process.

Then allow tests (unit and pipeline) to be run for both Py 2 and 3.

Then run the tests in Py 3 and address issues.

markgw commented 5 years ago

All test pipelines passing in both Python 2 and 3, as of 8d7047f4e31c92fa47b158dd82ea6c640bb26a88!

Unit tests not working yet. (Mostly for Python 2, but not at all for Python 3.)

markgw commented 5 years ago

All tests now passing on Py 2 and 3!

markgw commented 5 years ago

There are still some modules that need to be updated to the new datatypes system and these also need to be tested in Python 2&3. However, other than these (which need to be fixed anyway), everything seems to be working and pretty stable now.

Merge python3 branch in afd2ff5db87321a96d4dee63b5da558dfc6eb7ee.

Closing this issue, since this project is pretty much done! Woopee!