Getting messytables to run on Python 3

[pwalsh]

Does any know, informally or otherwise, what it will take to get messytables running on Python 3?

I'm keen to use various functions and modules from messytables, but I'm trying to maintain 2.7/3.3/3.4 support in my own libraries.

[rossjones]

It's mostly dependencies - it's discussed briefly at https://github.com/okfn/messytables/issues/114

[pwalsh]

I've just done a quick review of parts of the code as I wanted to replace some stuff I have with messy tables.types.

It is def. more than dependencies. I just identified these (again, quick scan):

• Use of old-style exceptions (e.g.: except Exception, e:)
• Use of basestring requires a compat layer (six or otherwise)
• file and other IO stuff needs to be ported to use the io module

Looks like a chunk of work. I'd be interested to do some of it, but it would also be good to know if anyone else has started in this direction/would be interested in co-ordinating.

[rossjones]

Not as far as I'm aware, and I didn't get around to looking :(

[pwalsh]

Question: Is Python 2.6 compat important?

[rossjones]

I think so as it's used by a few ckan extensions and 2.6 is still supported (for now).

[scraperdragon]

We've been discussing what it'd take to use Python 3 with messytables (which we're using as a base for http://github.com/scraperwiki/xypath) at ScraperWiki - we might be interested in this happening at some point.. but that'd possibly a few months down the line and may not happen at all.

[drj11]

@pwalsh you start by running the code through 2to3, not by taking the code as-is and running it on Python 3.

[scraperdragon]

@pwalsh there's now a highly experimental py3 branch which passes most of the tests - there might be some howlers of bugs in there though!

Known problems:

• PDF support is broken - I don't think PDFMiner is py3 compatible.
• HTML schema support is broken - it's a problem with json-schema. fixed

I'd be especially careful around anything which touches CSV files, especially unicode; there's some pretty precarious code which bypasses the entirety of ilines, automatically fixed code which does nothing (if x: if not x: do something) and probably showstopping bugs where the test coverage is poor.

No effort whatsoever has been made to make this lovely bilingual py2/3 code yet.

Good luck! And mind the bugs don't bite!

[scraperdragon]

@pwalsh there's now a py23 branch that has bilingual python 2 / 3 code working; please take a look!

[pwalsh]

Hey @scraperdragon that's great, I'll take a look.

[pwalsh]

already merged.

[L022937]

messytables==0.15.2 is broken for python3.10

File "/home/airflow/.local/lib/python3.10/site-packages/messytables/headers.py", line 3, in from messytables.core import Cell File "/home/airflow/.local/lib/python3.10/site-packages/messytables/core.py", line 2, in from collections import Mapping ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/lib/python3.10/collections/init.py)

[drj11]

@L022937 you should open a new issue for this. The underlying problem is that Mapping has been moved from collections to collections.abc.

Changing the import line to from collections.abc import Mapping would probably work (but i'm not involved with this project enough to open a PR to do that).