QST: Python 3.9 testing in CI

wumpus commented 4 years ago

I use Travis CI to test, and I've tested all of my projects against the upcoming python 3.9 release. The only problem I'm seeing is dependent packages is pandas.

Pandas + 3.9 takes a long time to build and then dies with a compiler error. It also does this when I install cython first, which is the usual workaround when a new version of Python is close to release. Is some not-yet-mainstream cython version needed? I suppose there will be a wheel by the time 3.9.0 comes out in October?

Example CI log: https://travis-ci.com/github/wumpus/paramsurvey/jobs/383337409

  pandas/_libs/writers.c:4942:5: error: â€˜_PyUnicode_get_wstr_lengthâ€™ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  pandas/_libs/writers.c:4942:5: error: â€˜PyUnicode_AsUnicodeâ€™ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
   Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                               ^
  pandas/_libs/writers.c:4942:5: error: â€˜_PyUnicode_get_wstr_lengthâ€™ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  cc1: all warnings being treated as errors
  error: command '/usr/bin/gcc' failed with exit code 1

The logfile contains complete version information about the travisci environment.

jbrockmendel commented 4 years ago

@WillAyd is cython3 the blocker here?

WillAyd commented 4 years ago

I think the problem is we import PyUnicode_GET_SIZE in writers.pyx which is deprecated. Might be able to swap out with PyUnicode_GET_LENGTH assuming that is available in Cython

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_GET_SIZE

WillAyd commented 4 years ago

@wumpus if you'd like to try I think you just need to update this and any occurrence within the module

https://github.com/pandas-dev/pandas/blob/ddf2f05e25ca94794e295cf173e3fbc351581a78/pandas/_libs/writers.pyx#L5

wumpus commented 4 years ago

I made that one change, and (no surprise) there are other calls to the "legacy" Unicode interface beyond that first one.

The deprecation plan is PEP 623 and py3.9 has the first deprecations.

I'm confused, though, why this isn't already fixed in pandas -- is -Wall turned off the the pandas CI build under py3.9?

WillAyd commented 4 years ago

We have the warnings turned on but they must not be throwing a deprecation warning until 3.9.

Can you post the full error list?

wumpus commented 4 years ago

Well, with just the one change to change GET_SIZE to GET_LENGTH (https://travis-ci.com/github/wumpus/pandas-appender/jobs/384012415) this is the next error:

  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -fPIC -DNPY_NO_DEPRECATED_API=0 -I/tmp/pip-build-env-589femm7/overlay/lib/python3.9/site-packages/numpy/core/include -I/home/travis/virtualenv/python3.9-dev/include -I/opt/python/3.9-dev/include/python3.9 -c pandas/_libs/writers.c -o build/temp.linux-x86_64-3.9/pandas/_libs/writers.o -Werror
  pandas/_libs/writers.c: In function ‘__pyx_f_6pandas_5_libs_7writers_word_len’:
  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  pandas/_libs/writers.c:4942:5: error: ‘PyUnicode_AsUnicode’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
   Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                               ^
  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  cc1: all warnings being treated as errors
  error: command '/usr/bin/gcc' failed with exit code 1

PEP 623 appears to have a complete list of what was deprecated in py3.9. And yes, they were first deprecated in 3.9.

Edit: I looked at the Pandas travis CI looking for a successful 3.9 build, got tired of looking before I found one.

WillAyd commented 4 years ago

Strange - I made the edits above and it seemed to get rid of those warnings. I'll push up a PR soon

wumpus commented 4 years ago

Well, I'm getting them with cython-0.29.21 (the current release), and the symbols aren't in pandas, they're inserted by cython.

wumpus commented 4 years ago

@WillAyd Looks like your PR built in Travis with cython-3, which is what @jbrockmendel asked you up top. With that clue, now I can try it for myself.

wumpus commented 4 years ago

... and that seems to have worked for my code to pass its tests. So I didn't run the pandas tests and my code doesn't use any Unicode, but indeed changing PyUnicode_GET_SIZE to _LENGTH and using Cython==3.0a6 seems to be a winning combination in python-3.9.0rc1

wumpus commented 4 years ago

Fixed by PR 36393

pandas-dev / pandas

QST: Python 3.9 testing in CI #36279