search

gramener / gramex

A visual analytics platform to build data-based web apps with less code.
https://gramener.com/gramex/guide/
Other
142 stars 57 forks source link

Python 3.8 Support #300

Open jaidevd opened 4 years ago

jaidevd commented 4 years ago

@sanand0 do we have a checklist of what is to be done before we can fully support Python 3.8?

sanand0 commented 4 years ago

Not yet @jaidevd. I do have a list of errors from Tornado 6, which is a subset of all Python 3.8 errors. But I don't think this is a full list.

======================================================================
ERROR: Failure: ImportError (cannot import name 'Task' from 'tornado.gen' (d:\anaconda\3.7\lib\site-packages\tornado\gen.py))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "d:\anaconda\3.7\lib\site-packages\nose\failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "d:\anaconda\3.7\lib\site-packages\nose\loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "d:\anaconda\3.7\lib\site-packages\nose\importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "d:\anaconda\3.7\lib\site-packages\nose\importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "d:\anaconda\3.7\lib\imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "d:\anaconda\3.7\lib\imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "D:\site\gramener.com\viz\async-gramex\testlib\test_log.py", line 7, in <module>
    from .test_transforms import eqfn
  File "D:\site\gramener.com\viz\async-gramex\testlib\test_transforms.py", line 9, in <module>
    from tornado.gen import coroutine, Task
ImportError: cannot import name 'Task' from 'tornado.gen' (d:\anaconda\3.7\lib\site-packages\tornado\gen.py)

======================================================================
ERROR: Failure: ImportError (cannot import name 'Task' from 'tornado.gen' (d:\anaconda\3.7\lib\site-packages\tornado\gen.py))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "d:\anaconda\3.7\lib\site-packages\nose\failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "d:\anaconda\3.7\lib\site-packages\nose\loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "d:\anaconda\3.7\lib\site-packages\nose\importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "d:\anaconda\3.7\lib\site-packages\nose\importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "d:\anaconda\3.7\lib\imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "d:\anaconda\3.7\lib\imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "D:\site\gramener.com\viz\async-gramex\testlib\test_transforms.py", line 9, in <module>
    from tornado.gen import coroutine, Task
ImportError: cannot import name 'Task' from 'tornado.gen' (d:\anaconda\3.7\lib\site-packages\tornado\gen.py)
-------------------- >> begin captured logging << --------------------
sqlitedict: DEBUG: closing SqliteDict(D:\site\gramener.com\viz\async-gramex\testlib\store\data.db)
sqlitedict.SqliteMultithread: DEBUG: received: --close--, send: --no more--
sqlitedict: DEBUG: closing SqliteDict(D:\site\gramener.com\viz\async-gramex\testlib\store\data.db)
sqlitedict.SqliteMultithread: DEBUG: received: --close--, send: --no more--
sqlitedict: DEBUG: closing SqliteDict(D:\site\gramener.com\viz\async-gramex\testlib\store\data.db)
sqlitedict.SqliteMultithread: DEBUG: received: --close--, send: --no more--
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: test_shell (tests.test_admin.TestAdmin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\site\gramener.com\viz\async-gramex\tests\test_admin.py", line 106, in test_shell
    ws = create_connection(ws_url)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_core.py", line 515, in create_connection
    websock.connect(url, **options)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_core.py", line 226, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_handshake.py", line 80, in handshake
    status, resp = _get_resp_headers(sock)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_handshake.py", line 165, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers)
websocket._exceptions.WebSocketBadStatusException: Handshake status 200 OK

======================================================================
ERROR: test_authorised_user (tests.test_websockethandler.TestWebSocketHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\site\gramener.com\viz\async-gramex\tests\test_websockethandler.py", line 57, in test_authorised_user
    'Cookie: {}'.format(get_cookie_header(self.session.cookies, Request(url=base_url)))
  File "d:\anaconda\3.7\lib\site-packages\websocket\_core.py", line 515, in create_connection
    websock.connect(url, **options)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_core.py", line 226, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_handshake.py", line 80, in handshake
    status, resp = _get_resp_headers(sock)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_handshake.py", line 165, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers)
websocket._exceptions.WebSocketBadStatusException: Handshake status 200 OK

======================================================================
ERROR: test_events (tests.test_websockethandler.TestWebSocketHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\site\gramener.com\viz\async-gramex\tests\test_websockethandler.py", line 24, in test_events
    ws = create_connection(base_url.replace('http://', 'ws://') + '/ws/socket')
  File "d:\anaconda\3.7\lib\site-packages\websocket\_core.py", line 515, in create_connection
    websock.connect(url, **options)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_core.py", line 226, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_handshake.py", line 80, in handshake
    status, resp = _get_resp_headers(sock)
  File "d:\anaconda\3.7\lib\site-packages\websocket\_handshake.py", line 165, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers)
websocket._exceptions.WebSocketBadStatusException: Handshake status 200 OK

======================================================================
FAIL: test_async (tests.test_functionhandler.TestFunctionHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\site\gramener.com\viz\async-gramex\tests\test_functionhandler.py", line 31, in test_async
    self.check('/func/async/args', text=text, **etag)
  File "D:\site\gramener.com\viz\async-gramex\tests\__init__.py", line 64, in check
    eq_(r.status_code, code, '%s: code %d != %d' % (url, r.status_code, code))
AssertionError: /func/async/args: code 500 != 200
-------------------- >> begin captured logging << --------------------
tornado.application: ERROR: Uncaught exception GET /func/async/args (::1)
HTTPServerRequest(protocol='http', host='localhost:9999', method='GET', uri='/func/async/args', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "d:\anaconda\3.7\lib\site-packages\tornado\web.py", line 1703, in _execute
    result = await result
  File "d:\anaconda\3.7\lib\site-packages\tornado\gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "D:\site\gramener.com\viz\async-gramex\gramex\handlers\functionhandler.py", line 61, in _get
    item = yield item
  File "d:\anaconda\3.7\lib\site-packages\tornado\gen.py", line 735, in run
    value = future.result()
  File "d:\anaconda\3.7\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "D:\site\gramener.com\viz\async-gramex\tests\utils.py", line 78, in async_args
    result = yield gen.Task(params_as_json, *args, **kwargs)
AttributeError: module 'tornado.gen' has no attribute 'Task'
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_iterator (tests.test_functionhandler.TestFunctionHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\site\gramener.com\viz\async-gramex\tests\test_functionhandler.py", line 45, in test_iterator
    self.check('/func/iterator/async?x=1&x=2&x=3', text='123', **no_etag)
  File "D:\site\gramener.com\viz\async-gramex\tests\__init__.py", line 64, in check
    eq_(r.status_code, code, '%s: code %d != %d' % (url, r.status_code, code))
AssertionError: /func/iterator/async?x=1&x=2&x=3: code 500 != 200
-------------------- >> begin captured logging << --------------------
tornado.application: ERROR: Uncaught exception GET /func/iterator/async?x=1&x=2&x=3 (::1)
HTTPServerRequest(protocol='http', host='localhost:9999', method='GET', uri='/func/iterator/async?x=1&x=2&x=3', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "d:\anaconda\3.7\lib\site-packages\tornado\web.py", line 1703, in _execute
    result = await result
  File "d:\anaconda\3.7\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "D:\site\gramener.com\viz\async-gramex\gramex\handlers\functionhandler.py", line 58, in _get
    for item in result:
  File "D:\site\gramener.com\viz\async-gramex\tests\utils.py", line 71, in iterator_async
    future = gen.Task(str_callback, val)
AttributeError: module 'tornado.gen' has no attribute 'Task'
--------------------- >> end captured logging << ---------------------
sanand0 commented 4 years ago

@jaidevd @bkamapantula -- could you try running the Gramex Guide on Python 3.8 and list all errors you see, please?

bkamapantula commented 4 years ago

sure, will update by evening

bkamapantula commented 4 years ago

Steps to create a python 3.8 environment:

next: test services (email, alerts, scheduler, watch files), will update on 5th Oct

update on 7th oct: captured the errors at https://github.com/gramener/gramex-guide/issues/33, not necessarily related to python 3.8

ref: https://docs.python.org/3.8/whatsnew/3.8.html#porting-to-python-3-8

sanand0 commented 2 years ago

As of d51a3c53, Gramex works on Tornado 6. This resolves one bottleneck to migration.

sanand0 commented 2 years ago

@jaidevd -- on the master branch, I've fixed most of the bugs for Pandas 1.3. The environment is Gramex plus these commands:

pip install --upgrade pandas numpy
conda install -c conda-forge rpy2==3.4.5

The tests that fail are now in MLHandler. Could you take a look, please?

======================================================================
FAIL: test_clear_cache (tests.test_mlhandler.TestMLHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\site\gramener.com\viz\async-gramex\tests\test_mlhandler.py", line 184, in test_clear_cache
    self.assertListEqual(self.get('/mlhandler?_cache').json(), [])
AssertionError: Lists differ: [{'sepal_length': 5.1, 'sepal_width': 3.5,[44389 chars]ca'}] != []

First list contains 414 additional elements.
First extra element 0:
{'sepal_length': 5.1, 'sepal_width': 3.5, 'petal_length': 1.4, 'petal_width': 0.2, 'species': 'setosa'}

Diff is 52307 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_get_cache (tests.test_mlhandler.TestMLHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\site\gramener.com\viz\async-gramex\tests\test_mlhandler.py", line 280, in test_get_cache
    pd.testing.assert_frame_equal(df, self.df)
  File "D:\anaconda\3.7\lib\site-packages\pandas\_testing\asserters.py", line 1250, in assert_frame_equal
    obj, f"{obj} shape mismatch", f"{repr(left.shape)}", f"{repr(right.shape)}"
  File "D:\anaconda\3.7\lib\site-packages\pandas\_testing\asserters.py", line 665, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame are different

DataFrame shape mismatch
[left]:  (552, 5)
[right]: (138, 5)

======================================================================
FAIL: test_retrain (tests.test_mlhandler.TestMLHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\site\gramener.com\viz\async-gramex\tests\test_mlhandler.py", line 414, in test_retrain
    self.assertListEqual(resp.json(), [])
AssertionError: Lists differ: [{'sepal_length': 5.1, 'sepal_width': 3.5,[59201 chars]ca'}] != []

First list contains 552 additional elements.
First extra element 0:
{'sepal_length': 5.1, 'sepal_width': 3.5, 'petal_length': 1.4, 'petal_width': 0.2, 'species': 'setosa'}

Diff is 69741 characters long. Set self.maxDiff to None to see it.
jaidevd commented 2 years ago

Thanks, @sanand0 - I'll fix this by tomorrow.

sanand0 commented 2 years ago

Pandas requires sqlalchemy 1.4+ -- I'll work on Gramex support for that since there are breaking changes in 1.4.