search

aliyun / aliyun-odps-python-sdk

ODPS Python SDK and data analysis framework
http://pyodps.readthedocs.io
Apache License 2.0
434 stars 97 forks source link

http.client.ResponseNotReady raised in 0.11.4 #205

Closed tongyifan closed 1 year ago

tongyifan commented 1 year ago

latest pyodps installed via pip install pyodps --index-url="https://pypi.org/simple"

pyodps: 0.11.4 urllib3: 2.0.2

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/odps/lib/requests/adapters.py", line 551, in send_with_fileio
    r = low_conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 195, in write_odps
    DataFrameODPS(data).persist(table_name, partition=partition_keys, drop_table=True, odps=o,
  File "/usr/local/lib/python3.8/dist-packages/odps/df/expr/expressions.py", line 43, in __call__
    return self._func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odps/df/expr/expressions.py", line 266, in persist
    return self._handle_delay_call('persist', self, name, partitions=partitions, partition=partition,
  File "/usr/local/lib/python3.8/dist-packages/odps/df/expr/expressions.py", line 43, in __call__
    return self._func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odps/df/expr/expressions.py", line 158, in _handle_delay_call
    result = getattr(engine, method)(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/core.py", line 740, in persist
    return self._action(*exprs_args_kwargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/core.py", line 554, in _action
    res = self._execute_dag(dag, ui=ui, async_=async_, n_parallel=n_parallel,
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/core.py", line 798, in _execute_dag
    return dag.execute(ui=ui, async_=async_, n_parallel=n_parallel, timeout=timeout,
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/core.py", line 337, in execute
    results = self._run(ui, progress_proportion)
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/core.py", line 212, in _run
    res = call(ui=ui, progress_proportion=progress_proportion / len(calls))
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/core.py", line 191, in __call__
    res = self.run(ui=ui, progress_proportion=progress_proportion)
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/core.py", line 690, in run
    result = engine._do_persist(expr_dag, expr, name, **kw)
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/pd/engine.py", line 61, in wrapped
    return fun(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/pd/engine.py", line 359, in _do_persist
    write_table(df, t, ui=ui, cast=cast, overwrite=overwrite, partitions=partitions, partition=partition,
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/utils.py", line 251, in write_table
    _write_table_no_partitions(frame, table, ui, cast=cast, overwrite=overwrite, partition=partition,
  File "/usr/local/lib/python3.8/dist-packages/odps/df/backends/utils.py", line 209, in _write_table_no_partitions
    writer.write(gen())
  File "/usr/local/lib/python3.8/dist-packages/odps/models/table.py", line 360, in __exit__
    self.close()
  File "/usr/local/lib/python3.8/dist-packages/odps/models/table.py", line 917, in close
    super(_TableWriter, self).close()
  File "/usr/local/lib/python3.8/dist-packages/odps/models/table.py", line 339, in close
    block_info.writer.close()
  File "/usr/local/lib/python3.8/dist-packages/odps/tunnel/io/writer.py", line 360, in close
    self._req_io.close()
  File "/usr/local/lib/python3.8/dist-packages/odps/rest.py", line 110, in close
    self.__exit__(None, None, None)
  File "/usr/local/lib/python3.8/dist-packages/odps/rest.py", line 118, in __exit__
    self._ctx.__exit__(*exc_info)
  File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.8/dist-packages/odps/rest.py", line 233, in _request
    yield writer
  File "/usr/local/lib/python3.8/dist-packages/odps/lib/requests/sessions.py", line 352, in __exit__
    self._ctx.__exit__(*exc_info)
  File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.8/dist-packages/odps/lib/requests/sessions.py", line 723, in _send
    yield writer
  File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.8/dist-packages/odps/lib/requests/adapters.py", line 554, in send_with_fileio
    r = low_conn.getresponse()
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 440, in getresponse
    raise ResponseNotReady()
http.client.ResponseNotReady
wjsi commented 1 year ago

Could you please provide minimal code to reproduce this error? As code below cannot simply reproduce.

from odps import DataFrame, options
import numpy as np, pandas as pd
pddf = pd.DataFrame(np.random.rand(10000, 10), columns=list("ABCDEFGHIJ"))
df = DataFrame(pddf).persist("test_pd_persist_table", lifecycle=1)
tongyifan commented 1 year ago

Could you please provide minimal code to reproduce this error? As code below cannot simply reproduce.

from odps import DataFrame, options
import numpy as np, pandas as pd
pddf = pd.DataFrame(np.random.rand(10000, 10), columns=list("ABCDEFGHIJ"))
df = DataFrame(pddf).persist("test_pd_persist_table", lifecycle=1)

But this code will throw ResponseNotReady in my environment.

here's another script to repro this issue

if name == "main": o = ODPS( "my_access_id", "my_secret_key", "my_project", "my_endpoint", )

df = pd.DataFrame(
    {
        "id_code": ["1", "2", "3"],
        "ds": pd.to_datetime(["2023-01-01", "2023-01-01", "2023-01-01"]),
        "value": [4, 5, 6],
    }
)

DataFrame(df).persist(
    "table_name",
    drop_table=True,
    odps=o,
    lifecycle=1,
)

- output

/Users/tongyifan/PycharmProjects/pyodps_debug/venv/bin/python /Users/tongyifan/PycharmProjects/pyodps_debug/main.py Traceback (most recent call last): File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/lib/requests/adapters.py", line 551, in send_with_fileio r = low_conn.getresponse(buffering=True) TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/tongyifan/PycharmProjects/pyodps_debug/main.py", line 20, in DataFrame(df).persist( File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/expr/expressions.py", line 43, in call return self._func(*args, kwargs) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/expr/expressions.py", line 266, in persist return self._handle_delay_call('persist', self, name, partitions=partitions, partition=partition, File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/expr/expressions.py", line 43, in call return self._func(*args, *kwargs) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/expr/expressions.py", line 158, in _handle_delay_call result = getattr(engine, method)(args, kwargs) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/core.py", line 740, in persist return self._action(exprs_args_kwargs, kwargs) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/core.py", line 554, in _action res = self._executedag(dag, ui=ui, async=async_, n_parallel=n_parallel, File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/core.py", line 798, in _executedag return dag.execute(ui=ui, async=async_, n_parallel=n_parallel, timeout=timeout, File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/core.py", line 337, in execute results = self._run(ui, progress_proportion) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/core.py", line 212, in _run res = call(ui=ui, progress_proportion=progress_proportion / len(calls)) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/core.py", line 191, in call res = self.run(ui=ui, progress_proportion=progress_proportion) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/core.py", line 690, in run result = engine._do_persist(expr_dag, expr, name, kw) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/pd/engine.py", line 61, in wrapped return fun(self, args, *kwargs) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/pd/engine.py", line 359, in _do_persist write_table(df, t, ui=ui, cast=cast, overwrite=overwrite, partitions=partitions, partition=partition, File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/utils.py", line 251, in write_table _write_table_no_partitions(frame, table, ui, cast=cast, overwrite=overwrite, partition=partition, File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/df/backends/utils.py", line 209, in _write_table_no_partitions writer.write(gen()) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/models/table.py", line 360, in exit self.close() File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/models/table.py", line 917, in close super(_TableWriter, self).close() File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/models/table.py", line 339, in close block_info.writer.close() File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/tunnel/io/writer.py", line 360, in close self._req_io.close() File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/rest.py", line 110, in close self.exit(None, None, None) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/rest.py", line 118, in exit self._ctx.exit(exc_info) File "/opt/homebrew/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 120, in exit next(self.gen) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/rest.py", line 233, in _request yield writer File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/lib/requests/sessions.py", line 352, in exit self._ctx.exit(*exc_info) File "/opt/homebrew/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 120, in exit next(self.gen) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/lib/requests/sessions.py", line 723, in _send yield writer File "/opt/homebrew/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 120, in exit next(self.gen) File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/odps/lib/requests/adapters.py", line 554, in send_with_fileio r = low_conn.getresponse() File "/Users/tongyifan/PycharmProjects/pyodps_debug/venv/lib/python3.8/site-packages/urllib3/connection.py", line 440, in getresponse raise ResponseNotReady() http.client.ResponseNotReady

Process finished with exit code 1

wjsi commented 1 year ago

Verified to be an issue of urllib3 2.0.x. Please use an earlier release of urllib3.

tongyifan commented 1 year ago

LGTM, v0.11.4.post0 works fine.