search

waditu / tushare

TuShare is a utility for crawling historical data of China stocks
BSD 3-Clause "New" or "Revised" License
12.91k stars 4.29k forks source link

多线程频繁进行ts pro初始化概率返回EmptyDataError，建议pandas操作文件能够主动释放文件句柄 #1727

Open samuel-lyu opened 10 months ago

samuel-lyu commented 10 months ago

当频繁多线程操作时，存在一定概率消耗完句柄返回异常:pandas.errors.EmptyDataError: No columns to parse from file

  File "/usr/local/lib/python3.10/site-packages/tushare/pro/data_pro.py", line 36, in pro_api
    token = upass.get_token()
  File "/usr/local/lib/python3.10/site-packages/tushare/util/upass.py", line 27, in get_token
    df = pd.read_csv(fp)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 912, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 577, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1407, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1679, in _make_engine
    return mapping[engine](f, **self.options)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 93, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 557, in pandas._libs.parsers.TextReader.__cinit__  File "/usr/local/lib/python3.10/site-packages/tushare/pro/data_pro.py", line 36, in pro_api
    token = upass.get_token()
  File "/usr/local/lib/python3.10/site-packages/tushare/util/upass.py", line 27, in get_token
    df = pd.read_csv(fp)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 912, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 577, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1407, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1679, in _make_engine
    return mapping[engine](f, **self.options)
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 93, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 557, in pandas._libs.parsers.TextReader.__cinit__

建议upass模块中to_csv和read_csv操作建议增加文件with上下文管理主动关闭句柄 例如：

with open(fp, mode='r') as f:
    df = pd.read_csv(p)