Open pan-cai opened 3 years ago
@pan-cai Hi, How did you trigger this error, or what actions did you do?
使用qlib下载的数据没问题,使用自己csv数据转换成qlib格式的数据后会出现
=========================================================================== csv数据是使用jqdatasdk下载2005-01-01至2021-01-29 日级别的csv格式数据;然后转换成qlib格式的csv数据;然后再将qlib格式的csv数据转换为qlib格式的数据(.bin结尾的)
=========================================================================== qrun.exe 运行类似 Alpha158.yaml 会触发
报错提示是 fetch_df_by_index 函数的,查看了出错时的参数 slc 为 slice('2017-01-01', '2020-08-01', None) idx_slc 为 (slice('2017-01-01', '2020-08-01', None), slice(None, None, None)) 报错 'MultiIndex slicing requires the index to be lexsorted: slicing on levels [0], lexsort depth 0'
=========================================================================== 还有下面的代码也会触发
instruments = ['SH600000']
# fields = ['$close', '$volume', 'Ref($close, 1)', 'Mean($close, 3)', '$high-$low']
fields = ['$close', '$volume', ]
features = D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31', freq='day')
下面是具体的报错
File "xxx/test_data.py", line 39, in <module>
features = D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31', freq='day').head()
File "xxx\lib\site-packages\qlib\data\data.py", line 975, in features
return DatasetD.dataset(instruments, fields, start_time, end_time, freq, disk_cache)
File "xxx\lib\site-packages\qlib\data\cache.py", line 313, in dataset
return self._dataset(instruments, fields, start_time, end_time, freq, disk_cache)
File "xxx\lib\site-packages\qlib\data\cache.py", line 639, in _dataset
with CacheUtils.writer_lock(self.r, "dataset-%s" % _cache_uri):
File "xxxx\anaconda\lib\contextlib.py", line 112, in __enter__
return next(self.gen)
File "xxx\lib\site-packages\qlib\data\cache.py", line 220, in writer_lock
CacheUtils.acquire(current_cache_wlock, lock_name)
File "xxx\lib\site-packages\qlib\data\cache.py", line 184, in acquire
"""
QlibCacheException: It sees the key(lock:xxxxx\\qlib\\qlib_data\\cn_data:dataset-cfebe46424b8064e04218c84ce30751b-wlock) of the redis lock has existed in your redis db now.
You can use the following command to clear your redis keys and rerun your commands:
$ redis-cli
> select 1
> del "lock:xxxxx\\qlib\\qlib_data\\cn_data:dataset-cfebe46424b8064e04218c84ce30751b-wlock"
> quit
If the issue is not resolved, use "keys *" to find if multiple keys exist. If so, try using "flushall" to clear all the keys.
I had the same errors
win10 qlib.version Out[20]: '0.6.2'
I used those command to solve it, bug the problem remains! $ redis-cli
select 1 del "lock:xxxxx\qlib\qlib_data\cn_data:dataset-cfebe46424b8064e04218c84ce30751b-wlock" quit
@hanson-young Could you give more details about your error?
@you-n-g
Windows AMD64 Windows-10-10.0.18362-SP0 10.0.18362
Python version: 3.8.5 (default, Sep 3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
Qlib version: 0.6.3.99 numpy==1.20.3 pandas==1.2.4 scipy==1.5.2 requests==2.24.0 sacred==0.8.2 python-socketio==3.1.2 redis==3.5.3 python-redis-lock==3.7.0 schedule==1.1.0 cvxpy==1.0.21 hyperopt==0.1.1 fire==0.4.0 statsmodels==0.12.0 xlrd==1.2.0 plotly==4.12.0 matplotlib==3.1.3 tables==3.6.1 pyyaml==5.3.1 mlflow==1.17.0 tqdm==4.50.2 loguru==0.5.3 lightgbm==3.2.1 tornado==6.0.4 joblib==0.17.0 fire==0.4.0 ruamel.yaml==0.17.9
[62152:MainThread](2021-06-12 23:54:23,909) INFO - qlib.Initialization - [init.py:48] - qlib successfully initialized based on client settings. [62152:MainThread](2021-06-12 23:54:23,910) INFO - qlib.Initialization - [init.py:49] - data_path=D:\Finance\digital_currency_api\qlib\qlib_data\cn_data [62152:MainThread](2021-06-12 23:54:23,912) ERROR - qlib.workflow - [utils.py:34] - An exception has been raised[QlibCacheException: It sees the key(lock:D:\Finance\digital_currency_api\qlib\qlib_data\cn_data:dataset-2502fd27db971bf723c6523d9e748f4d-wlock) of the redis lock has existed in your redis db now. You can use the following command to clear your redis keys and rerun your commands: $ redis-cli
select 1 del "lock:D:\Finance\digital_currency_api\qlib\qlib_data\cn_data:dataset-2502fd27db971bf723c6523d9e748f4d-wlock" quit If the issue is not resolved, use "keys *" to find if multiple keys exist. If so, try using "flushall" to clear all the keys. ]. File "D:/Finance/digital_currency_api/vnpy-master/hanson/test_qt_main.py", line 27, in
df = D.features(instruments, fields, start_time='2010-01-01', end_time='2013-01-31', freq='day').head() File "D:\Program\Anaconda\lib\site-packages\pyqlib-0.6.3.99-py3.8-win-amd64.egg\qlib\data\data.py", line 981, in features return DatasetD.dataset(instruments, fields, start_time, end_time, freq, disk_cache) File "D:\Program\Anaconda\lib\site-packages\pyqlib-0.6.3.99-py3.8-win-amd64.egg\qlib\data\cache.py", line 376, in dataset return self._dataset(instruments, fields, start_time, end_time, freq, disk_cache) File "D:\Program\Anaconda\lib\site-packages\pyqlib-0.6.3.99-py3.8-win-amd64.egg\qlib\data\cache.py", line 702, in _dataset with CacheUtils.writer_lock(self.r, "dataset-%s" % _cache_uri): File "D:\Program\Anaconda\lib\contextlib.py", line 113, in enter return next(self.gen) File "D:\Program\Anaconda\lib\site-packages\pyqlib-0.6.3.99-py3.8-win-amd64.egg\qlib\data\cache.py", line 283, in writer_lock CacheUtils.acquire(current_cache_wlock, lock_name) File "D:\Program\Anaconda\lib\site-packages\pyqlib-0.6.3.99-py3.8-win-amd64.egg\qlib\data\cache.py", line 239, in acquire raise QlibCacheException(
Hi, @zhupr , can you give any advice on this issue? Thanks
@hanson-young Hi,you can confirm the location of the error code by following these steps:
qlib.init
adds the expression_cache
and dataset_cache
parameters: qlib.init(xxx, expression_cache=None, dataset_cache=None)
.
qlib.init
removes expression_cache=None
and dataset_cache=None
, and then changes the code in the acquire
function in qlib/data/cache.py to:
@staticmethod
def acquire(lock, lock_name):
lock.acquire()
@zhupr
In the step 1,
qlib.init(provider_uri="D:\Finance\digital_currency_api\qlib\qlib_data\cn_data", region=REG_CN,expression_cache=None,dataset_cache=None)
[59156:MainThread](2021-06-14 16:50:39,726) INFO - qlib.Initialization - [config.py:284] - default_conf: client.
[59156:MainThread](2021-06-14 16:50:41,377) INFO - qlib.Initialization - [init.py:48] - qlib successfully initialized based on client settings.
[59156:MainThread](2021-06-14 16:50:41,378) INFO - qlib.Initialization - [init.py:49] - data_path=D:\Finance\digital_currency_api\qlib\qlib_data\cn_data
[22540:MainThread](2021-06-14 16:50:42,584) INFO - qlib.Initialization - [config.py:284] - default_conf: client.
[22540:MainThread](2021-06-14 16:50:44,193) INFO - qlib.Initialization - [init.py:48] - qlib successfully initialized based on client settings.
[22540:MainThread](2021-06-14 16:50:44,193) INFO - qlib.Initialization - [init.py:49] - data_path=D:\Finance\digital_currency_api\qlib\qlib_data\cn_data
[22540:MainThread](2021-06-14 16:50:44,246) ERROR - qlib.workflow - [utils.py:34] - An exception has been raised[RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.].
File "
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
On Windows, the code to get the features
from qlib
should be put under if __name__ == "__main__"
:
import qlib
from qlib.data import D
if __name__ == "__main__":
qlib.init(provider_uri="D:\Finance\digital_currency_api\qlib\qlib_data\cn_data")
df = D.features(D.instruments("all"), ["$close", "$close/Ref($close, 1)-1"])
I solved the above problem,thanks.@zhupr
@hanson-young
Would you mind sending a PR to improve the docs about this Question and become one of the contributors of Qlib? Other users who have similar confusion will be grateful to you :) Thanks
@you-n-g OK, I'd love to contribute pr
e.g. You can improve the docs here https://qlib.readthedocs.io/en/latest/FAQ/FAQ.html
It is also OK if you could find any better place.
On Windows, the code to get the
features
fromqlib
should be put underif __name__ == "__main__"
:import qlib from qlib.data import D if __name__ == "__main__": qlib.init(provider_uri="D:\Finance\digital_currency_api\qlib\qlib_data\cn_data") df = D.features(D.instruments("all"), ["$close", "$close/Ref($close, 1)-1"])
On Mac as well, D.features()
need to be used within a function.
I try
instruments = ['AAPL']
fields = ['$close', '$volume', '$open', '$close', '$high', '$low']
historical_df = D.features(instruments, fields, start_time='2020-01-01', end_time='2022-01-05', freq='day')
it works.
but if I only to get two instruments at the same time:
instruments = ['AAPL', 'ATAI']
fields = ['$close', '$volume', '$open', '$close', '$high', '$low']
historical_df = D.features(instruments, fields, start_time='2020-01-01', end_time='2022-01-05', freq='day')
This would fail and trigger QlibCacheException
But if I put them within main, then it's fine.
Don't know if this is a bug.
@bbbzhai Python has a flaw. When using multi-process, you need to enter from main. When multiple stocks are passed in, multi-process will be called by default.
qlib\data\dataset\utils.py
fetch_df_by_index
return df.loc[ pd.IndexSlice[idx_slc], ]
win10系统, fetch_df_by_index函数中的pd.IndexSlice[idx_slc]出错时, 会提示清理redis的数据;
错误应该和redis没有关系, 此处可能需要兼容一下此处异常。