Installation with anaconda failed with error Cannot uninstall 'ruamel-yaml'.

[arisliang]

🐛 Bug Description

When installing with anaconda, it fails with error:

Attempting uninstall: ruamel.yaml
    Found existing installation: ruamel-yaml 0.15.87
ERROR: Cannot uninstall 'ruamel-yaml'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

To Reproduce

Steps to reproduce the behavior:

1. clone conda environment from base: conda create --name qlibtest --clone base
2. pip install pyqlib

Expected Behavior

qlib installation successfully.

Screenshot

Environment

Note: User could run cd scripts && python collect_info.py all under project directory to get system information and paste them here directly.

• Qlib version: 0.6.1
• Python version: 3.7
• Conda 4.9.2
• OS (Windows, Linux, MacOS): Windows 10
• Commit number (optional, please provide it if you are using the dev version): from pip

Additional Notes

[Derek-Wds]

Hi, this is an error caused by the pip and ruamel-yaml you have already installed on your machine. You can either try with pip install pyqlib --ignore-installed ruamel.yaml or remove the installed files of ruamrl.yaml under your conda environment rm -rf /path/to/anaconda3/lib/python3.7/site-packages/ruamel*.

Hope it helps!

[arisliang]

pip install pyqlib --ignore-installed ruamel.yaml gives a different error:

ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'D:\\ProgramData\\Anaconda3\\envs\\qlibtest\\Lib\\site-packages\\_cffi_backend.cp38-win_amd64.pyd'
Consider using the `--user` option or check the permissions.

After removing ruamel folder, it still gives same error Cannot uninstall 'ruamel-yaml', probably somewhere else still has the registry of the package, since it's directly deleting folder, rather than using conda command to remove the package.

I managed to install from source by changing ruamel.yaml>=0.16.12 to ruamel.yaml. Is there any particular reason qlib needs a higher version of this ruamel package?

[Derek-Wds]

Thanks for letting me know!

Well, the ruamel-yaml package is used to read YAML files for Qlib's benchmark workflows. I haven't tried with lower version of ruamel-yaml when running models with the qrun command.

I will look into that and see if lower version of ruamel-yaml works well with the workflows.

[doudoujay]

got the same problem here. The only workaround is to ignore this ruamel.yaml?

[BigW]

Hi checking Anaconda Navigator shows multiple Yaml versions - is that expected? What version do you recommend I keep if only one is permitted? I am going to try and set up a new environment for QLIB and re-try to install.

[BigW]

Hi - I have attached my error logs for this as I have tried a couple of times to resolve this to get QLIB installed and running on Linux as advised. While Yaml looks likely to be the cause its not clear if there are other issues. Probably a result of me trying to install multiple times or doing something wrong? I greatly appreciate any help on this as it looks like a library with many valuable attributes that are hard to find in one package. qlib_error_log.txt

[BigW]

Hi - I have attached my error logs for this as I have tried a couple of times to resolve this to get QLIB installed and running on Linux as advised. While Yaml looks likely to be the cause its not clear if there are other issues. Probably a result of me trying to install multiple times or doing something wrong? I greatly appreciate any help on this as it looks like a library with many valuable attributes that are hard to find in one package. qlib_error_log.txt

Installing https://linuxconfig.org/how-to-install-g-the-c-compiler-on-ubuntu-20-04-lts-focal-fossa-linux seems to have solved the install issue ie installing G++ the C++ compiler on Ubuntu - not sure if that impacts all Ubuntu users or only my setup but worth checking for other users and including as a check in the install notes if thats the case.

[BigW]

Hi - after a successful install I ran the first example as per https://github.com/microsoft/qlib#auto-quant-research-workflow for the LighGBM example I get the following error. Does that show I have an old or outdated numpy file? ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject.

For any other newcomers to Qlib this error is easy to fix - no idea why this is the case but a simple numy uninstall and install fixes this problem. https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp

(QLIB3TEST) warren@warren-VivoBook-ASUSLaptop-X512DA-X512DA:~/qlib/examples$ qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

Traceback (most recent call last): File "/home/warren/anaconda3/envs/QLIB3TEST/bin/qrun", line 5, in from qlib.workflow.cli import run File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/workflow/cli.py", line 12, in from qlib.model.trainer import task_train File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/model/init.py", line 6, in from .base import Model File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/model/base.py", line 5, in from ..data.dataset import Dataset File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/data/init.py", line 8, in from .data import ( File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/data/data.py", line 20, in from .cache import H File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/data/cache.py", line 36, in from .ops import Operators File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/data/ops.py", line 19, in from ._libs.rolling import rolling_slope, rolling_rsquare, rolling_resi File "qlib/data/_libs/rolling.pyx", line 1, in init qlib.data._libs.rolling ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject (QLIB3TEST) warren@warren-VivoBook-ASUSLaptop-X512DA-X512DA:~/qlib/examples$

Any help resolving that would be greatly appreciated thank you as close to being able to run a test using Qlib!

[Derek-Wds]

Hi @BigW , this error occurs after numpy released a new version recently. Installing the latest numpy may solve this problem. By the way, could you please tell us how your install your qlib? We want to see if there are possible solutions to avoid such errors to occur in the future. Thanks!

[BigW]

@Derek-Wds no problem at all - I was going to recommend adding my own checklist to the readme file to help other users. My install is on Ubuntu and was in three steps

1 Install Anaconda and via Anaconda Navigator install the following packages and version numbers in a new environment ie do not use the default base environment but set up a new one specifically for qlib:

alembic 1.41
argon2-cffi 20.1.0`
async-generator 1.10
attrs 20.3.0
backcall 0.2.0
blas 1.0
bleach 3.3.0
blis 0.7.4
ca-certificates 2021.1.19
certifi 2020.12.5
chardet 4.0.0
click 7.1.2
cloudpickle 1.6.0
colorama 0.4.4
cvxpy 1.0.21
cycler 0.10.0
cython 0.29.22
cython-blis 0.7.4
databricks-cli 0.14.2
decorator 4.4.2
defusedxml 0.7.1
dill 0.3.3
docker 4.4.4
docopt 0.6.2
ecos 2.0.7
entrypoints 0.3
fastcache 1.1.0
fire 0.4.0
flask 1.1.2
future 0.18.2
gitdb 4.0.5
gitpython 3.1.14
greenlet 1.0.0
gunicorn 20.0.4
hyperopt 0.1.1
idna 2.10
importlib-metadata 3.7.3
intel-openmp 2020.2
ipykernel 5.3.4
ipython 7.21.0
ipython-genutils 0.2.0
itsdangerous 1.1.0
jedi 0.17.0
jinja2 2.11.3
joblib 1.0.1
jsonpickle 1.5.2
jsonschema 3.2.0
jupyter-client 6.1.7
jupyter_core 4.7.1
jupyterlab_pygments 0.1.2
kiwisolver 1.3
ld_impl_linux-64 2.33.1
libffi 3.3
libgcc-ng 9.1.0
libgfortran-ng 7.3.0
libsodium 1.0.18
libstdcxx-ng 9.1.0
lightgbm 3.1.1
mako 1.1.4
markupsafe 1.1.1
matplotlib 3.1.3
mistune 0.8.4
mkl 2020.2
mkl-fft 1.3.0
mkl-random 1.1.1
mkl-service 2.3.0
mklflow 1.14.1
multiprocess 0.70.11.1
munch 2.5.0
nbclient 0.5.3
nbconvert 6.0.7
nbformat 5.1.2
ncurses 6.2
nest-asyncio 1.5.1
networkx 2.5
notebook 6.2.0
numexpr 2.7.3
numpy 1.18.5
numpy-base 1.18.5
numpy-devel 1.18.5
openssl 1.1.1
osqp 0.6.2
packaging 20.9
pandas 1.2.3
pandoc 2.11
pandocfilters 1.4.3
parso 0.8.1
patsy 0.5.1
pexpect 4.8.0
pickleshare 0.7.5
pip 21.0.1
plotly 4.12.0
prometheus-client 0.9.0
promethius-flask-exporter 0.18.1
prompt-toolkit 3.0.8
protobuf 3.15.6
ptyprocess 0.7.0
py-cpuinfo 7.0.0
pycparser 2.20
pygments 2.8.1
pymongo 3.11.3
pyparsing 2.4.7
pyqlib 0.6.2
pyrsistent 0.17.3
python 3.8.8
python-dateutil 2.8.1
python-editor 1.0.4
python-engineio 4.0.1
python-redis-lock 3.7.0
python-socketio 3.1.2
pytz 2021.1
pyyaml 5.4.1
pyzmq 20.0.0
qdldl 0.1.5
querystring-parser 1.2.4
readline 8.1
redis 3.5.3
requests 2.25.1
retrying 1.3.3
ruamel.yaml 0.16.13
ruamel.yaml.clib 0.2.2
sacred 0.8.2
schedule 1.0.0
scikit-learn 0.24.1
scipy 1.6.1
scs 2.1.2
send2trash 1.5.0
setuptools 52.0.0
six 1.15.0
smmap 3.0.5
sqlalchemy 1.4.0
sqlite 3.35.0
sqlparse 0.4.1
statsmodels 0.12.2
tables 3.6.1
tabulate 0.8.9
termcolor 1.1.0
terminado 0.9.3
testpath 0.4.4
threadpoolctl 2.1.0
tk 8.6.10
tornado 6.1
tqdm 4.59.0
traitlets 5.0.5
urlib3 1.26.4
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 0.58.0
werkzeug 1.0.1
wheel 0.36.2
wrapt 1.12.1
xlrd 2.0.1
xz 5.2.5
zeromq 4.3.3
zipp 3.4.0
zlib 1.2.11

2 Install G++ the C++ compiler - important for users running this on Linux/Ubuntu as its not included in the install and needed.

3 Uninstall and reinstall numpy for the reasons you mentioned.

[atnjqt]

I ran into this trying to install jupyterhub on Ubuntu 20LTS... For some reason even just trying to pip install --upgrade ruamel.yaml failed given this is a base module I think.

Including the --ignore-installed flag helped me at least upgrade ruamel.yaml to then successfully run Jupyterhub

sudo /usr/local/anaconda3/bin/python3 -m pip install --ignore-installed ruamel.yaml