search

kingoflolz / mesh-transformer-jax

Model parallel transformers in JAX and Haiku
Apache License 2.0
6.29k stars 892 forks source link

looking at multiple versions of different packages (slow progress of requirements file) #194

Open rahimikia opened 2 years ago

rahimikia commented 2 years ago

Running 'requirements.txt' for fine-tuning on TPU takes hours for me without any end. It's always looking at multiple versions of different packages (like multidict), and each one takes hours to finish. Any solution for this?

Tatoris commented 2 years ago

This is probably due to pip dependency resolver. try upgrading pip to latest pip install --upgrade pip and run it again.

If that fails, you could try installing packages one by one cat requirements.txt | xargs -n 1 pip install (you may end up with some incompatibilities though)

mkosecki commented 2 years ago

This is the common issue, to limit the time of installation you can set some constraints (>=, ~=) to some of the packages that the installation is blocking, you can check the previous resolver pip install --use-feature=2020-resolver -r requirements.txt

mkosecki commented 2 years ago

Also there are some hints in here https://pip.pypa.io/en/stable/topics/dependency-resolution/#possible-ways-to-reduce-backtracking

safeeazeem commented 2 years ago

I had a similar problem earlier as well. What worked for me was that after cloning the repo I ran pip install git+https://github.com/EleutherAI/lm-evaluation-harness/ then i went to the requirements.txt, removed git+https://github.com/EleutherAI/lm-evaluation-harness/, uvicorn and fastapi

After that I ran pip install -r requirements.txt and everything got installed.