Cannot set up capreolus on Compute Canada

[edanerg]

I followed the setup-cc.md in the docs.

Errors I get while running

pip install tf-models-official==2.5 tensorflow-ranking==0.4.2

ERROR: tensorflow 2.6.0+computecanada has requirement numpy~=1.19.2, but you'll have numpy 1.21.0+computecanada which is incompatible.
ERROR: tensorflow 2.6.0+computecanada has requirement six~=1.15.0, but you'll have six 1.16.0 which is incompatible.
ERROR: tensorflow-metadata 1.2.0 has requirement absl-py<0.13,>=0.9, but you'll have absl-py 0.15.0 which is incompatible.

Errors I get while running:

cat requirements.txt | cut -d '#' -f 1  | grep "\S" | xargs -n 1 -i sh -c 'pip install --no-index {} || pip install {}'

ERROR: Could not find a version that satisfies the requirement profane (from versions: none)
ERROR: No matching distribution found for profane

Error: pg_config executable not found.

ERROR: Could not find a version that satisfies the requirement psycopg2-binary (from versions: none)
ERROR: No matching distribution found for psycopg2-binary

   PermissionError: [Errno 13] Permission denied: '/tmp/magnitude.install'

ERROR: Could not find a version that satisfies the requirement pymagnitude (from versions: none)
ERROR: No matching distribution found for pymagnitude

---

I do have miniconda installed, although I don't think it is used in the script.

What have I tried:

module load postgresql/12.4
TMPDIR=~/scratch
pip install --upgrade pip

This resolves the pg_config error, the tmp directory error and make pip use smart dependency resolver.

However I still get

ERROR: Could not find a version that satisfies the requirement profane (from versions: none)
ERROR: No matching distribution found for profane
ERROR: Could not find a version that satisfies the requirement psycopg2-binary (from versions: none)
ERROR: No matching distribution found for psycopg2-binary
ERROR: Could not find a version that satisfies the requirement pyjnius (from versions: none)
ERROR: No matching distribution found for pyjnius
ERROR: Could not find a version that satisfies the requirement pymagnitude (from versions: none)
ERROR: No matching distribution found for pymagnitude

when running

cat requirements.txt | cut -d '#' -f 1  | grep "\S" | xargs -n 1 -i sh -c 'pip install --no-index {} || pip install {}'

[nimasadri11]

@AlexWang000

1. Are you running on cedar?
2. Are u using a virtual env?

[edanerg]

1. Yes.
2. Yes.

[nimasadri11]

@AlexWang000 I suggest starting a new environment. In particular, you can try deleting your old env, and then running these:

module load java/11
module load python/3.7
module load scipy-stack

ENVDIR=$HOME/venv/capreolus-env
virtualenv --no-download $ENVDIR
source $ENVDIR/bin/activate

pip install tf-models-official==2.5 tensorflow-ranking==0.4.2
cat requirements.txt | cut -d '#' -f 1  | grep "\S" | xargs -n 1 -i sh -c 'pip install --no-index {} || pip install {}'
pip install --no-index torch==1.9.0 spacy==2.2.2

Note that if you get disconnected from the instance you need to run the module/source commands before proceeding, to ensure dependencies are being installed in the correct env and not the system env.

[edanerg]

@nimasadri11 I have tried this script several times, each time using a new env. I was also using module command every time. I get the same errors.

[crystina-z]

@AlexWang000 hi thanks for reporting this, the problem could probably solved by installing that perticular package using TMPDIR=~/scratch/tmp pip install pymagnitude --no-cache-dir

The reason behind that is probably someone has install the pymagnitude before in the machine, and the /tmp/pymagnitude.. was never deleted and you are not permitted to change other's file.

[edanerg]

Hey, @crystina-z @nimasadri11, thanks for the help

I was able to resolve the issue using:

module load java/11
module load python/3.7
module load scipy-stack

ENVDIR=$HOME/scratch/venv/capreolus-env
TMPDIR=$HOME/scratch/tmp
virtualenv --no-download $ENVDIR
source $ENVDIR/bin/activate

pip install tf-models-official==2.5 --no-cache-dir
pip install tensorflow-ranking==0.4.2 --no-cache-dir
cat requirements.txt | cut -d '#' -f 1  | grep "\S" | xargs -n 1 -i sh -c 'pip install --no-index {} || pip install {}'
pip install --no-index torch==1.9.0 spacy==2.2.2

It sill gives me similar errors (I guess some of them are from "pip install --no-index {}"), but now I can print the config and fine tune monobert.

[crystina-z]

@AlexWang000 great to hear that! I'll close the issue for now as the problem seems to be solved? feel free to reopen if it gets back later