search

moodlehq / moodle-mlbackend-python

Moodle machine learning backend
GNU General Public License v3.0
18 stars 19 forks source link

Not working (Moodle 3.9.1, Python 3.7.3, moodlemlbackend 2.4.0 & requirements) #35

Closed klorinczi closed 3 years ago

klorinczi commented 3 years ago

Unable to use Python machine learning backend (moodlemlbackend) v2.4.0 in Moodle v3.9.1.

I already had such problem in MDL-67193 with Moodlemlbackend v1.0.1 in Moodle v3.7.2. It was fixed successfully, but now I faced again and although I compiled tensorflow v1.14 from source, but still I get the error:

I also posted this report to Moodle Tracker (MDL-71523): https://tracker.moodle.org/browse/MDL-71523

python -c 'import tensorflow'
Illegal instruction

I use

Installed with sudo -H python3 -m pip install "moodlemlbackend==2.4.*" 

python3 -m moodlemlbackend.version
2.4.0

I rebuilt tensorflow from source with following build options: -march=nehalem -msse4.1 -msse4.2 -mpclmul -mpopcnt -maes -mno-avx -mno-avx2

But still I get the error:

python -c 'import tensorflow'
Illegal instruction

moodlemlbackend requirements satisfied:

sudo -H python3 -m pip install "moodlemlbackend==2.4.*"
Requirement already satisfied: moodlemlbackend==2.4.* in /usr/local/lib/python3.7/dist-packages (2.4.0)
# Requirement already satisfied: scikit-learn<0.22,>=0.21 in /usr/local/lib/python3.7/dist-packages (from moodlemlbackend==2.4.*) (0.21.3)
# Requirement already satisfied: joblib<0.14,>=0.13.0 in /usr/local/lib/python3.7/dist-packages (from moodlemlbackend==2.4.*) (0.13.2)
# Requirement already satisfied: boto3<1.10,>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from moodlemlbackend==2.4.*) (1.9.253)
# Requirement already satisfied: tensorflow<1.15,>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from moodlemlbackend==2.4.*) (1.14.0)
# Requirement already satisfied: matplotlib<3.1,>=3.0 in /usr/local/lib/python3.7/dist-packages (from moodlemlbackend==2.4.*) (3.0.3)
# Requirement already satisfied: flask<1.1,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from moodlemlbackend==2.4.*) (1.0.4)
# Requirement already satisfied: numpy<1.17,>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from moodlemlbackend==2.4.*) (1.16.6)
# Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn<0.22,>=0.21->moodlemlbackend==2.4.*) (1.6.3)
# Requirement already satisfied: botocore<1.13.0,>=1.12.253 in /usr/local/lib/python3.7/dist-packages (from boto3<1.10,>=1.9.0->moodlemlbackend==2.4.*) (1.12.253)
# Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from boto3<1.10,>=1.9.0->moodlemlbackend==2.4.*) (0.10.0)
# Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /usr/local/lib/python3.7/dist-packages (from boto3<1.10,>=1.9.0->moodlemlbackend==2.4.*) (0.2.1)
# Requirement already satisfied: wheel>=0.26 in /usr/lib/python3/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (0.32.3)
# Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (1.37.0)
# Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (1.1.0)
# Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (1.14.0)
# Requirement already satisfied: six>=1.10.0 in /usr/lib/python3/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (1.12.0)
# Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (1.1.2)
# Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (1.14.0)
# Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (0.8.1)
# Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (3.15.8)
# Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (1.0.8)
# Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (0.4.0)
# Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (0.2.0)
# Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (1.12.1)
# Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (0.12.0)
# Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib<3.1,>=3.0->moodlemlbackend==2.4.*) (2.4.7)
# Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib<3.1,>=3.0->moodlemlbackend==2.4.*) (1.3.1)
# Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib<3.1,>=3.0->moodlemlbackend==2.4.*) (0.10.0)
# Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib<3.1,>=3.0->moodlemlbackend==2.4.*) (2.8.1)
# Requirement already satisfied: Werkzeug>=0.14 in /usr/local/lib/python3.7/dist-packages (from flask<1.1,>=1.0.2->moodlemlbackend==2.4.*) (1.0.1)
# Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.7/dist-packages (from flask<1.1,>=1.0.2->moodlemlbackend==2.4.*) (1.1.0)
# Requirement already satisfied: Jinja2>=2.10 in /usr/local/lib/python3.7/dist-packages (from flask<1.1,>=1.0.2->moodlemlbackend==2.4.*) (2.11.3)
# Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.7/dist-packages (from flask<1.1,>=1.0.2->moodlemlbackend==2.4.*) (7.1.2)
# Requirement already satisfied: urllib3<1.26,>=1.20; python_version >= "3.4" in /usr/lib/python3/dist-packages (from botocore<1.13.0,>=1.12.253->boto3<1.10,>=1.9.0->moodlemlbackend==2.4.*) (1.24.1)
# Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.7/dist-packages (from botocore<1.13.0,>=1.12.253->boto3<1.10,>=1.9.0->moodlemlbackend==2.4.*) (0.15.2)
# Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (56.0.0)
# Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (3.3.4)
# Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from keras-applications>=1.0.6->tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (3.2.1)
# Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2>=2.10->flask<1.1,>=1.0.2->moodlemlbackend==2.4.*) (1.1.1)
# Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (4.0.1)
# Requirement already satisfied: cached-property; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from h5py->keras-applications>=1.0.6->tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (1.5.2)
# Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (3.4.1)
# Requirement already satisfied: typing-extensions>=3.6.4; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow<1.15,>=1.14.0->moodlemlbackend==2.4.*) (3.7.4.3)

Any idea how can I make it work? How could I debug the problem?

douglasbagnall commented 3 years ago

Have you tried building with -march=native?

It seems like you don't really have a nehalem CPU.

douglasbagnall commented 3 years ago

That would be -march=native and none of the other -m flags. The compiler will work it out.

This assumes you are compiling and running it on the same machine.

douglasbagnall commented 3 years ago

Also, BTW, if you use https://github.com/moodlehq/moodle-mlbackend-python/pull/31, you don't need tensorflow 1.14, and can use a supported version, installable via pip (though not apt-get).

klorinczi commented 3 years ago

Finally I was able to solve it by finishing the build from source correctly.

Just for reference for others, finishing install after building from source:

python3 -c 'import tensorflow'
pip3 uninstall tensorflow
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_output
pip3 install /tmp/tensorflow_output/tensorflow-1.14.0-cp37-cp37m-linux_x86_64.whl
python3 -c 'import tensorflow'

Now the install was successful.