search

moodlehq / moodle-mlbackend-python

Moodle machine learning backend
GNU General Public License v3.0
18 stars 19 forks source link

Could not find a version that satisfies the requirement tensorflow<2.5,>=2.4.2 (from moodlemlbackend==2.6.4) (from versions: 1.13.1, 1.13.2, 1.14.0) #50

Closed klorinczi closed 2 years ago

klorinczi commented 2 years ago

I try to install moodlemlbackend v2.6.4 for Moodle v3.9.1, but I get this error:

pip3 install moodlemlbackend==2.6.4
Collecting moodlemlbackend==2.6.4
  Downloading https://files.pythonhosted.org/packages/aa/f7/ffefab724bab3631403a3280acfa6c47695c15fe4c621b4be9ab8e41a39a/moodlemlbackend-2.6.4-py2.py3-none-any.whl
Collecting joblib<0.14,>=0.13.0 (from moodlemlbackend==2.6.4)
  Downloading https://files.pythonhosted.org/packages/cd/c1/50a758e8247561e58cb87305b1e90b171b8c767b15b12a1734001f41d356/joblib-0.13.2-py2.py3-none-any.whl (278kB)
    100% |████████████████████████████████| 286kB 425kB/s
Collecting numpy<1.20,>=1.19.2 (from moodlemlbackend==2.6.4)
  Downloading https://files.pythonhosted.org/packages/b1/e1/8c4c5632adaffc18dba4e03e97458dc1cb00583811e6982fc620b9d88515/numpy-1.19.5-cp37-cp37m-manylinux1_x86_64.whl (13.4MB)
    100% |████████████████████████████████| 13.4MB 20kB/s
Collecting matplotlib<3.1,>=3.0 (from moodlemlbackend==2.6.4)
  Downloading https://files.pythonhosted.org/packages/83/2a/e47bbd9396af32376863a426baed62d9bf3091f81defd1fe81c5f33b11a3/matplotlib-3.0.3-cp37-cp37m-manylinux1_x86_64.whl (13.0MB)
    100% |████████████████████████████████| 13.0MB 22kB/s
Collecting tensorflow<2.5,>=2.4.2 (from moodlemlbackend==2.6.4)
  Could not find a version that satisfies the requirement tensorflow<2.5,>=2.4.2 (from moodlemlbackend==2.6.4) (from versions: 1.13.1, 1.13.2, 1.14.0)
No matching distribution found for tensorflow<2.5,>=2.4.2 (from moodlemlbackend==2.6.4)
ilyatregubov commented 2 years ago

Hi @klorinczi ,

Which tensorflow version you have? So mlbackend 2.6.4 requires tensorflow 2.4.2 installed. So you might try to install required version of tensorflow with pip and then install mlbackend

klorinczi commented 2 years ago

Finally I was able to install mlbackend 2.6.4. But now I get error "Illegal instruction":

python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
Illegal instruction
douglasbagnall commented 2 years ago 
python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
Illegal instruction

That looks like a packaging problem with tensorflow that we can't do anything about. It has been compiled for a different sub-architecture than you have, but advertised by the packaging system as the one you need.

You'd need to take it up with whoever packages tensorflow.

klorinczi commented 2 years ago

With another install I got more descriptive error:

pip3 install tensorflow-cpu
pip3 install tensorflow-cpu --upgrade
python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
# The TensorFlow library was compiled to use AVX instructions, but these aren't available on your machine.
# Aborted

I installed moodlemlbackend v3.0.2 (I use Moodle v3.9). pip3 install moodlemlbackend==3.0.2

Now I try to meet the requirements: https://github.com/moodlehq/moodle-mlbackend-python/blob/3.0.2/requirements.txt Requires: tensorflow>=2.4.2,<2.5 numpy>=1.19.2,<1.20

I had to recompile tensorflow v2.4.4 with noavx option. Used options: -march=nehalem -msse4.1 -msse4.2 -mpclmul -mpopcnt -maes -mno-avx -mno-avx2

pip3 install /git/tensorflow/tensorflow_output/tensorflow-2.4.4-cp37-cp37m-linux_x86_64.whl

Also installed the required numpy v1.19.5: pip3 install numpy==1.19.5

Now I get the following error:

python3 -c 'import tensorflow'
# RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
# RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
# ImportError: numpy.core._multiarray_umath failed to import
# ImportError: numpy.core.umath failed to import
# 2022-03-25 17:43:52.642579: F tensorflow/python/lib/core/bfloat16.cc:714] Check failed: PyBfloat16_Type.tp_base != nullptr
# Aborted

Any idea how to get further?

douglasbagnall commented 2 years ago

It looks like when you compiled tensorflow it found numpy headers for some version greater than 1.19 and used those.

What do you see if you run

python3 -c 'import numpy;print(numpy.__version__)'

in the environment where you compiled tensorflow?

klorinczi commented 2 years ago

I have the following versions:

python3 -c 'import numpy;print(numpy.__version__)'
1.19.5
python3 -c 'import tensorflow;print(numpy.__version__)'
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
ImportError: numpy.core._multiarray_umath failed to import
ImportError: numpy.core.umath failed to import
2022-03-26 07:51:50.228819: F tensorflow/python/lib/core/bfloat16.cc:714] Check failed: PyBfloat16_Type.tp_base != nullptr
Aborted
pip3 list | grep -e tensorflow -e numpy
numpy                        1.19.5
tensorflow                   2.4.4
tensorflow-estimator         2.4.0
tensorflow-io-gcs-filesystem 0.24.0

EDIT: Maybe the problem is the following:

sudo pip3 list | grep -e tensorflow -e numpy
numpy                        1.21.5
tensorflow-estimator         2.4.0
tensorflow-io-gcs-filesystem 0.24.0

Because I compiled with sudo: date && sudo bazel build --disk_cache ./cache //tensorflow/tools/pip_package:build_pip_package && date

I will try to compile again without sudo: date && bazel build --disk_cache ./cache //tensorflow/tools/pip_package:build_pip_package && date

klorinczi commented 2 years ago

Ok, I think now the build was successful.

python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
2022-03-27 21:28:39.453575: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
tf.Tensor(288.99573, shape=(), dtype=float32)

But still on admin/settings.php?section=analyticssettings page, does not allow to switch to Python.

/usr/bin/python3.7 -m moodlemlbackend.version
3.0.2

Is moodlemlbackend 3.0.2 compatible with Moodle 3.9?

danmarsden commented 2 years ago

do you have pathtopython set under Administration > Server > System paths in Moodle?

klorinczi commented 2 years ago

Yes, I have: /usr/bin/python3.7

danmarsden commented 2 years ago

turn moodle debugging on with display errors - what happens when you try to save the analyticssettings page? - does it show any errors?

klorinczi commented 2 years ago

[Thu Apr 07 07:57:12.780070 2022] [php7:notice] [pid 18604] [client 113.221.115.32:34546] PHP Notice: String [courseurl,core_hub] is deprecated. Either you should no longer be using that string, or the string has been incorrectly deprecated, in which case you should report this as a bug. Please refer to https://docs.moodle.org/dev/String_deprecation line 394 of /lib/classes/string_manager_standard.php: call to debugging() line 7387 of /lib/moodlelib.php: call to core_string_manager_standard->get_string() line 55 of /mod/booking/settings.php: call to get_string() line 89 of /lib/classes/plugininfo/mod.php: call to include() line 47 of /admin/settings/plugins.php: call to core\plugininfo\mod->load_settings() line 8564 of /lib/adminlib.php: call to require() line 19 of /admin/settings.php: call to admin_get_root() in /www/moodle/htdocs/lib/weblib.php on line 3244, referer: https://mysite.com/admin/settings.php?section=analyticssettings

[Thu Apr 07 07:57:17.169758 2022] [php7:notice] [pid 18604] [client 113.221.115.32:34546] PHP Notice: String [courseurl,core_hub] is deprecated. Either you should no longer be using that string, or the string has been incorrectly deprecated, in which case you should report this as a bug. Please refer to https://docs.moodle.org/dev/String_deprecation line 394 of /lib/classes/string_manager_standard.php: call to debugging() line 7387 of /lib/moodlelib.php: call to core_string_manager_standard->get_string() line 55 of /mod/booking/settings.php: call to get_string() line 89 of /lib/classes/plugininfo/mod.php: call to include() line 47 of /admin/settings/plugins.php: call to core\plugininfo\mod->load_settings() line 8564 of /lib/adminlib.php: call to require() line 8692 of /lib/adminlib.php: call to admin_get_root() line 44 of /admin/settings.php: call to admin_write_settings() in /www/moodle/htdocs/lib/weblib.php on line 3244, referer: https://mysite.com/admin/settings.php?section=analyticssettings

klorinczi commented 2 years ago

Do I need to install moodlemlbackend for www-data user with sudo? sudo -u www-data pip3 install moodlemlbackend==3.0.2

Is moodlemlbackend v3.0.2 compatible with Moodle v3.9.1?

klorinczi commented 2 years ago

I did also read a notice: If we wanted the latest tensorflow we would have to pip install -U pip, but moodle-mlbackend-python works with the older 1.14.

Is it still valid? Should I use tensorflow v1.14 with moodlemlbackend v3.0.2?

ilyatregubov commented 2 years ago

Hi @klorinczi

I have updated documentation. You should use 2.6.5 version for Moodle 3.9. the minimum tensorflow version is 2.4.2

https://docs.moodle.org/39/en/Analytics_settings#Versions

So 3.x version of mlbackend is for Moodle 4.0 onwards. And for 4.0 you should use 3.0.4. I will clean up to remove 3.0.2

klorinczi commented 2 years ago

Hi @ilyatregubov,

The https://docs.moodle.org/39/en/Analytics_settings#Versions documentation says: Moodle 3.9.11 and up uses package 2.6.* (minimum 2.6.5)

Moodle 3.9.1 is < than Moodle 3.9.11, so I should use moodlemlbackend 2.3.1>=2.6.4 ?

So if I use Moodle v3.9.1, do I need to use maximum moodlemlbackend v2.6.4 ?

klorinczi commented 2 years ago

pip3 install moodlemlbackend==2.6.5

python3 -c 'import tensorflow' Result: none, so import was successful

On /admin/settings.php?section=analyticssettings I set "analytics | predictionsprocessor" to Python.

I get the following error:

'The selected predictions processor is not ready: The moodlemlbackend Python package is not installed or there is a problem with it. Please execute "/usr/bin/python3.7 -m moodlemlbackend.version" from command line interface for more info.'

Executed from command line by my local Linux user: /usr/bin/python3.7 -m moodlemlbackend.version Result: 2.6.5

klorinczi commented 2 years ago

Following install helped (as described on https://docs.moodle.org/39/en/Analytics_settings#Versions): sudo -H pip3 install moodlemlbackend==2.6.5

On /admin/settings.php?section=analyticssettings I set "analytics | predictionsprocessor" to Python:

I got result: 'Changes saved'

This is great, but I get error to the following line:

sudo -H python3 -c 'import tensorflow'
Illegal instruction
sudo -H pip3 list | grep -e tensorflow -e numpy
numpy                        1.19.5
tensorflow                   2.4.4
tensorflow-estimator         2.4.0
tensorflow-io-gcs-filesystem 0.24.0
sudo -H pip3 uninstall tensorflow
...
Successfully uninstalled tensorflow-2.4.4
sudo -H pip3 install /iscsi_v39/git/tensorflow/tensorflow_output/tensorflow-2.4.4-cp37-cp37m-linux_x86_64.whl
...
Installing collected packages: tensorflow
Successfully installed tensorflow-2.4.4
sudo -H python3 -c 'import tensorflow'
Result nothing, so it means: SUCCESS

SUCCESS!

Thanks for redirecting me to the right doc part: https://docs.moodle.org/39/en/Analytics_settings#Versions

ilyatregubov commented 2 years ago

Hola @klorinczi ,

So you managed to install it and it works? Let me know if there are any problems or I should close the issue

Thanks, Ilya

klorinczi commented 2 years ago

Yes, it seems moodlemlbackend now works fine, the models are analyzed successfully.

Closing the issue.

Thank you for the help!

ilyatregubov commented 2 years ago

Awesome! Thanks!