Investigate upgrade to Python 3.10 and Tensorflow 2.9

[douglascamata]

Describe the bug

@jaakkopasanen reported:

Better, but still consumes all memory even when using only 2 threads. The process consumes less than 1 GB of memory with one thread so looks like it's not a problems of N threads consuming N times the memory. I won't revert this though, since it works as before with the default value.

I'm on Windows 10 and tested using Python 3.10 and Tensorflow 2.9 and 2.8. However, when I switched to Python 3.8 and Tensorflow 2.3 everything went just smoothly, even with --thread_count=max. I then tested with Python 3.8 and Tensorflow 2.9 and the process failed again. Testing different versions of Tensorflow revealed that the maximum version which still works is 2.3.

Python 3.9 and 3.10 don't support Tensorflow 2.3 and I'd prefer not to stay with old versions of Python and Tensorflow forever, especially since Tensorflow 2.3 doesn't install on the ARM Macs.

To Reproduce

python autoeq.py --input_dir="measurements/oratory1990/data/inear" --output_dir="my_results/oratory1990" --compensation="compensation/harman_in-ear_2019v2.csv" --equalize --parametric_eq --max_filters=10 --thread_count 1

Expected behavior

Things work smoothly.

[douglascamata]

I cannot reproduce the high memory usage using a Windows 11 machine, running WSL2, with a total of 16gb of memory (although WSL2 can only access close to 8gb), and using --thread_count 2. I tried:

• Python 3.8.12 and Tensorflow 3.9.1 (average memory usage was stable around 5.4 to 5.9% per thread)
• Python 3.10.5 and Tensorflow 3.9.1 (average memory usage was stable around 6.4 to 6.9% per thread)

Here's the version of all the dependencies that I've got:

❯ pip freeze
absl-py==1.2.0
astunparse==1.6.3
cachetools==5.2.0
certifi==2022.6.15
cffi==1.15.1
charset-normalizer==2.1.0
cycler==0.11.0
filprofiler==2022.7.1
flatbuffers==1.12
fonttools==4.34.4
gast==0.4.0
google-auth==2.9.1
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.48.0
h5py==3.7.0
idna==3.3
keras==2.9.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.4
libclang==14.0.1
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib==3.5.2
numpy==1.23.1
oauthlib==3.2.0
opt-einsum==3.3.0
packaging==21.3
pandas==1.4.3
Pillow==9.2.0
protobuf==3.19.4
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.1
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.9
scipy==1.8.1
six==1.16.0
SoundFile==0.10.3.post1
tabulate==0.8.10
tensorboard==2.9.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.9.1
tensorflow-estimator==2.9.0
tensorflow-io-gcs-filesystem==0.26.0
termcolor==1.1.0
threadpoolctl==3.1.0
tqdm==4.64.0
typing_extensions==4.3.0
urllib3==1.26.11
Werkzeug==2.2.1
wrapt==1.14.1

@jaakkopasanen can you run pip freeze and paste the output, please?

[douglascamata]

I did a small fix to a dependency that was missing from requirements.txt in #495, but that's completely unrelated.

[douglascamata]

You could be having some conflicts with the dependencies, @jaakkopasanen. I would recommend to reinstall the dependencies with pip install --upgrade -r requirements.txt. If you are using Python's venv, destroy it and reinstall the dependencies from scratch.

[jaakkopasanen]

I did create a fresh virtualenv.

[jaakkopasanen]

Should be all good now as Tensorflow is no longer a dependency. Please reopen is the problem still persists.