many-examples: tensorflow_cpu dependency fail with Python 3.8

[alexcg1]

Related to #425

The version of tensorflow_cpu we specify in requirements.txt doesn't work with Python 3.8. If we tell it to use version 2.4.x (latest version) that fails sometimes because of Jina's other dependencies.

Proposed solution

• Switch to tensorflow_cpu==2.3 in requirements.txt. This ensures dependencies install with both Python 3.7 and 3.8 and on a version of Jina specified in requirements.txt

I'm working on this now. This should fix several examples

[alexcg1]

Update on Pokedex: It only works with Python 3.7 and tensorflow_cpu 2.1.x.

Any bump on tf_cpu version breaks things with FailedPreconditionError() on index and query (Flows close successfully, but don't work)

[alexcg1]

Huh, Pokedex works with Jina 1.2 and tensorflow_cpu 2.4 on Python 3.8. I'll make a PR to fix

[alexcg1]

Here's a sheet with testing results.

If seems whatever I do, I can't get same versions of Jina/Tensorflow_cpu working with both Python 3.7 and 3.8

[rutujasurve94]

we can aim to have the latest2.4.1 tensorflow_cpu compatible with python 3.8, jina

[slettner]

@alexcg1 I faced this issue in this PR as well. I have set the Python version in the CI to 3.7 (was 3.8 before) to fix the problem.

I would suggest that we test all examples with both version we support (3.7 and 3.8, correct me if wrong) in the CI. Do you think we should have this in the future?

[alexcg1]

I certainly think we should have this @slettner :)

But I also think we need to fix the problem at the root. We need to have these examples working with 3.8 (and preferably 3.9) moving forwards. So many of our examples rely on this that when (and it's always a when, not an if) things go wrong (and with tensorflow-cpu they constantly do) everything breaks

[slettner]

Totally agree. Regarding the testing, I have used tox in the past which is good for creating separate environments for different tests (e.g. for different versions/examples)

A small example for tumblr-gif-search could look like this

[tox]
envlist =
  {py37, py38}-tumblr-gif-search
skipsdist = true

[testenv:{py37, py38}-tumblr-gif-search]
deps =
  pytest==6.2.4
  pytest-mock==3.6.1
  -rtumblr-gif-search/requirements.txt
changedir = {toxinidir}/tumblr-gif-search
commands =
  pytest tests

Running tox would create two venv's with 3.7 and 3.8 respectively and execute pytest.

As a first step, we could add this to Github Actions to stay updated on which versions work. What do you think?

[alexcg1]

I really like this! Our example testing has always been less than optimal, so anything would be an improvement :) Tox specifically though looks like a good solution

[alexcg1]

I'm not sure whose call it is to make. @nan-wang who is in charge of our examples repo as a whole (specifically CI/CD)?

[nan-wang]

I'm not sure whose call it is to make. @nan-wang who is in charge of our examples repo as a whole (specifically CI/CD)?

We are using this https://github.com/actions/setup-python for jina repo. I am taking care of the CICD in the example repo since last week. I've seen the problem of tensorflow_cpu and this is due to the fact 2.1.0 has security issues.

[nan-wang]

The point is that the tests are not well written and therefore only fixing CICD can NOT solve the problem. I'm updating all the tests. #512

[nan-wang]

added into CI at #596

[slettner]

All clear! https://github.com/actions/setup-python also does the job quite well. Regarding the tests itself, I made a suggestion here on improving the tests for the examples. What do you think @nan-wang ?

[slettner]

One small advantage of tox is that developer can easily reproduce the functionality locally from the cmd-line (i.e. creating individual venv's and running tests with correct venv's) with only one command. It is nice for debugging the CI and I personally find it more verbose than the CI scripts.