TFX 1.15 docker image contains conflicting dependencies

[IzakMaraisTAL]

System information

• Have I specified the code to reproduce the issue: Yes
• Environment in which the code is executed: Linux
• TFX Version: 1.15.0 and 1.15.1
• Python version: 3.10.14

Describe the current behavior

The tfx docker image contains conflicting dependencies: apache-beam and google-cloud-datastore.

docker run --rm --entrypoint python tensorflow/tfx:1.15.1 -m pip list | grep -E 'apache|google-cloud-datastore'

apache-beam                              2.56.0
google-cloud-datastore                   1.15.5

From the apache-beam dependency constraints we can see that version 2.56.0 requires google-cloud-datastore>=2.0.0,<3. Having google-cloud-datastore 1.15.5 in the base image violates this constraint and may cause bugs.

Describe the expected behavior

The docker TFX image should contain no conflicting dependencies.

Name of your Organization (Optional)

Takealot.com

Other info / logs

We use the TFX docker image as a base image into which we install additional packages. For this to work, we must not install conflicting dependencies. The solution is the use a dependency solver (like pip-tools pip-compile or uv pip compile) to constrain the dependencies we install against those already in the image. However, if the dependencies already in the image are conflicting, it is impossible to install additional packages reliably.

This was not a problem in TFX 1.14

[IzakMaraisTAL]

Using a dependency solver suggests you need to update to google-cloud-datastore==2.19.0 and google-api-core==2.19.1

[lego0901]

Hi @IzakMaraisTAL, sorry for taking this issue too late.

I also tried to build the docker image from my environment, but the actual issue arose from the old Python package cloud-tpu-client from the base image "gcr.io/deeplearning-platform-release/tf2-gpu.2-15.py310"

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 1.12.11 which is incompatible.

I personally thought it is not a good idea to lower the google-api-python-client or the other things' versions. I will ask TF team to know if cloud-tpu-client (or the other hindering packages) is needed for the base docker image. Thanks!