Release new version that's compatible with `tfx[kfp]==1.10.0`

[davidxia]

Currently master branch has Depends on tensorflow>=1.15.5,<2 or tensorflow>=2.10,<2.11 from this commit. But this isn't released yet.

Here's a minimal repro of the dependency conflicts with tfx[kfp]==1.10.0.

❯ cat requirements.in
tfx[kfp]==1.10.0
tensorflow-ranking

❯ pip-compile requirements.in
Could not find a version that matches tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,<3,>=1.15.5,>=2.10.0 (from tfx[kfp]==1.10.0->-r requirements.in (line 1))
Tried: 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.9.0, 2.9.1, 2.9.2, 2.10.0
Skipped pre-versions: 2.6.0rc0, 2.6.0rc1, 2.6.0rc2, 2.7.0rc0, 2.7.0rc1, 2.8.0rc0, 2.8.0rc1, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.11.0rc0, 2.11.0rc1
There are incompatible versions in the resolved dependencies:
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tfx-bsl==1.10.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tensorflow-model-analysis==0.41.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5 (from tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5 (from tensorflow-transform==1.10.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tensorflow-data-validation==1.10.0->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow<3,>=2.10.0 (from tensorflow-serving-api==2.10.0->tfx[kfp]==1.10.0->-r requirements.in (line 1))

Any rough timeline of when this will be released? Thanks!

[singhniraj08]

@davidxia,

What version of Transform you are using? Can you please try with latest release 1.10.1 and see if the dependency conflicts are resolved. TFX latest release 1.10.0 supports Transform version 1.10.1. Please find dependency updates in TFX 1.10.0 release note.

Thank you!

[davidxia]

Thanks for quick response.

What version of Transform you are using?

You can see from my snippet above that I'm using tensorflow-transform==1.10.1 which is pulling in tensorflow ...<2.10.

The commit history for setup.py has the commit "Upgrades TF dependency to 2.10" which allows tensorflow 2.10 on this line. But this commit doesn't seem to be in any release tag. Github shows only master here.

Moreover, 1.10.1 was published to PyPI on Aug 29, but this commit was created on Sep 10. 🤔 So it makes sense that I'm seeing this behavior since it seems like the change isn't actually in the 1.10.1 release.

[singhniraj08]

@davidxia,

I tried installing tensorflow_transform nightly versionand the dependency conflict still occurs with tensorflow 2.10.

When we pip install tensorflow_transform, it automatically installs tensorflow 2.9.2. While the team works on this issue, the workaround is to use tensorflow 2.9.2 with tensorflow_transform 1.10.1.

Thank you!

[davidxia]

Thanks, I should clarify that I'm able to repro dep conflicts with just the following.

cat requirements.in
tfx[kfp]==1.10.0

pip-compile requirements.in

Could not find a version that matches tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,<3,>=1.15.5,>=2.10.0 (from tfx[kfp]==1.10.0->-r requirements.in (line 1))
Tried: 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.9.0, 2.9.1, 2.9.2, 2.10.0
Skipped pre-versions: 2.6.0rc0, 2.6.0rc1, 2.6.0rc2, 2.7.0rc0, 2.7.0rc1, 2.8.0rc0, 2.8.0rc1, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.11.0rc0, 2.11.0rc1
There are incompatible versions in the resolved dependencies:
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tfx-bsl==1.10.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5 (from tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tensorflow-data-validation==1.10.0->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tensorflow-model-analysis==0.41.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow<3,>=2.10.0 (from tensorflow-serving-api==2.10.0->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5 (from tensorflow-transform==1.10.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))

The conflict is specifically from pip-compile thinking tensorflow has to be both <2.10 and >=2.10.0. The <2.10 restriction is from tensorflow-transform==1.10.1->tfx[kfp]==1.10.0 (hence this issue) and interestingly also from tfx[kfp]==1.10.0 itself (corresponding issue).

More curious is that despite pip-compile failing, pip install -r requirements.in works with resulting packages.

tensorflow==2.9.2
tensorflow-serving-api==2.9.2
tensorflow-transform==1.10.1

[rcrowe-google]

Adding @1025KB Please keep in mind that the master branch is work in progress, and is not guaranteed to work with the released version.

[singhniraj08]

@davidxia,

TFT 1.10 supports only TF 1.15.5 / 2.9. See: Transform compatible versions table In order to use a later TF version, users can now either install tensorflow-transform 1.11.0, or nightly.

Thank you!

[davidxia]

The pip-compile error I had above was actually incorrect behavior from legacy pip-compile resolver logic. pip-compile ≥ 6.10.0 with --resolver backtracking successfully resolves the dependencies.

The legacy resolver will occasionally fail to resolve dependencies. The backtracking resolver is more robust, but can take longer to run in general.

— https://pypi.org/project/pip-tools/