Closes #346
The quickstart pipeline uses the TFX image as it's base, which is ~9GB and includes several extra tools which aren't needed by the quickstart pipeline. This PR uses a slim python image as the base instead, and installs the tfx python package and other dependencies directly which will reduce the image size dramatically.
Image size when using base TFX image:
20 GB when built locally
9GB (virtual size)
Image size after using tfx python package on slim python:
2.46GB (when built locally)
784MB (virtual size)
We encountered a recent breaking change in the setuptools dependency indirectly imported by TFX which was causing CsvExampleGen to fail (see main card). This was fixed by pinning the dependency version.
aidandunlop
commented
3 months ago
Closes #346 The quickstart pipeline uses the TFX image as it's base, which is ~9GB and includes several extra tools which aren't needed by the quickstart pipeline. This PR uses a slim python image as the base instead, and installs the
tfxpython package and other dependencies directly which will reduce the image size dramatically.Image size when using base TFX image:
Image size after using
tfxpython package on slim python:We encountered a recent breaking change in the
setuptoolsdependency indirectly imported by TFX which was causingCsvExampleGento fail (see main card). This was fixed by pinning the dependency version.Tasks