search

tensorflow / tfx

TFX is an end-to-end platform for deploying production ML pipelines
https://tensorflow.org/tfx
Apache License 2.0
2.11k stars 706 forks source link

TFX with Dataflow Python Version Error #1216

Closed luischinchillagarcia closed 4 years ago

luischinchillagarcia commented 4 years ago

When running a pipeline with Dataflow, BQExampleGen job always works, however, the statisticsGen, schemaGen, ExampleValidator always fails, saying the python version in the 'setup.py' is the issue. Here is a Colab file reproducing the issue.

This Beam issue briefly addresses the issue, however, upon testing the recommended Python versions >=3.7.4, I get an error that Dataflow requires Python 3.6.9. Is there any way to circumvent.

The exact error on the Dataflow logs appears as follows: 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py", line 650, in do_work
    work_executor.execute()
  File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py", line 150, in execute
    test_shuffle_sink=self._test_shuffle_sink)
  File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py", line 116, in create_operation
    is_streaming=False)
  File "apache_beam/runners/worker/operations.py", line 932, in apache_beam.runners.worker.operations.create_operation
  File "apache_beam/runners/worker/operations.py", line 766, in apache_beam.runners.worker.operations.create_pgbk_op
  File "apache_beam/runners/worker/operations.py", line 822, in apache_beam.runners.worker.operations.PGBKCVOperation.__init__
  File "/usr/local/lib/python3.6/site-packages/apache_beam/internal/pickler.py", line 283, in loads
    return dill.loads(s)
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 317, in loads
    return load(file, ignore)
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 305, in load
    obj = pik.load()
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 577, in _load_type
    return _reverse_typemap[name]
KeyError: 'ClassType'
gowthamkpr commented 4 years ago

I don't have access to your colab. I just requested it.

luischinchillagarcia commented 4 years ago

Just saw this, try it again and there should be no restrictions!

jiyongjung0 commented 4 years ago

Hello, @luischinchillagarcia Thank you for reporting an issue. I tried the colab, but failed to run due to other issues.

By the way, I found an issue in the beam mailing thread which looks very similar to this issue. Can you try after pinning cloudpickle version?

luischinchillagarcia commented 4 years ago

I'll try it now!

luischinchillagarcia commented 4 years ago

Unfortunately, the suggestions in the beam mailing thread did not resolve the issue. Perhaps a better avenue to ask is: what would be the suggested list of packages and versions to include under the install_requires parameter in the setup.py file required for Dataflow (including the python version since anything 3.7 fails as well)?

Here is a new, cleaned up, Colab which I used to test these suggestions.

Something to add: the pipeline works perfectly in a local environment..

jiyongjung0 commented 4 years ago

You can find package dependencies in the Google Cloud documentation, but when I tried to run your colab, Dataflow workers used Beam SDK version 2.19 which is not listed in the doc. It looks like it uses python 3.6.10.

luischinchillagarcia commented 4 years ago

I tried running it now with all the compatible versions as shown in the above documentation, and BigQueryExampleGen runs, but the rest doesn't. The logs still show an issue with pickling.

charlesccychen commented 4 years ago

Can you also paste the result of pip freeze on your local machine? The link above pinned a local dependency, not a remote Dataflow one.

luischinchillagarcia commented 4 years ago

For tfx==0.15.0 and apache-beam==2.17.0, here's what I have

apache-beam==2.17.0
ml-metadata==0.15.0
pyarrow==0.15.0
tensorflow==1.15.0
tensorflow-data-validation==0.15.0
tensorflow-datasets==2.0.0
tensorflow-estimator==1.15.1
tensorflow-gan==2.0.0
tensorflow-hub==0.7.0
tensorflow-metadata==0.15.0
tensorflow-model-analysis==0.15.2
tensorflow-privacy==0.2.2
tensorflow-probability==0.7.0
tensorflow-serving-api==2.1.0
tensorflow-transform==0.15.0
tfx==0.15.0
tfx-bsl==0.15.1
dill==0.3.0
tvalentyn commented 4 years ago

@luischinchillagarcia

For tfx==0.15.0 and apache-beam==2.17.0, here's what I have

apache-beam==2.17.0
...

Can you please post complete output of pip freeze without filtering any dependencies?

luischinchillagarcia commented 4 years ago

Local env:

absl-py==0.8.1
apache-beam==2.17.0
appnope==0.1.0
astor==0.8.1
attrs==19.3.0
avro-python3==1.9.1
backcall==0.1.0
bleach==3.1.0
cachetools==3.1.1
certifi==2019.11.28
chardet==3.0.4
Click==7.0
cloudpickle==1.2.2
crcmod==1.7
decorator==4.4.1
defusedxml==0.6.0
dill==0.3.0
docopt==0.6.2
entrypoints==0.3
fastavro==0.21.24
fasteners==0.15
future==0.18.2
gast==0.2.2
google-api-core==1.16.0
google-api-python-client==1.7.11
google-apitools==0.5.28
google-auth==1.10.1
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.4.1
google-cloud-bigquery==1.23.1
google-cloud-bigtable==1.0.0
google-cloud-core==1.2.0
google-cloud-datastore==1.7.4
google-cloud-pubsub==1.0.2
google-pasta==0.1.8
google-resumable-media==0.4.1
googleapis-common-protos==1.51.0
grpc-google-iam-v1==0.12.3
grpcio==1.26.0
h5py==2.10.0
hdfs==2.5.8
httplib2==0.12.0
idna==2.8
importlib-metadata==1.4.0
ipykernel==5.1.3
ipython==7.11.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.15.2
Jinja2==2.10.3
joblib==0.14.1
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==6.1.0
jupyter-core==4.6.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
Markdown==3.1.1
MarkupSafe==1.1.1
mistune==0.8.4
ml-metadata==0.15.0
mock==2.0.0
monotonic==1.5
more-itertools==8.1.0
mpmath==1.1.0
nbconvert==5.6.1
nbformat==5.0.3
notebook==6.0.3
numpy==1.17.4
oauth2client==3.0.0
oauthlib==3.1.0
opt-einsum==3.1.0
pandas==0.25.3
pandocfilters==1.4.2
parso==0.5.2
pbr==5.4.4
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.7.1
promise==2.3
prompt-toolkit==3.0.2
protobuf==3.11.2
psutil==5.6.7
ptyprocess==0.6.0
pyarrow==0.15.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pydot==1.4.1
Pygments==2.5.2
pymongo==3.10.1
pyparsing==2.4.6
pyrsistent==0.15.7
python-dateutil==2.8.1
pytz==2019.3
PyYAML==3.13
pyzmq==18.1.1
qtconsole==4.6.0
requests==2.22.0
requests-oauthlib==1.3.0
rsa==4.0
scikit-learn==0.21.3
scipy==1.1.0
Send2Trash==1.5.0
six==1.14.0
tensorboard==1.15.0
tensorflow==1.15.0
tensorflow-data-validation==0.15.0
tensorflow-datasets==2.0.0
tensorflow-estimator==1.15.1
tensorflow-gan==2.0.0
tensorflow-hub==0.7.0
tensorflow-metadata==0.15.0
tensorflow-model-analysis==0.15.2
tensorflow-privacy==0.2.2
tensorflow-probability==0.7.0
tensorflow-serving-api==2.1.0
tensorflow-transform==0.15.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
tfx==0.15.0
tfx-bsl==0.15.1
tornado==6.0.3
tqdm==4.42.1
traitlets==4.3.3
uritemplate==3.0.1
urllib3==1.25.8
wcwidth==0.1.8
webencodings==0.5.1
Werkzeug==0.16.0
widgetsnbextension==3.5.1
wrapt==1.11.2
zipp==2.0.0

Colab env:

absl-py==0.9.0
alabaster==0.7.12
albumentations==0.1.12
altair==4.0.1
apache-beam==2.17.0
asgiref==3.2.3
astor==0.8.1
astropy==4.0
atari-py==0.2.6
atomicwrites==1.3.0
attrs==19.3.0
audioread==2.1.8
autograd==1.3
avro-python3==1.9.1
Babel==2.8.0
backcall==0.1.0
backports.tempfile==1.0
backports.weakref==1.0.post1
beautifulsoup4==4.6.3
bleach==3.1.0
blis==0.2.4
bokeh==1.4.0
boto==2.49.0
boto3==1.11.9
botocore==1.14.9
Bottleneck==1.3.1
branca==0.3.1
bs4==0.0.1
bz2file==0.98
cachetools==4.0.0
certifi==2019.11.28
cffi==1.13.2
chainer==6.5.0
chardet==3.0.4
chart-studio==1.0.0
Click==7.0
cloudpickle==1.2.2
cmake==3.12.0
colorlover==0.3.0
community==1.0.0b1
contextlib2==0.5.5
convertdate==2.2.0
coverage==3.7.1
coveralls==0.5
crcmod==1.7
cufflinks==0.17.0
cvxopt==1.2.4
cvxpy==1.0.25
cycler==0.10.0
cymem==2.0.3
Cython==0.29.14
daft==0.0.4
dask==2.9.2
dataclasses==0.7
datascience==0.10.6
decorator==4.4.1
defusedxml==0.6.0
descartes==1.1.0
dill==0.3.0
distributed==1.25.3
Django==3.0.2
dlib==19.18.0
dm-sonnet==1.35
docopt==0.6.2
docutils==0.15.2
dopamine-rl==1.0.5
earthengine-api==0.1.212
easydict==1.9
ecos==2.0.7.post1
editdistance==0.5.3
en-core-web-sm==2.1.0
entrypoints==0.3
et-xmlfile==1.0.1
fa2==0.3.5
fancyimpute==0.4.3
fastai==1.0.60
fastavro==0.21.24
fastdtw==0.3.4
fastprogress==0.2.2
fastrlock==0.4
fbprophet==0.5
feather-format==0.4.0
featuretools==0.4.1
filelock==3.0.12
fix-yahoo-finance==0.0.22
Flask==1.1.1
folium==0.8.3
fsspec==0.6.2
future==0.16.0
gast==0.2.2
GDAL==2.2.2
gdown==3.6.4
gensim==3.6.0
geographiclib==1.50
geopy==1.17.0
gevent==1.4.0
gin-config==0.3.0
glob2==0.7
google==2.0.3
google-api-core==1.16.0
google-api-python-client==1.7.11
google-auth==1.4.2
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.4.1
google-cloud-bigquery==1.21.0
google-cloud-core==1.0.3
google-cloud-datastore==1.8.0
google-cloud-language==1.2.0
google-cloud-storage==1.16.2
google-cloud-translate==1.5.0
google-colab==1.0.0
google-pasta==0.1.8
google-resumable-media==0.4.1
googleapis-common-protos==1.51.0
googledrivedownloader==0.4
graph-nets==1.0.5
graphviz==0.10.1
greenlet==0.4.15
grpcio==1.15.0
gspread==3.0.1
gspread-dataframe==3.0.3
gunicorn==20.0.4
gym==0.15.4
h5py==2.8.0
hdfs==2.5.8
HeapDict==1.0.1
holidays==0.9.12
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.11.3
httplib2shim==0.0.3
humanize==0.5.1
hyperopt==0.1.2
ideep4py==2.0.0.post3
idna==2.8
image==1.5.28
imageio==2.4.1
imagesize==1.2.0
imbalanced-learn==0.4.3
imblearn==0.0
imgaug==0.2.9
importlib-metadata==1.5.0
imutils==0.5.3
inflect==2.1.0
intel-openmp==2020.0.133
intervaltree==2.1.0
ipykernel==4.6.1
ipython==5.5.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.5.1
itsdangerous==1.1.0
jax==0.1.58
jaxlib==0.1.38
jdcal==1.4.1
jedi==0.16.0
jieba==0.42.1
Jinja2==2.11.1
jmespath==0.9.4
joblib==0.14.1
jpeg4py==0.1.4
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==5.2.0
jupyter-core==4.6.1
kaggle==1.5.6
kapre==0.1.3.1
Keras==2.2.5
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keras-vis==0.4.1
kfac==0.2.0
kiwisolver==1.1.0
knnimpute==0.1.0
librosa==0.6.3
lightgbm==2.2.3
llvmlite==0.31.0
lmdb==0.98
lucid==0.3.8
lunardate==0.2.0
lxml==4.2.6
magenta==0.3.19
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.2
matplotlib-venn==0.11.5
mesh-tensorflow==0.1.9
mido==1.2.6
mir-eval==0.5
missingno==0.4.2
mistune==0.8.4
mizani==0.6.0
mkl==2019.0
ml-metadata==0.15.0
mlxtend==0.14.0
mock==2.0.0
more-itertools==8.2.0
moviepy==0.2.3.5
mpi4py==3.0.3
mpmath==1.1.0
msgpack==0.5.6
multiprocess==0.70.9
multitasking==0.0.9
murmurhash==1.0.2
music21==5.5.0
natsort==5.5.0
nbconvert==5.6.1
nbformat==5.0.4
networkx==2.4
nibabel==2.3.3
nltk==3.2.5
notebook==5.2.2
np-utils==0.5.12.1
numba==0.47.0
numexpr==2.7.1
numpy==1.17.5
nvidia-ml-py3==7.352.0
oauth2client==3.0.0
oauthlib==3.1.0
okgrade==0.4.3
opencv-contrib-python==4.1.2.30
opencv-python==4.1.2.30
openpyxl==2.5.9
opt-einsum==3.1.0
osqp==0.6.1
packaging==20.1
palettable==3.3.0
pandas==0.25.3
pandas-datareader==0.7.4
pandas-gbq==0.11.0
pandas-profiling==1.4.1
pandocfilters==1.4.2
parso==0.6.1
pathlib==1.0.1
patsy==0.5.1
pbr==5.4.4
pexpect==4.8.0
pickleshare==0.7.5
Pillow==6.2.2
pip-tools==4.2.0
plac==0.9.6
plotly==4.4.1
plotnine==0.6.0
pluggy==0.7.1
portpicker==1.2.0
prefetch-generator==1.0.1
preshed==2.0.1
pretty-midi==0.2.8
prettytable==0.7.2
progressbar2==3.38.0
prometheus-client==0.7.1
promise==2.3
prompt-toolkit==1.0.18
protobuf==3.10.0
psutil==5.6.7
psycopg2==2.7.6.1
ptyprocess==0.6.0
py==1.8.1
pyarrow==0.15.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.0
pycparser==2.19
pydata-google-auth==0.2.1
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyglet==1.3.2
Pygments==2.1.3
pygobject==3.26.1
pymc3==3.7
PyMeeus==0.3.6
pymongo==3.10.1
pymystem3==0.2.0
PyOpenGL==3.1.5
pyparsing==2.4.6
pypng==0.0.20
pyrsistent==0.15.7
pysndfile==1.3.8
PySocks==1.7.1
pystan==2.19.1.1
pytest==3.6.4
python-apt==1.6.5+ubuntu0.2
python-chess==0.23.11
python-dateutil==2.8.1
python-louvain==0.13
python-rtmidi==1.4.0
python-slugify==4.0.0
python-utils==2.3.0
pytz==2018.9
PyWavelets==1.1.1
PyYAML==3.13
pyzmq==17.0.0
qtconsole==4.6.0
regex==2019.12.20
requests==2.21.0
requests-oauthlib==1.3.0
resampy==0.2.2
retrying==1.3.3
rpy2==2.9.5
rsa==4.0
s3fs==0.4.0
s3transfer==0.3.2
scikit-image==0.16.2
scikit-learn==0.21.3
scipy==1.1.0
screen-resolution-extra==0.0.0
scs==2.1.1.post2
seaborn==0.9.1
semantic-version==2.8.4
Send2Trash==1.5.0
setuptools-git==1.2
Shapely==1.7.0
simplegeneric==0.8.1
six==1.12.0
sklearn==0.0
sklearn-pandas==1.8.0
smart-open==1.9.0
snowballstemmer==2.0.0
sortedcontainers==2.1.0
spacy==2.1.9
Sphinx==1.8.5
sphinxcontrib-websupport==1.1.2
SQLAlchemy==1.3.13
sqlparse==0.3.0
srsly==1.0.1
stable-baselines==2.2.1
statsmodels==0.10.2
sympy==1.1.1
tables==3.4.4
tabulate==0.8.6
tblib==1.6.0
tensor2tensor==1.14.1
tensorboard==1.15.0
tensorboardcolab==0.0.22
tensorflow==1.15.0
tensorflow-data-validation==0.15.0
tensorflow-datasets==2.0.0
tensorflow-estimator==1.15.1
tensorflow-gan==2.0.0
tensorflow-hub==0.7.0
tensorflow-metadata==0.15.0
tensorflow-model-analysis==0.15.2
tensorflow-privacy==0.2.2
tensorflow-probability==0.7.0
tensorflow-serving-api==2.1.0
tensorflow-transform==0.15.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
text-unidecode==1.3
textblob==0.15.3
textgenrnn==1.4.1
tflearn==0.3.2
tfx==0.15.0
tfx-bsl==0.15.1
Theano==1.0.4
thinc==7.0.8
toolz==0.10.0
torch==1.4.0
torchsummary==1.5.1
torchtext==0.3.1
torchvision==0.5.0
tornado==4.5.3
tqdm==4.28.1
traitlets==4.3.3
tweepy==3.6.0
typing==3.6.6
typing-extensions==3.6.6
tzlocal==1.5.1
umap-learn==0.3.10
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.8.0
wasabi==0.6.0
wcwidth==0.1.8
webencodings==0.5.1
Werkzeug==0.16.1
widgetsnbextension==3.5.1
wordcloud==1.5.0
wrapt==1.11.2
xarray==0.14.1
xgboost==0.90
xkit==0.0.0
xlrd==1.1.0
xlwt==1.3.0
yellowbrick==0.9.1
zict==1.0.0
zipp==2.1.0
zmq==0.0.0
tvalentyn commented 4 years ago

As we can see, both 'dill' and 'cloudpickle==1.2.2' are installed, which may be causing your issue as per the discussion on Beam mailing list.

luischinchillagarcia commented 4 years ago

I'll create a clean virtualenv and only have dill to see if that works!

Also, when installing the necessary dependencies of tfx onto a clean virtualenv, I get this error. It's strange that it asks for pyarrow to have two incompatible ranges. Perhaps this might lead to something as well.:

ERROR: tfx 0.15.0 has requirement tfx-bsl<0.16,>=0.15.3, but you'll have tfx-bsl 0.15.1 which is incompatible.
ERROR: tfx-bsl 0.15.1 has requirement pyarrow<0.15.0,>=0.14.0, but you'll have pyarrow 0.15.0 which is incompatible.
ERROR: tensorflow-serving-api 2.1.0 has requirement tensorflow~=2.1.0, but you'll have tensorflow 1.15.0 which is incompatible.
ERROR: tensorflow-data-validation 0.15.0 has requirement pyarrow<0.15,>=0.14, but you'll have pyarrow 0.15.0 which is incompatible.
ERROR: apache-beam 2.17.0 has requirement pyarrow<0.16.0,>=0.15.1; python_version >= "3.0" or platform_system != "Windows", but you'll have pyarrow 0.15.0 which is incompatible.
tvalentyn commented 4 years ago

I would try to use a more recent version of beam and/or tfx to find a combination that does not have a conflict with pyarrow. It general, it may be difficult to achieve complete compatibility for any two versions of two package, which both have a complex dependency chain.

luischinchillagarcia commented 4 years ago

The issue is, I have tried with the package combinations listed for tfx 0.14.0, 0.15.0, 0.21.0rc and they all have the exact same error. They run locally, but not in Dataflow specifically, and the error always leads to a pickling issue.

tvalentyn commented 4 years ago

You are dealing with two separate issues here:

Can you also please paste the output of pip install pipdeptree; pipdeptree?

luischinchillagarcia commented 4 years ago

A quick update:

The SDK dependencies for apache-beam=2.17.0 (the compatible version to tfx==0.21.0rc) contains a list of packages already installed (assuming Python 3.7.4) for Dataflow workers. However, the only way I got the Dataflow job to work is to experiment with different versions of python (and slightly changing pyarrow to 0.15.1 to be compatible). Python 3.7.5 seemed to be the only successful one.

pip install -qqq \
         tfx==0.21.0rc\
         tensorflow-data-validation==0.21.0\
         tensorflow-model-analysis==0.21.1\
         tensorflow-metadata==0.21.0\
         tensorflow-transform==0.21.0\
         ml-metadata==0.21.0\
         apache-beam[gcp]==2.17.0\
         pyarrow==0.15.1
         tfx-bsl==0.21.0\
         apache-beam==2.17.0\
         absl-py==0.8.1
         tensorflow==1.15.0\
         dill==0.3.0

Doing this same thing with tfx 0.15.0, I wasn't able to successfully run with any recommended version of python or these packages.

pip install -qqq \
         tfx==0.15.0\
         tensorflow-data-validation==0.15.0\
         tensorflow-model-analysis==0.15.2\
         tensorflow-metadata==0.15.0\
         tensorflow-transform==0.15.0\
         ml-metadata==0.15.0\
         apache-beam[gcp]==2.16.0\
         pyarrow==0.14.1\
         tfx-bsl==0.15.3\
         apache-beam==2.16.0\
         absl-py==0.8.1\
         tensorflow==1.15.0\
         dill==0.3.0

Possible reasons: Cloudpickle was one source of error since I had done this testing on different versions of python before and with no luck. Now absent on both versions with the complete compatible libraries, only the tfx==0.21.0rc works. I don't know why nothing has worked in terms of tfx==0.15.0, since the errors still point to dill being the issue.

luischinchillagarcia commented 4 years ago

Here's the pipdeptree for the environment with tfx==0.15.0:

https://pastebin.com/0DYi6PZw

tvalentyn commented 4 years ago

We expect that the pickling issue will be fixed in cloudpickle==1.3.0. Per [1], this release may be available next week.

Once available, you can install the fixed version of cloudpickle until the AI notebook image picks up the new version.

In the meantime, you can uninstall cloudpickle as a workaround.

[1] https://github.com/cloudpipe/cloudpickle/pull/337

tvalentyn commented 4 years ago

Thanks for reporting this issue, @luischinchillagarcia

luischinchillagarcia commented 4 years ago

Thanks for all the help! @tvalentyn

luischinchillagarcia commented 4 years ago

One last question related to this issue that others may also share:

When running a Dataflow job, does Dataflow have containers for different versions of Beam? If so, how do we choose one that is not the latest version? For instance, (1) do we specify the version of dataflow we want to use on the 'setup.py', or (2) does it match the beam version related container based on the beam version on the machine you are running the job from?

Thanks!

tvalentyn commented 4 years ago

You can ask this question on Beam user mailing list (user@beam.apache.org), Stackoverflow, or via Dataflow customer support channels. TFX bug tracker is not the best forum for this discussion. Beam automatically chooses the correct container when starting a Dataflow job, that corresponds to Beam version you are using. These containers include following dependencies: https://cloud.google.com/dataflow/docs/concepts/sdk-worker-dependencies.

luischinchillagarcia commented 4 years ago

Thanks for the help and resources!

stfines-clgx commented 3 years ago

I have run into this issue with TFX == 1.2.0, TF == 2.5.1, here is the exact stack trace: File "/usr/local/lib/python3.7/site-packages/apache_beam/internal/pickler.py", line 285, in loads return dill.loads(s) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 275, in loads return load(file, ignore, **kwds) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 270, in load return Unpickler(file, ignore=ignore, **kwds).load() File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 577, in _load_type return _reverse_typemap[name] KeyError: 'ClassType'

I also tried removing cloudpickle; any other suggestions as to why I would be seeing the same error?

pip_log.txt

tvalentyn commented 3 years ago

I am not seeing cloudpickle in the list of deps, but if you see it, it should be cloudpickle==1.3.0 or higher. If you have a minimally reproducible example, that could help debug it.

stfines-clgx commented 3 years ago

I had uninstalled cloudpickle as a test and I wind up with the same error