Closed luischinchillagarcia closed 4 years ago
I don't have access to your colab. I just requested it.
Just saw this, try it again and there should be no restrictions!
Hello, @luischinchillagarcia Thank you for reporting an issue. I tried the colab, but failed to run due to other issues.
By the way, I found an issue in the beam mailing thread which looks very similar to this issue. Can you try after pinning cloudpickle version?
I'll try it now!
Unfortunately, the suggestions in the beam mailing thread did not resolve the issue. Perhaps a better avenue to ask is: what would be the suggested list of packages and versions to include under the install_requires parameter in the setup.py file required for Dataflow (including the python version since anything 3.7 fails as well)?
Here is a new, cleaned up, Colab which I used to test these suggestions.
Something to add: the pipeline works perfectly in a local environment..
You can find package dependencies in the Google Cloud documentation, but when I tried to run your colab, Dataflow workers used Beam SDK version 2.19 which is not listed in the doc. It looks like it uses python 3.6.10.
I tried running it now with all the compatible versions as shown in the above documentation, and BigQueryExampleGen runs, but the rest doesn't. The logs still show an issue with pickling.
Can you also paste the result of pip freeze
on your local machine? The link above pinned a local dependency, not a remote Dataflow one.
For tfx==0.15.0 and apache-beam==2.17.0, here's what I have
apache-beam==2.17.0
ml-metadata==0.15.0
pyarrow==0.15.0
tensorflow==1.15.0
tensorflow-data-validation==0.15.0
tensorflow-datasets==2.0.0
tensorflow-estimator==1.15.1
tensorflow-gan==2.0.0
tensorflow-hub==0.7.0
tensorflow-metadata==0.15.0
tensorflow-model-analysis==0.15.2
tensorflow-privacy==0.2.2
tensorflow-probability==0.7.0
tensorflow-serving-api==2.1.0
tensorflow-transform==0.15.0
tfx==0.15.0
tfx-bsl==0.15.1
dill==0.3.0
@luischinchillagarcia
For tfx==0.15.0 and apache-beam==2.17.0, here's what I have
apache-beam==2.17.0 ...
Can you please post complete output of pip freeze without filtering any dependencies?
Local env:
absl-py==0.8.1
apache-beam==2.17.0
appnope==0.1.0
astor==0.8.1
attrs==19.3.0
avro-python3==1.9.1
backcall==0.1.0
bleach==3.1.0
cachetools==3.1.1
certifi==2019.11.28
chardet==3.0.4
Click==7.0
cloudpickle==1.2.2
crcmod==1.7
decorator==4.4.1
defusedxml==0.6.0
dill==0.3.0
docopt==0.6.2
entrypoints==0.3
fastavro==0.21.24
fasteners==0.15
future==0.18.2
gast==0.2.2
google-api-core==1.16.0
google-api-python-client==1.7.11
google-apitools==0.5.28
google-auth==1.10.1
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.4.1
google-cloud-bigquery==1.23.1
google-cloud-bigtable==1.0.0
google-cloud-core==1.2.0
google-cloud-datastore==1.7.4
google-cloud-pubsub==1.0.2
google-pasta==0.1.8
google-resumable-media==0.4.1
googleapis-common-protos==1.51.0
grpc-google-iam-v1==0.12.3
grpcio==1.26.0
h5py==2.10.0
hdfs==2.5.8
httplib2==0.12.0
idna==2.8
importlib-metadata==1.4.0
ipykernel==5.1.3
ipython==7.11.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.15.2
Jinja2==2.10.3
joblib==0.14.1
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==6.1.0
jupyter-core==4.6.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
Markdown==3.1.1
MarkupSafe==1.1.1
mistune==0.8.4
ml-metadata==0.15.0
mock==2.0.0
monotonic==1.5
more-itertools==8.1.0
mpmath==1.1.0
nbconvert==5.6.1
nbformat==5.0.3
notebook==6.0.3
numpy==1.17.4
oauth2client==3.0.0
oauthlib==3.1.0
opt-einsum==3.1.0
pandas==0.25.3
pandocfilters==1.4.2
parso==0.5.2
pbr==5.4.4
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.7.1
promise==2.3
prompt-toolkit==3.0.2
protobuf==3.11.2
psutil==5.6.7
ptyprocess==0.6.0
pyarrow==0.15.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pydot==1.4.1
Pygments==2.5.2
pymongo==3.10.1
pyparsing==2.4.6
pyrsistent==0.15.7
python-dateutil==2.8.1
pytz==2019.3
PyYAML==3.13
pyzmq==18.1.1
qtconsole==4.6.0
requests==2.22.0
requests-oauthlib==1.3.0
rsa==4.0
scikit-learn==0.21.3
scipy==1.1.0
Send2Trash==1.5.0
six==1.14.0
tensorboard==1.15.0
tensorflow==1.15.0
tensorflow-data-validation==0.15.0
tensorflow-datasets==2.0.0
tensorflow-estimator==1.15.1
tensorflow-gan==2.0.0
tensorflow-hub==0.7.0
tensorflow-metadata==0.15.0
tensorflow-model-analysis==0.15.2
tensorflow-privacy==0.2.2
tensorflow-probability==0.7.0
tensorflow-serving-api==2.1.0
tensorflow-transform==0.15.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
tfx==0.15.0
tfx-bsl==0.15.1
tornado==6.0.3
tqdm==4.42.1
traitlets==4.3.3
uritemplate==3.0.1
urllib3==1.25.8
wcwidth==0.1.8
webencodings==0.5.1
Werkzeug==0.16.0
widgetsnbextension==3.5.1
wrapt==1.11.2
zipp==2.0.0
Colab env:
absl-py==0.9.0
alabaster==0.7.12
albumentations==0.1.12
altair==4.0.1
apache-beam==2.17.0
asgiref==3.2.3
astor==0.8.1
astropy==4.0
atari-py==0.2.6
atomicwrites==1.3.0
attrs==19.3.0
audioread==2.1.8
autograd==1.3
avro-python3==1.9.1
Babel==2.8.0
backcall==0.1.0
backports.tempfile==1.0
backports.weakref==1.0.post1
beautifulsoup4==4.6.3
bleach==3.1.0
blis==0.2.4
bokeh==1.4.0
boto==2.49.0
boto3==1.11.9
botocore==1.14.9
Bottleneck==1.3.1
branca==0.3.1
bs4==0.0.1
bz2file==0.98
cachetools==4.0.0
certifi==2019.11.28
cffi==1.13.2
chainer==6.5.0
chardet==3.0.4
chart-studio==1.0.0
Click==7.0
cloudpickle==1.2.2
cmake==3.12.0
colorlover==0.3.0
community==1.0.0b1
contextlib2==0.5.5
convertdate==2.2.0
coverage==3.7.1
coveralls==0.5
crcmod==1.7
cufflinks==0.17.0
cvxopt==1.2.4
cvxpy==1.0.25
cycler==0.10.0
cymem==2.0.3
Cython==0.29.14
daft==0.0.4
dask==2.9.2
dataclasses==0.7
datascience==0.10.6
decorator==4.4.1
defusedxml==0.6.0
descartes==1.1.0
dill==0.3.0
distributed==1.25.3
Django==3.0.2
dlib==19.18.0
dm-sonnet==1.35
docopt==0.6.2
docutils==0.15.2
dopamine-rl==1.0.5
earthengine-api==0.1.212
easydict==1.9
ecos==2.0.7.post1
editdistance==0.5.3
en-core-web-sm==2.1.0
entrypoints==0.3
et-xmlfile==1.0.1
fa2==0.3.5
fancyimpute==0.4.3
fastai==1.0.60
fastavro==0.21.24
fastdtw==0.3.4
fastprogress==0.2.2
fastrlock==0.4
fbprophet==0.5
feather-format==0.4.0
featuretools==0.4.1
filelock==3.0.12
fix-yahoo-finance==0.0.22
Flask==1.1.1
folium==0.8.3
fsspec==0.6.2
future==0.16.0
gast==0.2.2
GDAL==2.2.2
gdown==3.6.4
gensim==3.6.0
geographiclib==1.50
geopy==1.17.0
gevent==1.4.0
gin-config==0.3.0
glob2==0.7
google==2.0.3
google-api-core==1.16.0
google-api-python-client==1.7.11
google-auth==1.4.2
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.4.1
google-cloud-bigquery==1.21.0
google-cloud-core==1.0.3
google-cloud-datastore==1.8.0
google-cloud-language==1.2.0
google-cloud-storage==1.16.2
google-cloud-translate==1.5.0
google-colab==1.0.0
google-pasta==0.1.8
google-resumable-media==0.4.1
googleapis-common-protos==1.51.0
googledrivedownloader==0.4
graph-nets==1.0.5
graphviz==0.10.1
greenlet==0.4.15
grpcio==1.15.0
gspread==3.0.1
gspread-dataframe==3.0.3
gunicorn==20.0.4
gym==0.15.4
h5py==2.8.0
hdfs==2.5.8
HeapDict==1.0.1
holidays==0.9.12
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.11.3
httplib2shim==0.0.3
humanize==0.5.1
hyperopt==0.1.2
ideep4py==2.0.0.post3
idna==2.8
image==1.5.28
imageio==2.4.1
imagesize==1.2.0
imbalanced-learn==0.4.3
imblearn==0.0
imgaug==0.2.9
importlib-metadata==1.5.0
imutils==0.5.3
inflect==2.1.0
intel-openmp==2020.0.133
intervaltree==2.1.0
ipykernel==4.6.1
ipython==5.5.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.5.1
itsdangerous==1.1.0
jax==0.1.58
jaxlib==0.1.38
jdcal==1.4.1
jedi==0.16.0
jieba==0.42.1
Jinja2==2.11.1
jmespath==0.9.4
joblib==0.14.1
jpeg4py==0.1.4
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==5.2.0
jupyter-core==4.6.1
kaggle==1.5.6
kapre==0.1.3.1
Keras==2.2.5
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keras-vis==0.4.1
kfac==0.2.0
kiwisolver==1.1.0
knnimpute==0.1.0
librosa==0.6.3
lightgbm==2.2.3
llvmlite==0.31.0
lmdb==0.98
lucid==0.3.8
lunardate==0.2.0
lxml==4.2.6
magenta==0.3.19
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.2
matplotlib-venn==0.11.5
mesh-tensorflow==0.1.9
mido==1.2.6
mir-eval==0.5
missingno==0.4.2
mistune==0.8.4
mizani==0.6.0
mkl==2019.0
ml-metadata==0.15.0
mlxtend==0.14.0
mock==2.0.0
more-itertools==8.2.0
moviepy==0.2.3.5
mpi4py==3.0.3
mpmath==1.1.0
msgpack==0.5.6
multiprocess==0.70.9
multitasking==0.0.9
murmurhash==1.0.2
music21==5.5.0
natsort==5.5.0
nbconvert==5.6.1
nbformat==5.0.4
networkx==2.4
nibabel==2.3.3
nltk==3.2.5
notebook==5.2.2
np-utils==0.5.12.1
numba==0.47.0
numexpr==2.7.1
numpy==1.17.5
nvidia-ml-py3==7.352.0
oauth2client==3.0.0
oauthlib==3.1.0
okgrade==0.4.3
opencv-contrib-python==4.1.2.30
opencv-python==4.1.2.30
openpyxl==2.5.9
opt-einsum==3.1.0
osqp==0.6.1
packaging==20.1
palettable==3.3.0
pandas==0.25.3
pandas-datareader==0.7.4
pandas-gbq==0.11.0
pandas-profiling==1.4.1
pandocfilters==1.4.2
parso==0.6.1
pathlib==1.0.1
patsy==0.5.1
pbr==5.4.4
pexpect==4.8.0
pickleshare==0.7.5
Pillow==6.2.2
pip-tools==4.2.0
plac==0.9.6
plotly==4.4.1
plotnine==0.6.0
pluggy==0.7.1
portpicker==1.2.0
prefetch-generator==1.0.1
preshed==2.0.1
pretty-midi==0.2.8
prettytable==0.7.2
progressbar2==3.38.0
prometheus-client==0.7.1
promise==2.3
prompt-toolkit==1.0.18
protobuf==3.10.0
psutil==5.6.7
psycopg2==2.7.6.1
ptyprocess==0.6.0
py==1.8.1
pyarrow==0.15.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.0
pycparser==2.19
pydata-google-auth==0.2.1
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyglet==1.3.2
Pygments==2.1.3
pygobject==3.26.1
pymc3==3.7
PyMeeus==0.3.6
pymongo==3.10.1
pymystem3==0.2.0
PyOpenGL==3.1.5
pyparsing==2.4.6
pypng==0.0.20
pyrsistent==0.15.7
pysndfile==1.3.8
PySocks==1.7.1
pystan==2.19.1.1
pytest==3.6.4
python-apt==1.6.5+ubuntu0.2
python-chess==0.23.11
python-dateutil==2.8.1
python-louvain==0.13
python-rtmidi==1.4.0
python-slugify==4.0.0
python-utils==2.3.0
pytz==2018.9
PyWavelets==1.1.1
PyYAML==3.13
pyzmq==17.0.0
qtconsole==4.6.0
regex==2019.12.20
requests==2.21.0
requests-oauthlib==1.3.0
resampy==0.2.2
retrying==1.3.3
rpy2==2.9.5
rsa==4.0
s3fs==0.4.0
s3transfer==0.3.2
scikit-image==0.16.2
scikit-learn==0.21.3
scipy==1.1.0
screen-resolution-extra==0.0.0
scs==2.1.1.post2
seaborn==0.9.1
semantic-version==2.8.4
Send2Trash==1.5.0
setuptools-git==1.2
Shapely==1.7.0
simplegeneric==0.8.1
six==1.12.0
sklearn==0.0
sklearn-pandas==1.8.0
smart-open==1.9.0
snowballstemmer==2.0.0
sortedcontainers==2.1.0
spacy==2.1.9
Sphinx==1.8.5
sphinxcontrib-websupport==1.1.2
SQLAlchemy==1.3.13
sqlparse==0.3.0
srsly==1.0.1
stable-baselines==2.2.1
statsmodels==0.10.2
sympy==1.1.1
tables==3.4.4
tabulate==0.8.6
tblib==1.6.0
tensor2tensor==1.14.1
tensorboard==1.15.0
tensorboardcolab==0.0.22
tensorflow==1.15.0
tensorflow-data-validation==0.15.0
tensorflow-datasets==2.0.0
tensorflow-estimator==1.15.1
tensorflow-gan==2.0.0
tensorflow-hub==0.7.0
tensorflow-metadata==0.15.0
tensorflow-model-analysis==0.15.2
tensorflow-privacy==0.2.2
tensorflow-probability==0.7.0
tensorflow-serving-api==2.1.0
tensorflow-transform==0.15.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
text-unidecode==1.3
textblob==0.15.3
textgenrnn==1.4.1
tflearn==0.3.2
tfx==0.15.0
tfx-bsl==0.15.1
Theano==1.0.4
thinc==7.0.8
toolz==0.10.0
torch==1.4.0
torchsummary==1.5.1
torchtext==0.3.1
torchvision==0.5.0
tornado==4.5.3
tqdm==4.28.1
traitlets==4.3.3
tweepy==3.6.0
typing==3.6.6
typing-extensions==3.6.6
tzlocal==1.5.1
umap-learn==0.3.10
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.8.0
wasabi==0.6.0
wcwidth==0.1.8
webencodings==0.5.1
Werkzeug==0.16.1
widgetsnbextension==3.5.1
wordcloud==1.5.0
wrapt==1.11.2
xarray==0.14.1
xgboost==0.90
xkit==0.0.0
xlrd==1.1.0
xlwt==1.3.0
yellowbrick==0.9.1
zict==1.0.0
zipp==2.1.0
zmq==0.0.0
As we can see, both 'dill' and 'cloudpickle==1.2.2' are installed, which may be causing your issue as per the discussion on Beam mailing list.
I'll create a clean virtualenv and only have dill to see if that works!
Also, when installing the necessary dependencies of tfx onto a clean virtualenv, I get this error. It's strange that it asks for pyarrow to have two incompatible ranges. Perhaps this might lead to something as well.:
ERROR: tfx 0.15.0 has requirement tfx-bsl<0.16,>=0.15.3, but you'll have tfx-bsl 0.15.1 which is incompatible.
ERROR: tfx-bsl 0.15.1 has requirement pyarrow<0.15.0,>=0.14.0, but you'll have pyarrow 0.15.0 which is incompatible.
ERROR: tensorflow-serving-api 2.1.0 has requirement tensorflow~=2.1.0, but you'll have tensorflow 1.15.0 which is incompatible.
ERROR: tensorflow-data-validation 0.15.0 has requirement pyarrow<0.15,>=0.14, but you'll have pyarrow 0.15.0 which is incompatible.
ERROR: apache-beam 2.17.0 has requirement pyarrow<0.16.0,>=0.15.1; python_version >= "3.0" or platform_system != "Windows", but you'll have pyarrow 0.15.0 which is incompatible.
I would try to use a more recent version of beam and/or tfx to find a combination that does not have a conflict with pyarrow. It general, it may be difficult to achieve complete compatibility for any two versions of two package, which both have a complex dependency chain.
The issue is, I have tried with the package combinations listed for tfx 0.14.0, 0.15.0, 0.21.0rc and they all have the exact same error. They run locally, but not in Dataflow specifically, and the error always leads to a pickling issue.
You are dealing with two separate issues here:
Can you also please paste the output of pip install pipdeptree; pipdeptree
?
A quick update:
The SDK dependencies for apache-beam=2.17.0 (the compatible version to tfx==0.21.0rc) contains a list of packages already installed (assuming Python 3.7.4) for Dataflow workers. However, the only way I got the Dataflow job to work is to experiment with different versions of python (and slightly changing pyarrow to 0.15.1 to be compatible). Python 3.7.5 seemed to be the only successful one.
pip install -qqq \
tfx==0.21.0rc\
tensorflow-data-validation==0.21.0\
tensorflow-model-analysis==0.21.1\
tensorflow-metadata==0.21.0\
tensorflow-transform==0.21.0\
ml-metadata==0.21.0\
apache-beam[gcp]==2.17.0\
pyarrow==0.15.1
tfx-bsl==0.21.0\
apache-beam==2.17.0\
absl-py==0.8.1
tensorflow==1.15.0\
dill==0.3.0
Doing this same thing with tfx 0.15.0, I wasn't able to successfully run with any recommended version of python or these packages.
pip install -qqq \
tfx==0.15.0\
tensorflow-data-validation==0.15.0\
tensorflow-model-analysis==0.15.2\
tensorflow-metadata==0.15.0\
tensorflow-transform==0.15.0\
ml-metadata==0.15.0\
apache-beam[gcp]==2.16.0\
pyarrow==0.14.1\
tfx-bsl==0.15.3\
apache-beam==2.16.0\
absl-py==0.8.1\
tensorflow==1.15.0\
dill==0.3.0
Possible reasons: Cloudpickle was one source of error since I had done this testing on different versions of python before and with no luck. Now absent on both versions with the complete compatible libraries, only the tfx==0.21.0rc works. I don't know why nothing has worked in terms of tfx==0.15.0, since the errors still point to dill being the issue.
Here's the pipdeptree for the environment with tfx==0.15.0:
We expect that the pickling issue will be fixed in cloudpickle==1.3.0. Per [1], this release may be available next week.
Once available, you can install the fixed version of cloudpickle until the AI notebook image picks up the new version.
In the meantime, you can uninstall cloudpickle as a workaround.
Thanks for reporting this issue, @luischinchillagarcia
Thanks for all the help! @tvalentyn
One last question related to this issue that others may also share:
When running a Dataflow job, does Dataflow have containers for different versions of Beam? If so, how do we choose one that is not the latest version? For instance, (1) do we specify the version of dataflow we want to use on the 'setup.py', or (2) does it match the beam version related container based on the beam version on the machine you are running the job from?
Thanks!
You can ask this question on Beam user mailing list (user@beam.apache.org), Stackoverflow, or via Dataflow customer support channels. TFX bug tracker is not the best forum for this discussion. Beam automatically chooses the correct container when starting a Dataflow job, that corresponds to Beam version you are using. These containers include following dependencies: https://cloud.google.com/dataflow/docs/concepts/sdk-worker-dependencies.
Thanks for the help and resources!
I have run into this issue with TFX == 1.2.0, TF == 2.5.1,
here is the exact stack trace:
File "/usr/local/lib/python3.7/site-packages/apache_beam/internal/pickler.py", line 285, in loads return dill.loads(s) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 275, in loads return load(file, ignore, **kwds) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 270, in load return Unpickler(file, ignore=ignore, **kwds).load() File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 577, in _load_type return _reverse_typemap[name] KeyError: 'ClassType'
I also tried removing cloudpickle; any other suggestions as to why I would be seeing the same error?
I am not seeing cloudpickle in the list of deps, but if you see it, it should be cloudpickle==1.3.0 or higher. If you have a minimally reproducible example, that could help debug it.
I had uninstalled cloudpickle as a test and I wind up with the same error
When running a pipeline with Dataflow, BQExampleGen job always works, however, the statisticsGen, schemaGen, ExampleValidator always fails, saying the python version in the 'setup.py' is the issue. Here is a Colab file reproducing the issue.
This Beam issue briefly addresses the issue, however, upon testing the recommended Python versions >=3.7.4, I get an error that Dataflow requires Python 3.6.9. Is there any way to circumvent.
The exact error on the Dataflow logs appears as follows: