tensorflow / tfx

TFX is an end-to-end platform for deploying production ML pipelines
https://tensorflow.github.io/tfx/
Apache License 2.0
2.11k stars 707 forks source link

Missing tfx.orchestration in Colab tutorials #5610

Closed michaelwsherman closed 1 year ago

michaelwsherman commented 1 year ago

System information

Describe the current behavior In TFX tutorial colabs, when I run the code: tfx.orchestration.LocalDagRunner().run(whatever) I get: AttributeError: module 'tfx.v1' has no attribute 'orchestration'

I've seen this in two separate TFX tutorial notebooks: penguin_simple.ipynb and penguin_tfdv.ipynb. I am following the instructions in the notebook to restart the runtime after pipping TFX.

I've experienced the issue in penguin_simple.ipynb using colab on two separate Google accounts. In my own local dev environment, I can run the code from penguin_simple.ipynb and it works. In both colab and my own environment, print('TFX version: {}'.format(tfx.__version__)) outputs TFX version: 1.12.0.

Investigating a bit, in colab dir(tfx) shows no orchestration, but in my own environment orchestration is present.

My environment is Python 3.7.10 but colab is Python version 3.8.16, maybe this is related?

Describe the expected behavior The notebooks should work in colab.

Standalone code to reproduce the issue Not quite standalone code, but if you go to the Simple TFX Pipeline Tutorial page and open the Run in Google Colab link, then run through the notebook (including restarting after pip install -U tfx), you'll get this error on the second to last code block.

pip freeze:

absl-py==1.3.0
aeppl==0.0.33
aesara==2.7.9
aiohttp==3.8.3
aiosignal==1.3.1
alabaster==0.7.12
albumentations==1.2.1
altair==4.2.0
apache-beam==2.43.0
appdirs==1.4.4
arviz==0.12.1
astor==0.8.1
astropy==4.3.1
astunparse==1.6.3
async-timeout==4.0.2
atari-py==0.2.9
atomicwrites==1.4.1
attrs==21.4.0
audioread==3.0.0
autograd==1.5
Babel==2.11.0
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==5.0.1
blis==0.7.9
bokeh==2.3.3
branca==0.6.0
bs4==0.0.1
CacheControl==0.12.11
cachetools==4.2.4
catalogue==2.0.8
certifi==2022.12.7
cffi==1.15.1
cftime==1.6.2
chardet==3.0.4
charset-normalizer==2.1.1
click==7.1.2
clikit==0.6.2
cloudpickle==2.2.0
cmake==3.22.6
cmdstanpy==1.0.8
colorcet==3.0.1
colorlover==0.3.0
community==1.0.0b1
confection==0.0.3
cons==0.4.5
contextlib2==0.5.5
convertdate==2.4.0
crashtest==0.3.1
crcmod==1.7
cufflinks==0.17.3
cvxopt==1.3.0
cvxpy==1.2.2
cycler==0.11.0
cymem==2.0.7
Cython==0.29.32
daft==0.0.4
dask==2022.2.1
datascience==0.17.5
db-dtypes==1.0.5
debugpy==1.0.0
decorator==4.4.2
defusedxml==0.7.1
descartes==1.1.0
dill==0.3.1.1
distributed==2022.2.1
dlib==19.24.0
dm-tree==0.1.7
dnspython==2.2.1
docker==4.4.4
docopt==0.6.2
docutils==0.17.1
dopamine-rl==1.0.5
earthengine-api==0.1.335
easydict==1.10
ecos==2.0.10
editdistance==0.5.3
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.1/en_core_web_sm-3.4.1-py3-none-any.whl
entrypoints==0.4
ephem==4.1.3
et-xmlfile==1.1.0
etils==0.9.0
etuples==0.3.8
fa2==0.3.5
fastai==2.7.10
fastavro==1.7.0
fastcore==1.5.27
fastdownload==0.0.7
fastdtw==0.3.4
fasteners==0.18
fastjsonschema==2.16.2
fastprogress==1.0.3
fastrlock==0.8.1
feather-format==0.4.1
filelock==3.8.2
firebase-admin==5.3.0
fix-yahoo-finance==0.0.22
Flask==1.1.4
flatbuffers==22.12.6
folium==0.12.1.post1
frozenlist==1.3.3
fsspec==2022.11.0
future==0.16.0
gast==0.4.0
GDAL==2.2.2
gdown==4.4.0
gensim==3.6.0
geographiclib==1.52
geopy==1.17.0
gin-config==0.5.0
glob2==0.7
google==2.0.3
google-api-core==1.32.0
google-api-python-client==1.12.11
google-apitools==0.5.31
google-auth==1.35.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-aiplatform==1.17.1
google-cloud-bigquery==2.34.4
google-cloud-bigquery-storage==2.13.2
google-cloud-bigtable==1.7.3
google-cloud-core==2.3.2
google-cloud-datastore==1.15.5
google-cloud-dlp==3.9.2
google-cloud-firestore==2.7.2
google-cloud-language==1.3.2
google-cloud-pubsub==2.13.11
google-cloud-pubsublite==1.6.0
google-cloud-recommendations-ai==0.7.1
google-cloud-resource-manager==1.6.3
google-cloud-spanner==3.26.0
google-cloud-storage==2.5.0
google-cloud-translate==3.8.4
google-cloud-videointelligence==1.16.3
google-cloud-vision==1.0.2
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.4.0
googleapis-common-protos==1.57.0
googledrivedownloader==0.4
graphviz==0.10.1
greenlet==2.0.1
grpc-google-iam-v1==0.12.4
grpcio==1.51.1
grpcio-status==1.48.2
gspread==3.4.2
gspread-dataframe==3.0.8
gym==0.25.2
gym-notices==0.0.8
h5py==3.1.0
hdfs==2.7.0
HeapDict==1.0.1
hijri-converter==2.2.4
holidays==0.17.2
holoviews==1.14.9
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.17.4
httpstan==4.6.1
humanize==0.5.1
hyperopt==0.1.2
idna==2.10
imageio==2.9.0
imagesize==1.4.1
imbalanced-learn==0.8.1
imblearn==0.0
imgaug==0.4.0
importlib-metadata==5.1.0
importlib-resources==5.10.1
imutils==0.5.4
inflect==2.1.0
intel-openmp==2022.2.1
intervaltree==2.1.0
ipykernel==5.3.4
ipython==7.9.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.7.1
itsdangerous==1.1.0
jax==0.3.25
jaxlib @ https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.3.25+cuda11.cudnn805-cp38-cp38-manylinux2014_x86_64.whl
jedi==0.18.2
jieba==0.42.1
Jinja2==2.11.3
joblib==1.2.0
jpeg4py==0.1.4
jsonschema==4.3.3
jupyter-client==6.1.12
jupyter-console==6.1.0
jupyter_core==5.1.0
jupyterlab-widgets==3.0.4
kaggle==1.5.12
kapre==0.3.7
keras==2.11.0
Keras-Preprocessing==1.1.2
keras-tuner==1.1.3
keras-vis==0.4.1
kiwisolver==1.4.4
korean-lunar-calendar==0.3.1
kt-legacy==1.0.4
kubernetes==12.0.1
langcodes==3.3.0
libclang==14.0.6
librosa==0.8.1
lightgbm==2.2.3
llvmlite==0.39.1
lmdb==0.99
locket==1.0.0
logical-unification==0.4.5
LunarCalendar==0.0.9
lxml==4.9.2
Markdown==3.4.1
MarkupSafe==2.0.1
marshmallow==3.19.0
matplotlib==3.2.2
matplotlib-venn==0.11.7
miniKanren==1.0.3
missingno==0.5.1
mistune==0.8.4
mizani==0.7.3
mkl==2019.0
ml-metadata==1.12.0
ml-pipelines-sdk==1.12.0
mlxtend==0.14.0
more-itertools==9.0.0
moviepy==0.2.3.5
mpmath==1.2.1
msgpack==1.0.4
multidict==6.0.3
multipledispatch==0.6.0
multitasking==0.0.11
murmurhash==1.0.9
music21==5.5.0
natsort==5.5.0
nbconvert==5.6.1
nbformat==5.7.0
netCDF4==1.6.2
networkx==2.8.8
nibabel==3.0.2
nltk==3.7
notebook==5.7.16
numba==0.56.4
numexpr==2.8.4
numpy==1.21.6
oauth2client==4.1.3
oauthlib==3.2.2
objsize==0.5.2
okgrade==0.4.3
opencv-contrib-python==4.6.0.66
opencv-python==4.6.0.66
opencv-python-headless==4.6.0.66
openpyxl==3.0.10
opt-einsum==3.3.0
orjson==3.8.3
osqp==0.6.2.post0
overrides==6.5.0
packaging==20.9
palettable==3.3.0
pandas==1.3.5
pandas-datareader==0.9.0
pandas-gbq==0.17.9
pandas-profiling==1.4.1
pandocfilters==1.5.0
panel==0.12.1
param==1.12.3
parso==0.8.3
partd==1.3.0
pastel==0.2.1
pathlib==1.0.1
pathy==0.10.1
patsy==0.5.3
pep517==0.13.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.1.2
pip-tools==6.2.0
platformdirs==2.6.0
plotly==5.5.0
plotnine==0.8.0
pluggy==0.7.1
pooch==1.6.0
portpicker==1.3.9
prefetch-generator==1.0.3
preshed==3.0.8
prettytable==3.5.0
progressbar2==3.38.0
prometheus-client==0.15.0
promise==2.3
prompt-toolkit==2.0.10
prophet==1.1.1
proto-plus==1.22.1
protobuf==3.19.6
psutil==5.4.8
psycopg2==2.9.5
ptyprocess==0.7.0
py==1.11.0
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.6
pycparser==2.21
pyct==0.4.8
pydantic==1.10.2
pydata-google-auth==1.4.0
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyerfa==2.0.0.1
pyfarmhash==0.3.2
Pygments==2.6.1
PyGObject==3.26.1
pylev==1.4.0
pymc==4.1.4
PyMeeus==0.5.12
pymongo==3.13.0
pymystem3==0.2.0
PyOpenGL==3.1.6
pyparsing==3.0.9
pyrsistent==0.19.2
pysimdjson==3.2.0
pysndfile==1.3.8
PySocks==1.7.1
pystan==3.3.0
pytest==3.6.4
python-apt==0.0.0
python-dateutil==2.8.2
python-louvain==0.16
python-slugify==7.0.0
python-utils==3.4.5
pytz==2022.6
pyviz-comms==2.2.1
PyWavelets==1.4.1
PyYAML==5.4.1
pyzmq==23.2.1
qdldl==0.1.5.post2
qudida==0.0.4
regex==2022.6.2
requests==2.28.1
requests-oauthlib==1.3.1
resampy==0.4.2
rpy2==3.5.5
rsa==4.9
scikit-image==0.18.3
scikit-learn==1.0.2
scipy==1.7.3
screen-resolution-extra==0.0.0
scs==3.2.2
seaborn==0.11.2
Send2Trash==1.8.0
setuptools-git==1.2
shapely==2.0.0
six==1.15.0
sklearn-pandas==1.8.0
smart-open==6.3.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soundfile==0.11.0
spacy==3.4.4
spacy-legacy==3.0.10
spacy-loggers==1.0.4
Sphinx==1.8.6
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib-websupport==1.2.4
SQLAlchemy==1.4.45
sqlparse==0.4.3
srsly==2.4.5
statsmodels==0.12.2
sympy==1.7.1
tables==3.7.0
tabulate==0.8.10
tblib==1.7.0
tenacity==8.1.0
tensorboard==2.11.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-data-validation==1.12.0
tensorflow-datasets==4.6.0
tensorflow-estimator==2.11.0
tensorflow-gcs-config==2.9.1
tensorflow-hub==0.12.0
tensorflow-io-gcs-filesystem==0.28.0
tensorflow-metadata==1.12.0
tensorflow-model-analysis==0.43.0
tensorflow-probability==0.17.0
tensorflow-serving-api==2.11.0
tensorflow-transform==1.12.0
termcolor==2.1.1
terminado==0.13.3
testpath==0.6.0
text-unidecode==1.3
textblob==0.15.3
tfx==1.12.0
tfx-bsl==1.12.0
thinc==8.1.5
threadpoolctl==3.1.0
tifffile==2022.10.10
toml==0.10.2
tomli==2.0.1
toolz==0.12.0
torch @ https://download.pytorch.org/whl/cu116/torch-1.13.0%2Bcu116-cp38-cp38-linux_x86_64.whl
torchaudio @ https://download.pytorch.org/whl/cu116/torchaudio-0.13.0%2Bcu116-cp38-cp38-linux_x86_64.whl
torchsummary==1.5.1
torchtext==0.14.0
torchvision @ https://download.pytorch.org/whl/cu116/torchvision-0.14.0%2Bcu116-cp38-cp38-linux_x86_64.whl
tornado==6.0.4
tqdm==4.64.1
traitlets==5.7.1
tweepy==3.10.0
typeguard==2.7.1
typer==0.7.0
typing_extensions==4.4.0
tzlocal==1.5.1
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.9.0
wasabi==0.10.1
wcwidth==0.2.5
webargs==8.2.0
webencodings==0.5.1
websocket-client==1.4.2
Werkzeug==1.0.1
widgetsnbextension==3.6.1
wordcloud==1.8.2.2
wrapt==1.14.1
xarray==2022.12.0
xarray-einstats==0.4.0
xgboost==0.90
xkit==0.0.0
xlrd==1.2.0
xlwt==1.3.0
yarl==1.8.2
yellowbrick==1.5
zict==2.2.0
zipp==3.11.0
zstandard==0.19.0
gaikwadrahul8 commented 1 year ago

Hi, @michaelwsherman

Apologize for the inconvenience and Yes, currently we're experiencing that issue with orchestration and our team is making some code level changes so once it's done issue will be resolved and I'll update the same here once I got the confirmation from our concerned team about issue resolution

Thank you!

gaikwadrahul8 commented 1 year ago

Hi, @michaelwsherman

I was able to replicate the issue without error and for that you'll have to uninstall shapely by adding this line of code !pip uninstall shapely -y and even it's updated in google colab nootbook tutorials also, for your reference I have added gist files penguin_simple and penguin_tfdv

Could you please try and confirm, Is it working both the tutorials as expected from your end ?

Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved ?

Thank you!

michaelwsherman commented 1 year ago

Can confirm both gists work, restarting where instructed to.

Note this applies to more of the TFX tutorials than these two, I just stopped checking after these two. I'd suggest fixing the others before resolving this issue.

gaikwadrahul8 commented 1 year ago

Hi, @michaelwsherman

I checked all the TFX tutorials under the Tutorials section and they're working as expected after uninstalling shapely package and also it's updated in all those tutorials notebook except one tutorial and yes as mentioned in tutorials section Did you restart the runtime?, you've to restart runtime to work everything as expected.

@lego0901, I found one tutorial where new changes are not reflected so could you please look into this tutorial where uninstall shapely section is missing ?

Thank you!

lego0901 commented 1 year ago

Thanks for catching the missing notebook!

I think PR you made will solve the issue correctly! And I guess we should wait for the TensorFlow folks to response.

I appreciate your consideration!

gaikwadrahul8 commented 1 year ago

Hi, @lego0901

You're welcome and yes, I have submitted PR for that so we'll wait for response from TensorFlow folks.

Thank you!

github-actions[bot] commented 1 year ago

This issue has been marked stale because it has no recent activity since 14 days. It will be closed if no further activity occurs. Thank you.

google-ml-butler[bot] commented 1 year ago

Are you satisfied with the resolution of your issue? Yes No