Closed mshearer0 closed 4 years ago
@drcat101 Any suggestions?
Resolved by adding ipython extra package:
setup_options.extra_packages = [ './tensorflow_data_validation-0.23.0-cp37-cp37m-manylinux2010_x86_64.whl', 'ipython-7.17.0-py3-none-any.whl']
@mshearer0 thank you for adding the solution.
@mshearer0 Can you please share your statsgen
setup? Are you visualizing the results from the components after the Dataflow Runner completes the execution?
@hanneshapke as per Chapter4:
import tensorflow_data_validation as tfdv
from apache_beam.options.pipeline_options import ( PipelineOptions, GoogleCloudOptions, StandardOptions)
options = PipelineOptions() google_cloud_options = options.view_as(GoogleCloudOptions) google_cloud_options.project = 'xxx' google_cloud_options.job_name = 'beamtfdv' google_cloud_options.staging_location = 'gs://xxx/staging' google_cloud_options.temp_location = 'gs://xxx/tmp' google_cloud_options.region='europe-west1' options.view_as(StandardOptions).runner = 'DataflowRunner'
from apache_beam.options.pipeline_options import SetupOptions
setup_options = options.view_as(SetupOptions) setup_options.extra_packages = [ 'tensorflow_data_validation-0.23.0-cp37-cp37m-manylinux2010_x86_64.whl', 'ipython-7.18.1-py3-none-any.whl']
data_set_path = 'gs://xxx/consumer-complaints.tfrecords' output_path = 'gs://xxx/' tfdv.generate_statistics_from_tfrecord(data_set_path, output_path=output_path, pipeline_options=options)
Thank you @mshearer0 . I wonder if iPython was previously pre-installed in the dataflow instances. I am glad the setup is working for you now.
I get the following when trying to generate_statistics using dataflow:
File "/usr/local/lib/python3.7/site-packages/tensorflow_data_validation/utils/display_util.py", line 39, in
'tensorflow-data-validation[visualization]": {}'.format(e))
ImportError: To use visualization features, make sure ipython is installed, or install TFDV using "pip install tensorflow-data-validation[visualization]": No module named 'IPython'
Pip list shows tensorflow==2.3.0, tensorflow-data-validation==0.23.0, ipython ==7.17.0 as per https://pypi.org/project/tensorflow-data-validation/
I'm using: tensorflow_data_validation-0.23.0-cp37-cp37m-manylinux2010_x86_64.whl
Works fine with DirectRunner