Background: Trying to run embedding_extraction/run.sh
Issue: Dataflow job does not write embeddings to bucket
GCP Log
I 2019-09-19T16:01:52.195809Z RuntimeError
I 2019-09-19T16:01:52.195945Z :
I 2019-09-19T16:01:52.195989Z module compiled against API version 0xc but this version of numpy is 0xb
I 2019-09-19T16:01:52.196181Z Traceback (most recent call last):
I 2019-09-19T16:01:52.196234Z File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
I 2019-09-19T16:01:52.196333Z
I 2019-09-19T16:01:52.196370Z "main", fname, loader, pkg_name)
I 2019-09-19T16:01:52.196409Z File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
I 2019-09-19T16:01:52.196478Z
I 2019-09-19T16:01:52.196510Z exec code in run_globals
I 2019-09-19T16:01:52.196547Z File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/start.py", line 26, in
I 2019-09-19T16:01:52.196631Z
I 2019-09-19T16:01:52.196664Z from apache_beam.runners.worker import logger
I 2019-09-19T16:01:52.196705Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/init.py", line 98, in
I 2019-09-19T16:01:52.196793Z
I 2019-09-19T16:01:52.196825Z from apache_beam import io
I 2019-09-19T16:01:52.196864Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/init.py", line 29, in
I 2019-09-19T16:01:52.196932Z
I 2019-09-19T16:01:52.196970Z from apache_beam.io.parquetio import
I 2019-09-19T16:01:52.197009Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/parquetio.py", line 45, in
I 2019-09-19T16:01:52.197075Z
I 2019-09-19T16:01:52.197107Z import pyarrow as pa
I 2019-09-19T16:01:52.197143Z File "/usr/local/lib/python2.7/dist-packages/pyarrow/init.py", line 49, in
I 2019-09-19T16:01:52.197207Z
I 2019-09-19T16:01:52.197238Z from pyarrow.lib import cpu_count, set_cpu_count
I 2019-09-19T16:01:52.197275Z File "pyarrow/lib.pyx", line 40, in init pyarrow.lib
I 2019-09-19T16:01:52.197407Z ImportError
I 2019-09-19T16:01:52.197446Z :
I 2019-09-19T16:01:52.197476Z numpy.core.multiarray failed to import
I 2019-09-19T16:01:52.230228Z RuntimeError
I 2019-09-19T16:01:52.230374Z :
I 2019-09-19T16:01:52.230432Z module compiled against API version 0xc but this version of numpy is 0xb
I 2019-09-19T16:01:52.230674Z Traceback (most recent call last):
I 2019-09-19T16:01:52.230732Z File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
I 2019-09-19T16:01:52.230843Z
I 2019-09-19T16:01:52.230881Z "main", fname, loader, pkg_name)
I 2019-09-19T16:01:52.230929Z File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
I 2019-09-19T16:01:52.231003Z
I 2019-09-19T16:01:52.231038Z exec code in run_globals
I 2019-09-19T16:01:52.231073Z File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/start.py", line 26, in
I 2019-09-19T16:01:52.231158Z
I 2019-09-19T16:01:52.231195Z from apache_beam.runners.worker import logger
I 2019-09-19T16:01:52.231232Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/init.py", line 98, in
I 2019-09-19T16:01:52.231310Z
I 2019-09-19T16:01:52.231346Z from apache_beam import io
I 2019-09-19T16:01:52.231381Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/init.py", line 29, in
I 2019-09-19T16:01:52.231447Z
I 2019-09-19T16:01:52.231482Z from apache_beam.io.parquetio import
I 2019-09-19T16:01:52.231517Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/parquetio.py", line 45, in
I 2019-09-19T16:01:52.231598Z
I 2019-09-19T16:01:52.231637Z import pyarrow as pa
I 2019-09-19T16:01:52.231670Z File "/usr/local/lib/python2.7/dist-packages/pyarrow/init.py", line 49, in
I 2019-09-19T16:01:52.231739Z
I 2019-09-19T16:01:52.231774Z from pyarrow.lib import cpu_count, set_cpu_count
I 2019-09-19T16:01:52.231808Z File "pyarrow/lib.pyx", line 40, in init pyarrow.lib
I 2019-09-19T16:01:52.231904Z ImportError
I 2019-09-19T16:01:52.231952Z :
I 2019-09-19T16:01:52.231981Z numpy.core.multiarray failed to import
I 2019-09-19T16:01:52.296778Z /usr/bin/python failed with exit status 1
I 2019-09-19T16:01:52.322414Z /usr/bin/python failed with exit status 1
Background: Trying to run embedding_extraction/run.sh Issue: Dataflow job does not write embeddings to bucket GCP Log I 2019-09-19T16:01:52.195809Z RuntimeError I 2019-09-19T16:01:52.195945Z :
I 2019-09-19T16:01:52.196631Z
I 2019-09-19T16:01:52.196793Z
I 2019-09-19T16:01:52.196932Z
I 2019-09-19T16:01:52.197075Z
I 2019-09-19T16:01:52.197207Z
I 2019-09-19T16:01:52.231158Z
I 2019-09-19T16:01:52.231310Z
I 2019-09-19T16:01:52.231447Z
I 2019-09-19T16:01:52.231517Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/parquetio.py", line 45, in
I 2019-09-19T16:01:52.231598Z
I 2019-09-19T16:01:52.231739Z
I 2019-09-19T16:01:52.195989Z module compiled against API version 0xc but this version of numpy is 0xb I 2019-09-19T16:01:52.196181Z Traceback (most recent call last): I 2019-09-19T16:01:52.196234Z File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main I 2019-09-19T16:01:52.196333Z
I 2019-09-19T16:01:52.196370Z "main", fname, loader, pkg_name) I 2019-09-19T16:01:52.196409Z File "/usr/lib/python2.7/runpy.py", line 72, in _run_code I 2019-09-19T16:01:52.196478Z
I 2019-09-19T16:01:52.196510Z exec code in run_globals I 2019-09-19T16:01:52.196547Z File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/start.py", line 26, in
I 2019-09-19T16:01:52.196664Z from apache_beam.runners.worker import logger I 2019-09-19T16:01:52.196705Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/init.py", line 98, in
I 2019-09-19T16:01:52.196825Z from apache_beam import io I 2019-09-19T16:01:52.196864Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/init.py", line 29, in
I 2019-09-19T16:01:52.196970Z from apache_beam.io.parquetio import I 2019-09-19T16:01:52.197009Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/parquetio.py", line 45, in
I 2019-09-19T16:01:52.197107Z import pyarrow as pa I 2019-09-19T16:01:52.197143Z File "/usr/local/lib/python2.7/dist-packages/pyarrow/init.py", line 49, in
I 2019-09-19T16:01:52.197238Z from pyarrow.lib import cpu_count, set_cpu_count I 2019-09-19T16:01:52.197275Z File "pyarrow/lib.pyx", line 40, in init pyarrow.lib I 2019-09-19T16:01:52.197407Z ImportError I 2019-09-19T16:01:52.197446Z :
I 2019-09-19T16:01:52.197476Z numpy.core.multiarray failed to import I 2019-09-19T16:01:52.230228Z RuntimeError I 2019-09-19T16:01:52.230374Z :
I 2019-09-19T16:01:52.230432Z module compiled against API version 0xc but this version of numpy is 0xb I 2019-09-19T16:01:52.230674Z Traceback (most recent call last): I 2019-09-19T16:01:52.230732Z File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main I 2019-09-19T16:01:52.230843Z
I 2019-09-19T16:01:52.230881Z "main", fname, loader, pkg_name) I 2019-09-19T16:01:52.230929Z File "/usr/lib/python2.7/runpy.py", line 72, in _run_code I 2019-09-19T16:01:52.231003Z
I 2019-09-19T16:01:52.231038Z exec code in run_globals I 2019-09-19T16:01:52.231073Z File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/start.py", line 26, in
I 2019-09-19T16:01:52.231195Z from apache_beam.runners.worker import logger I 2019-09-19T16:01:52.231232Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/init.py", line 98, in
I 2019-09-19T16:01:52.231346Z from apache_beam import io I 2019-09-19T16:01:52.231381Z File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/init.py", line 29, in
I 2019-09-19T16:01:52.231482Z from apache_beam.io.parquetio import
I 2019-09-19T16:01:52.231637Z import pyarrow as pa I 2019-09-19T16:01:52.231670Z File "/usr/local/lib/python2.7/dist-packages/pyarrow/init.py", line 49, in
I 2019-09-19T16:01:52.231774Z from pyarrow.lib import cpu_count, set_cpu_count I 2019-09-19T16:01:52.231808Z File "pyarrow/lib.pyx", line 40, in init pyarrow.lib I 2019-09-19T16:01:52.231904Z ImportError I 2019-09-19T16:01:52.231952Z :
I 2019-09-19T16:01:52.231981Z numpy.core.multiarray failed to import I 2019-09-19T16:01:52.296778Z /usr/bin/python failed with exit status 1 I 2019-09-19T16:01:52.322414Z /usr/bin/python failed with exit status 1