Closed tarrade closed 4 years ago
issue fixed: need extra package+config. Musmatch betwen the VM and TPU cluster
Hi, can you tell me what package and config did you use? I have the same problem.
Hi @ZyingHang,
In my case I was using Tensorflow 2.2.0 (didn't try with TF 2.3.0). Since then, the GCPdocumentation was updated: https://cloud.google.com/tpu/docs/version-switching
you need this:
import tensorflow as tf
from cloud_tpu_client import Client
c = Client()
c.configure_tpu_version(tf.__version__, restart_type='ifNeeded')
This be update the TF version on the TPU cluster is you use a different version.
Hope this help
if you have issue you can connect to me on twitter https://twitter.com/fabtar and I can add you in a thread with TPU/GCP experts that helped me solving this issue
mismatch Tensorflow version CPU and TPU
dataset.map(pp.parse_tfrecord_glue_files, num_parallel_calls=tf.data.experimental.AUTOTUNE) File "/root/.local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1628, in map preserve_cardinality=True) File "/root/.local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 4050, in init **self._flat_structure) File "/root/.local/lib/python3.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 4692, in parallel_map_dataset _ops.raise_from_not_ok_status(e, name) File "/root/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.NotFoundError: 'ParallelInterleaveDatasetV3' is neither a type of a primitive operation nor a name of a function registered in binary running on n-c3e638d3-w-0. One possible root cause is the client and server binaries are not built with the same version. Please make sure the operation or function is registered in the binary running in this process. [Op:ParallelMapDataset]