perretv
commented
1 year ago We also observed this issue in my team, this led us to abandon tfio for querying a BigQuery table as this grpc assertion failed error was consistently happening.
It turns out that you can build a tensorflow dataset with a native google cloud BigQuery client, and it will not suffer from this problem:
from typing import Generator
import tensorflow as tf
from google.cloud import bigquery
client = bigquery.Client()
query_obj = client.query("SELECT * FROM project_id.dataset.table_id")
rows = query_obj.result()
def bigquery_generator() -> Generator[dict[str, tf.Tensor], None, None]:
"""Yield the BigQuery query results in tensor."""
for row in rows:
yield {
"column1": tf.convert_to_tensor(row["column1"]),
"column2": tf.convert_to_tensor(row["column2"]),
}
tf_dataset = tf.data.Dataset.from_generator(
bigquery_generator,
output_signature=(
tf.TensorSpec(shape=(), dtype=tf.string),
tf.TensorSpec(shape=(), dtype=tf.string),
),
)Obviously, you will need to adapt the code to match the table data schema :)
nishprabhu
mvidyasagar-sc
manish181192
When using
BigQueryClient, we get sporadic failures of the form:while streaming the data. I unfortunately can't share the code or dataset, but I can say that: