googleapis / python-bigquery-storage

Apache License 2.0
116 stars 46 forks source link

writer.AppendRowsStream missing from v2 bigquery_storage #752

Closed CharlesFr closed 7 months ago

CharlesFr commented 9 months ago

In the V1 client, writer was importable as shown below, which allowed for creation of a AppendRowsStream row stream objects. However, after migrating from V2 it seems writer is no longer available which is a breaking change that is not covered in the migration guide. Could you please let me know what class in V2 replaces AppendRowsStream from V1?

Legacy import:

from google.cloud.bigquery_storage_v1 import writer, BigQueryWriteClient

append_rows_stream = writer.AppendRowsStream(
    self.write_client, request_template
)

New import (fails):

from google.cloud.bigquery_storage import writer

And here's the context in which it's used in my codebase:

def setup_rows_stream(self):
    # Create a template with fields needed for the first request.
    request_template = AppendRowsRequest()

    # The initial request must contain the stream name.
    request_template.write_stream = self.write_stream.name

    # So that BigQuery knows how to parse the serialized_rows, generate a
    # protocol buffer representation of your message descriptor.
    proto_schema = ProtoSchema()
    proto_descriptor = descriptor_pb2.DescriptorProto()
    self.proto_obj.DESCRIPTOR.CopyToProto(proto_descriptor)
    proto_schema.proto_descriptor = proto_descriptor
    proto_data = AppendRowsRequest.ProtoData()
    proto_data.writer_schema = proto_schema
    request_template.proto_rows = proto_data

    append_rows_stream = writer.AppendRowsStream(
        self.write_client, request_template
    )

    return append_rows_stream

Thank you

xionams commented 9 months ago

"Could you please let me know what class in V2 replaces AppendRowsStream from V1?"

As far as I understand, the new "way" to write is by using the AppendRows method instead of AppendRowsStream to write.

You can read about AppendRows here: https://cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1#google.cloud.bigquery.storage.v1.BigQueryWrite.AppendRows

Linchin commented 8 months ago

Thank you @CharlesFr for raising the issue. Could you tell me more about what you mean by V2, and why you are not using google.cloud.bigquery_storage_v1 for imports? All the APIs you mentioned are available there.