goccy / bigquery-emulator

BigQuery emulator server implemented in Go
MIT License
845 stars 108 forks source link

Storage API Avro encoding does not work with nullable fields #252

Open ianb-pomelo opened 11 months ago

ianb-pomelo commented 11 months ago

I've created a table with nullable columns but when I use the Storage API to stream read the table, if there are rows that don't have data in all column, the avro encoding fails with value does not match its schema: long: expected: Go numeric; received: <nil>

Example: Create Table

StandardTableDefinition.of(
  Schema.of(
    Field.newBuilder(ENCOUNTER_ID_COLUMN, INT64).setMode(NULLABLE).build(),
    Field.newBuilder(PATIENT_ID_COLUMN, INT64).setMode(NULLABLE).build(),
    Field.newBuilder(APPOINTMENT_ID_COLUMN, INT64)
      .setMode(NULLABLE)
      .setDefaultValueExpression("NULL")
      .build(),
    Field.newBuilder(ENCOUNTER_DATE_COLUMN, DATE).setMode(NULLABLE).build()
  )
)

Insert Row:

RowToInsert.of(
  mapOf(
    ENCOUNTER_ID_COLUMN to 1234,
    PATIENT_ID_COLUMN to 1234,
    APPOINTMENT_ID_COLUMN to null,
    ENCOUNTER_DATE_COLUMN to null
  )
)

Throws: io.grpc.StatusRuntimeException: UNKNOWN: failed to encode binary from go value: cannot encode binary record "testproject.athena_dataviewer.encounters" field "APPOINTMENTID": value does not match its schema: long: expected: Go numeric; received: <nil>