goccy / bigquery-emulator

BigQuery emulator server implemented in Go
MIT License
846 stars 108 forks source link

Support for batch inserts #228

Open bruno-vilhena opened 1 year ago

bruno-vilhena commented 1 year ago

Would it be possible to add support for batch inserts in BigQuery emulator?

Essentially support for this type of insert, instead of streaming inserts, ex:

Job job = null;
WriteChannelConfiguration configuration =
        WriteChannelConfiguration.newBuilder(tableId)
                .setFormatOptions(FormatOptions.json())
                .setCreateDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED)
                .setSchema(contentTableManager.getSchema())
                .build();
TableDataWriteChannel channel = null;
try {
    channel = bigQueryClient.writer(configuration);
    String jsonBatch = messages
            .stream()
            .map(message -> {
                try {
                    return objectMapper.writeValueAsString(message);
                } catch (Exception e) {
                    LOGGER.error("Batch insert failed", e);
                    throw new RuntimeException(e);
                }
            }).collect(Collectors.joining("\n"));
    channel.write(ByteBuffer.wrap(jsonBatch.getBytes(StandardCharsets.UTF_8)));
} catch (Exception e) {
    LOGGER.error("Batch insert failed", e);
    throw new RuntimeException(e);
} finally {
    try {
        channel.close();
    } catch (IOException e) {
        LOGGER.error("Batch insert failed", e);
        throw new RuntimeException(e);
    }
}
augustobreno commented 6 months ago

I'm also looking for this.

ismailsimsek commented 1 month ago

+1