Open bruno-vilhena opened 1 year ago
Would it be possible to add support for batch inserts in BigQuery emulator?
Essentially support for this type of insert, instead of streaming inserts, ex:
Job job = null; WriteChannelConfiguration configuration = WriteChannelConfiguration.newBuilder(tableId) .setFormatOptions(FormatOptions.json()) .setCreateDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED) .setSchema(contentTableManager.getSchema()) .build(); TableDataWriteChannel channel = null; try { channel = bigQueryClient.writer(configuration); String jsonBatch = messages .stream() .map(message -> { try { return objectMapper.writeValueAsString(message); } catch (Exception e) { LOGGER.error("Batch insert failed", e); throw new RuntimeException(e); } }).collect(Collectors.joining("\n")); channel.write(ByteBuffer.wrap(jsonBatch.getBytes(StandardCharsets.UTF_8))); } catch (Exception e) { LOGGER.error("Batch insert failed", e); throw new RuntimeException(e); } finally { try { channel.close(); } catch (IOException e) { LOGGER.error("Batch insert failed", e); throw new RuntimeException(e); } }
I'm also looking for this.
+1
Would it be possible to add support for batch inserts in BigQuery emulator?
Essentially support for this type of insert, instead of streaming inserts, ex: