Plans to support batch write API?

[seglo]

I was testing using the preview batch write API and noticed an improved throughput when working with a Cloud Spanner instance. However, when running the same code against the emulator as part of a larger integration test the emulator returns a NOT IMPLEMENTED gRPC error. I understand the emulator is a very limited tool to use, but I was expecting it to at least be able to accept the batch write request without an error.

My only option is to execute different persistence code for emulated vs cloud spanner environments which defeats the purpose of the integration test IMO.

Latest versions at the time of this post: Emulator version: gcr.io/cloud-spanner-emulator/emulator:1.5.17 Google Cloud Spanner Java library "com.google.cloud" % "google-cloud-spanner-jdbc" % "6.68.1"

Sensitive package names censored.

com.google.api.gax.rpc.UnimplementedException: io.grpc.StatusRuntimeException: UNIMPLEMENTED
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:108)
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
    at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:82)
    at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:84)
    at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:148)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:570)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: java.lang.RuntimeException: Asynchronous task failed
        at com.google.api.gax.rpc.ServerStreamIterator.hasNext(ServerStreamIterator.java:105)
        at scala.collection.convert.JavaCollectionWrappers$JIteratorWrapper.hasNext(JavaCollectionWrappers.scala:46)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1303)
        at XXX.BigQueryToSpannerWriter.$anonfun$writeToSpannerBatchWriteApi$1(BigQueryToSpannerWriter.scala:118)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
        at org.apache.pekko.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:73)
        at org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:110)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
        at org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:110)
        at org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59)
        at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedThreadPool$TimingRunnable.run(ExecutorInstrumentation.scala:466)
        ... 3 common frames omitted
Caused by: io.grpc.StatusRuntimeException: UNIMPLEMENTED
    at io.grpc.Status.asRuntimeException(Status.java:537)
    ... 21 common frames omitted

[kberezin-nshl]

@seglo I've done a super naive (but working) implementation in this commit in our fork.

Feel free to use that and build your own version, or if you're comfortable with using binaries built by 3rd party, you can use the emulator version which we built for this release

[bshaffer]

@kberezin-nshl for some reason, running the emulator with your container fails with the following error:

runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f5a498e1e2c m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: g 0: unknown pc 0x7f5a498e1e2c
stack: frame={sp:0x7ffd4419f860, fp:0x0} stack=[0x7ffd439a0cf0,0x7ffd4419fd00)

[kberezin-nshl]

@bshaffer seems related to this issue.

It runs fine for me under Mac OS X and also under linux in our CI environment.