Seen them flaky once on Linux. I don’t think it’s frequent but the error looks very bad so reporting
16:47:17.697 [HttpServiceWithPostgresIntTest-akka.actor.default-dispatcher-9] ERROR com.daml.http.ContractsFetch - contractsIo3 exception: java.sql.BatchUpdateException: Batch entry 0
INSERT INTO some_nice_prefix_contract
VALUES ('001424d7f65f0cfff8f8150f3eb825a5f8c0479bd73678790e6099beeb05d53c73', 1, 'null'::jsonb, NULL, '{"observers":[],"issuer":"Alice","amount":333.33,"currency":"GBP","owner":"Alice"}'::jsonb, ?, ?, '')
ON CONFLICT (contract_id) DO NOTHING
was aborted: ERROR: insert or update on table "some_nice_prefix_contract" violates foreign key constraint "some_nice_prefix_contract_tpid_fkey"
Detail: Key (tpid)=(1) is not present in table "some_nice_prefix_template_id". Call getNextException to see other errors in the batch., state: 23503, context: {ledger_id: "HttpServiceWithPostgresIntTest", cmd: "GetActiveContractsRequest(OneAnd(TemplateId(None,Iou,Iou),Set()),Map(amount -> 111.11))", request_id: "073e5f39-8b2c-427f-b88a-675ce643d4be", instance_uuid: "20be5118-0449-42f3-9b1f-090f88b8d400", application_id: "test", read_as: "List()", act_as: "List(Alice)"}
16:47:17.700 [HttpServiceWithPostgresIntTest-akka.actor.default-dispatcher-8] ERROR com.daml.http.Endpoints - Source failed, context: {ledger_id: "HttpServiceWithPostgresIntTest", cmd: "GetActiveContractsRequest(OneAnd(TemplateId(None,Iou,Iou),Set()),Map(amount -> 111.11))", request_id: "073e5f39-8b2c-427f-b88a-675ce643d4be", instance_uuid: "20be5118-0449-42f3-9b1f-090f88b8d400", application_id: "test", read_as: "List()", act_as: "List(Alice)"}
java.sql.BatchUpdateException: Batch entry 0
INSERT INTO some_nice_prefix_contract
VALUES ('001424d7f65f0cfff8f8150f3eb825a5f8c0479bd73678790e6099beeb05d53c73', 1, 'null'::jsonb, NULL, '{"observers":[],"issuer":"Alice","amount":333.33,"currency":"GBP","owner":"Alice"}'::jsonb, ?, ?, '')
ON CONFLICT (contract_id) DO NOTHING
was aborted: ERROR: insert or update on table "some_nice_prefix_contract" violates foreign key constraint "some_nice_prefix_contract_tpid_fkey"
Detail: Key (tpid)=(1) is not present in table "some_nice_prefix_template_id". Call getNextException to see other errors in the batch.
at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:169)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2286)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:521)
at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:870)
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:893)
at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1644)
at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
at doobie.free.KleisliInterpreter$PreparedStatementInterpreter.$anonfun$executeBatch$2(kleisliinterpreter.scala:946)
at doobie.free.KleisliInterpreter.$anonfun$primitive$2(kleisliinterpreter.scala:109)
at evalOn @ doobie.util.transactor$Transactor$fromDataSource$FromDataSourceUnapplied.$anonfun$apply$13(transactor.scala:294)
at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:167)
at $anonfun$tailRecM$1 @ doobie.free.KleisliInterpreter$ConnectionInterpreter.$anonfun$bracketCase$28(kleisliinterpreter.scala:750)
at $anonfun$tailRecM$1 @ doobie.free.KleisliInterpreter$ConnectionInterpreter.$anonfun$bracketCase$28(kleisliinterpreter.scala:750)
at $anonfun$tailRecM$1 @ doobie.free.KleisliInterpreter$ConnectionInterpreter.$anonfun$bracketCase$28(kleisliinterpreter.scala:750)
at evalOn @ doobie.util.transactor$Transactor$fromDataSource$FromDataSourceUnapplied.$anonfun$apply$13(transactor.scala:294)
at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:167)
at $anonfun$tailRecM$1 @ doobie.free.KleisliInterpreter$ConnectionInterpreter.$anonfun$bracketCase$28(kleisliinterpreter.scala:750)
at $anonfun$tailRecM$1 @ doobie.free.KleisliInterpreter$ConnectionInterpreter.$anonfun$bracketCase$28(kleisliinterpreter.scala:750)
at $anonfun$tailRecM$1 @ doobie.free.KleisliInterpreter$ConnectionInterpreter.$anonfun$bracketCase$28(kleisliinterpreter.scala:750)
at evalOn @ doobie.util.transactor$Transactor$fromDataSource$FromDataSourceUnapplied.$anonfun$apply$13(transactor.scala:294)
Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on table "some_nice_prefix_contract" violates foreign key constraint "some_nice_prefix_contract_tpid_fkey"
Detail: Key (tpid)=(1) is not present in table "some_nice_prefix_template_id".
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2285)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:521)
at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:870)
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:893)
at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1644)
at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
at doobie.free.KleisliInterpreter$PreparedStatementInterpreter.$anonfun$executeBatch$2(kleisliinterpreter.scala:946)
at doobie.free.KleisliInterpreter.$anonfun$primitive$2(kleisliinterpreter.scala:109)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104)
at cats.effect.internals.IORunLoop$.restartCancelable(IORunLoop.scala:51)
at cats.effect.internals.IOBracket$BracketStart.run(IOBracket.scala:100)
at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67)
at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35)
at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90)
at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90)
at cats.effect.internals.Trampoline.execute(Trampoline.scala:43)
at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42)
at cats.effect.internals.IOBracket$BracketStart.apply(IOBracket.scala:80)
at cats.effect.internals.IOBracket$BracketStart.apply(IOBracket.scala:58)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:183)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422)
at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
at cats.effect.internals.PoolUtils$$anon$2$$anon$3.run(PoolUtils.scala:52)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
16:47:17.702 [HttpServiceWithPostgresIntTest-akka.actor.default-dispatcher-10] INFO com.daml.http.util.TestUtil$ - ---- response data: {"result":[],"errors":["Endpoints.ServerError: java.sql.BatchUpdateException: Batch entry 0 \n INSERT INTO some_nice_prefix_contract\n VALUES ('001424d7f65f0cfff8f8150f3eb825a5f8c0479bd73678790e6099beeb05d53c73', 1, 'null'::jsonb, NULL, '{\"observers\":[],\"issuer\":\"Alice\",\"amount\":333.33,\"currency\":\"GBP\",\"owner\":\"Alice\"}'::jsonb, ?, ?, '')\n ON CONFLICT (contract_id) DO NOTHING\n was aborted: ERROR: insert or update on table \"some_nice_prefix_contract\" violates foreign key constraint \"some_nice_prefix_contract_tpid_fkey\"\n Detail: Key (tpid)=(1) is not present in table \"some_nice_prefix_template_id\". Call getNextException to see other errors in the batch."],"status":501}
16:47:17.710 [HttpServiceWithPostgresIntTest-akka.actor.default-dispatcher-9] INFO com.daml.http.util.TestUtil$ - ---- response data: {"result":[{"agreementText":"","contractId":"00b2ccb449c77d08ac61681180166e93444c6a5acc7b608b76c9216cd5fb8d91ab","observers":[],"payload":{"amount":"111.11","currency":"EUR","issuer":"Alice","observers":[],"owner":"Alice"},"signatories":["Alice"],"templateId":"456826ec357defbe50a0f2be2152b40dbe7f1a5346a6323b17b9b4e4b34bc630:Iou:Iou"}],"status":200}
16:47:17.741 [HttpServiceWithPostgresIntTest-akka.actor.default-dispatcher-11] INFO c.daml.http.HttpServiceTestFixture$ - Shutting down http service
16:47:17.741 [HttpServiceWithPostgresIntTest-akka.actor.default-dispatcher-11] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-37 - Shutdown initiated...
16:47:17.751 [HttpServiceWithPostgresIntTest-akka.actor.default-dispatcher-11] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-37 - Shutdown completed.
16:47:17.752 [HttpServiceWithPostgresIntTest-akka.actor.default-dispatcher-8] INFO c.d.p.a.services.ApiCommandService - Shutting down Command Service, context: {participantId: "sandbox-participant"}
16:47:17.752 [HttpServiceWithPostgresIntTest-akka.actor.default-dispatcher-8] INFO c.d.p.a.services.tracking.TrackerMap - Shutting down 1 trackers, context: {participantId: "sandbox-participant"}
- query with query, can use number or string for numeric field *** FAILED *** (723 milliseconds)
{"result":[{"agreementText":"","contractId":"00b2ccb449c77d08ac61681180166e93444c6a5acc7b608b76c9216cd5fb8d91ab","observers":[],"payload":{"amount":"111.11","currency":"EUR","issuer":"Alice","observers":[],"owner":"Alice"},"signatories":["Alice"],"templateId":"456826ec357defbe50a0f2be2152b40dbe7f1a5346a6323b17b9b4e4b34bc630:Iou:Iou"}],"status":200} was not equal to {"errors":["Endpoints.ServerError: java.sql.BatchUpdateException: Batch entry 0 \n INSERT INTO some_nice_prefix_contract\n VALUES ('001424d7f65f0cfff8f8150f3eb825a5f8c0479bd73678790e6099beeb05d53c73', 1, 'null'::jsonb, NULL, '{\"observers\":[],\"issuer\":\"Alice\",\"amount\":333.33,\"currency\":\"GBP\",\"owner\":\"Alice\"}'::jsonb, ?, ?, '')\n ON CONFLICT (contract_id) DO NOTHING\n was aborted: ERROR: insert or update on table \"some_nice_prefix_contract\" violates foreign key constraint \"some_nice_prefix_contract_tpid_fkey\"\n Detail: Key (tpid)=(1) is not present in table \"some_nice_prefix_template_id\". Call getNextException to see other errors in the batch."],"result":[],"status":501},
inside List({"result":[{"agreementText":"","contractId":"00b2ccb449c77d08ac61681180166e93444c6a5acc7b608b76c9216cd5fb8d91ab","observers":[],"payload":{"amount":"111.11","currency":"EUR","issuer":"Alice","observers":[],"owner":"Alice"},"signatories":["Alice"],"templateId":"456826ec357defbe50a0f2be2152b40dbe7f1a5346a6323b17b9b4e4b34bc630:Iou:Iou"}],"status":200}, {"errors":["Endpoints.ServerError: java.sql.BatchUpdateException: Batch entry 0 \n INSERT INTO some_nice_prefix_contract\n VALUES ('001424d7f65f0cfff8f8150f3eb825a5f8c0479bd73678790e6099beeb05d53c73', 1, 'null'::jsonb, NULL, '{\"observers\":[],\"issuer\":\"Alice\",\"amount\":333.33,\"currency\":\"GBP\",\"owner\":\"Alice\"}'::jsonb, ?, ?, '')\n ON CONFLICT (contract_id) DO NOTHING\n was aborted: ERROR: insert or update on table \"some_nice_prefix_contract\" violates foreign key constraint \"some_nice_prefix_contract_tpid_fkey\"\n Detail: Key (tpid)=(1) is not present in table \"some_nice_prefix_template_id\". Call getNextException to see other errors in the batch."],"result":[],"status":501}) (AbstractHttpServiceIntegrationTest.scala:799)
org.scalatest.exceptions.TestFailedException:
at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:402)
at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7021)
at com.daml.http.AbstractHttpServiceIntegrationTest$$anonfun$$nestedInanonfun$new$55$1.applyOrElse(AbstractHttpServiceIntegrationTest.scala:799)
at com.daml.http.AbstractHttpServiceIntegrationTest$$anonfun$$nestedInanonfun$new$55$1.applyOrElse(AbstractHttpServiceIntegrationTest.scala:798)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
at org.scalatest.Inside$.insideWithPos(Inside.scala:169)
at org.scalatest.Inside.inside(Inside.scala:105)
at org.scalatest.Inside.inside$(Inside.scala:104)
at com.daml.http.AbstractHttpServiceIntegrationTest.inside(AbstractHttpServiceIntegrationTest.scala:607)
at com.daml.http.AbstractHttpServiceIntegrationTest.$anonfun$new$55(AbstractHttpServiceIntegrationTest.scala:798)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:434)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:53)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Seen them flaky once on Linux. I don’t think it’s frequent but the error looks very bad so reporting
Attached the full log file