search

paypal / squbs

Akka Streams & Akka HTTP for Large-Scale Production Deployments
http://paypal.github.io/squbs
Apache License 2.0
1.43k stars 237 forks source link

Fix intermittent failure in travis CI #569

Open sebady opened 6 years ago

sebady commented 6 years ago

Travis Intermittent test failure in BroadcastBufferSpec

akara commented 5 years ago

We found another build issue to look at:

[info]      java.util.NoSuchElementException:
[info]       at scala.collection.LinearSeqOptimized.last(LinearSeqOptimized.scala:150)
[info]       at scala.collection.LinearSeqOptimized.last$(LinearSeqOptimized.scala:149)
[info]       at scala.collection.immutable.List.last(List.scala:89)
[info]       at org.squbs.stream.PerpetualStreamMatValueSpec.$anonfun$new$8(PerpetualStreamMatValueSpec.scala:77)
[info]       at org.squbs.stream.PerpetualStreamMatValueSpec.$anonfun$new$8$adapted(PerpetualStreamMatValueSpec.scala:74)
[info]       at org.squbs.stream.PerpetualStreamMatValueSpecHelper$.useSystem(PerpetualStreamMatValueSpec.scala:149)
[info]       at org.squbs.stream.PerpetualStreamMatValueSpec.$anonfun$new$7(PerpetualStreamMatValueSpec.scala:74)
[info]       at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]       at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]       at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
akara commented 5 years ago

One additional failure to look at:

[info] - should fail fast using default failure decider *** FAILED ***
[info]   Vector(Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1)))) did not contain the same elements as List(Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Success(HttpResponse(500 Internal Server Error,List(Server: testServer, Date: Sun, 01 Jan 2017 00:00:00 GMT),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!), Failure(org.squbs.streams.circuitbreaker.CircuitBreakerOpenException: Circuit Breaker is open; calls are failing fast!)) (ClientFlowCircuitBreakerSpec.scala:174)
sebady commented 5 years ago

Another failure here:
08:24:10 [info] - should handle request with empty web-context 08:24:11 08:24:11.005 [pool-1-thread-1] INFO org.squbs.unicomplex.UnicomplexBoot$ - Started cube org.squbs.unicomplex.test.MultiListener 0.0.1 08:24:11 08:24:11.031 [pool-1-thread-1] INFO org.squbs.unicomplex.UnicomplexBoot$ - Web Service started in 22 milliseconds 08:24:11 08:24:11.032 [pool-1-thread-1] INFO org.squbs.unicomplex.UnicomplexBoot - [MultiListen] activated 08:24:11 [info] MultiListenerSpec: 08:24:11 [info] The unicomplex 08:24:11 [info] - should not have 'third-listener' in the portBindings FAILED 08:24:11 [info] Map("third-listener" -> 25, "default-listener" -> 39909, "second-listener" -> 41176) had size 3 instead of expected size 2 (MultiListenerSpec.scala:97) 08:24:11 [info] - should run up two listeners on different ports 08:24:11 [info] - should only have started the application once

sebady commented 5 years ago

Another travis failure https://api.travis-ci.org/v3/job/485606748/log.txt

[info] - should recover from downstream failure 08:00:23.408 [pool-1-thread-1-ScalaTest-running-BroadcastBooleanBufferNoAutoCommitSpec] DEBUG net.openhft.chronicle.bytes.MappedFile - Allocation of 0 chunk in /tmp/persistent_queue2046121375657204605/directory-listing.cq4t took 0.1 ms. 08:00:23.410 [pool-1-thread-1-ScalaTest-running-BroadcastBooleanBufferNoAutoCommitSpec] DEBUG net.openhft.chronicle.bytes.MappedFile - Allocation of 0 chunk in /tmp/persistent_queue2046121375657204605/directory-listing.cq4t took 0.019 ms. 08:00:23.413 [BroadcastBooleanBufferAtLeastOnceSpec-akka.actor.default-dispatcher-4] DEBUG net.openhft.chronicle.bytes.MappedFile - Allocation of 0 chunk in /tmp/persistent_queue2046121375657204605/20190129-080023.cq4 took 0.088 ms. 08:00:23.413 [BroadcastBooleanBufferAtLeastOnceSpec-akka.actor.default-dispatcher-4] INFO org.squbs.pattern.stream.PersistentQueue - File acquired 1548748823 - /tmp/persistent_queue2046121375657204605/20190129-080023.cq4 08:00:23.413 [BroadcastBooleanBufferAtLeastOnceSpec-akka.actor.default-dispatcher-4] INFO org.squbs.pattern.stream.PersistentQueue - File acquired 1548748823 - /tmp/persistent_queue2046121375657204605/20190129-080023.cq4 08:00:23.414 [BroadcastBooleanBufferAtLeastOnceSpec-akka.actor.default-dispatcher-4] DEBUG net.openhft.chronicle.bytes.MappedFile - Allocation of 0 chunk in /tmp/persistent_queue2046121375657204605/tailer.idx took 0.048 ms. 08:00:23.756 [BroadcastBooleanBufferAtLeastOnceSpec-akka.actor.default-dispatcher-9] ERROR org.squbs.pattern.stream.BroadcastBufferBase$$anon$1$$anon$3 - Received upstream failure signal: java.lang.NumberFormatException: This is a fake exception [info] - should recover from upstream failure FAILED  [info]  java.lang.NullPointerException: [info]  at net.openhft.chronicle.bytes.NativeBytesStore.writeLong(NativeBytesStore.java:409) [info]  at org.squbs.pattern.stream.PersistentQueue.commit(PersistentQueue.scala:140) [info]  at org.squbs.pattern.stream.BroadcastBufferAtLeastOnce.$anonfun$commit$1(BroadcastBuffer.scala:106) [info]  at akka.stream.impl.fusing.Map$$anon$1.onPush(Ops.scala:52) [info]  at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:519) [info]  at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:411) [info]  at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:588) [info]  at akka.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute(ActorGraphInterpreter.scala:45) [info]  at akka.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute$(ActorGraphInterpreter.scala:41) [info]  at akka.stream.impl.fusing.ActorGraphInterpreter$BatchingActorInputBoundary$OnNext.execute(ActorGraphInterpreter.scala:78) [info]  ...

akara commented 5 years ago

@sebady, yes I have seen this before, especially in the Travis environment. This seems to come out of a failure to allocate memory based on looking through the Chronicle code. Still figuring out how to prevent it.