Closed vpipkt closed 4 years ago
tests/GeotrellisTests.py::GeotrellisTests::test_write_geotrellis_layer
seems to be causing Circle CI (s22s org) to exceed 10 minute time out. :-(
Still stuck after 15 min. Not sure if the output that we see is a useful clue or not.
Interesting that locally this test runs for me in ~15 seconds.
Also, unable to duplicate this with the circleci
CLI tool on my machine, maybe due to resources?
tests/GeoTiffWriterTests.py::GeoTiffWriter::test_unstructured_write_schemaless PASSED [ 7%]
tests/GeotrellisTests.py::GeotrellisTests::test_write_geotrellis_layer 2020-02-25 21:23:59,011 shutdown-hooks-run-all ERROR No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2
Too long with no output (exceeded 15m0s): context deadline exceeded
ssh
into the test container reveals the following state when the output seems to hang:
/tmp
dirs are like this:$ circleci@6bd0336a4227:~$ ls /tmp/tmp*
/tmp/tmp_av2v8ux:
attributes gt_layer
/tmp/tmphi8kei3f:
attributes gt_layer
Presume this means that the Geotrellis tests have actually run
There are some python processes going:
The main process running the CI job is is idle, the python processes seem idle. Everything is stalled.
Interestingly the web UI shows the following:
tests/GeoTiffWriterTests.py::GeoTiffWriter::test_identity_write PASSED [ 5%]
tests/GeoTiffWriterTests.py::GeoTiffWriter::test_unstructured_write PASSED [ 6%]
tests/GeoTiffWriterTests.py::GeoTiffWriter::test_unstructured_write_schemaless PASSED [ 7%]
And doesn't print the Geotrellis test that passed until it is kind of killing/ flushing everythign. based on the /tmp
output I gather both GT tests passed and somehow something else is hanging ...
Running now $ pushd repo/pyrasterframes/src/main/python && python setup.py test
... See the hanging behavior with the test_write_geotrellis_layer but not sure why. When I comment out the lines about reading in the GT layer again, it runs to completion. Just to declare rf_gt = self.spark.read
runs. Doing the count
hangs ...
More poking around in the CircleCI test container.
Write GT layer, do no action, delete temp dir = 👌
Write GT layer, rf_gt.take
, delete temp dir = ❌
Pardon the dust here, but I ssh'ed into the container running on circleci and then in target dir python setup.py install
the package, the run python -m unittest -v tests/GeotrellisTests.py
I tweaked the test to print more stuff.. Observe:
writing done, calling spark.read
Calling count on the dataframe
[Stage 6:> (0 + 36) / 36]Exception in thread "geotrellis-default-io-30" java.lang.NoClassDefFoundError: fs2/Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169$$anonfun$apply$170
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2003)
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2001)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:139)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:127)
at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:70)
at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:36)
at cats.effect.internals.TrampolineEC$JVMTrampoline.cats$effect$internals$TrampolineEC$JVMTrampoline$$super$startLoop(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply$mcV$sp(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:92)
at cats.effect.internals.Trampoline.execute(Trampoline.scala:43)
at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:44)
at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:133)
at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:120)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$2.apply(Deferred.scala:205)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$2.apply(Deferred.scala:205)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply$mcV$sp(Deferred.scala:241)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply(Deferred.scala:241)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply(Deferred.scala:241)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169$$anonfun$apply$170
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 32 more
Exception in thread "geotrellis-default-io-10" java.lang.NoClassDefFoundError: fs2/Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169$$anonfun$apply$170
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2003)
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2001)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:139)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:127)
at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:70)
at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:36)
at cats.effect.internals.TrampolineEC$JVMTrampoline.cats$effect$internals$TrampolineEC$JVMTrampoline$$super$startLoop(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply$mcV$sp(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:92)
at cats.effect.internals.Trampoline.execute(Trampoline.scala:43)
at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:44)
at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:133)
at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:120)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$2.apply(Deferred.scala:205)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$2.apply(Deferred.scala:205)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply$mcV$sp(Deferred.scala:241)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply(Deferred.scala:241)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply(Deferred.scala:241)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "geotrellis-default-io-12" java.lang.NoClassDefFoundError: fs2/Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169$$anonfun$apply$170
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2003)
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2001)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:139)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:127)
at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:70)
at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:36)
at cats.effect.internals.TrampolineEC$JVMTrampoline.cats$effect$internals$TrampolineEC$JVMTrampoline$$super$startLoop(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply$mcV$sp(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:92)
at cats.effect.internals.Trampoline.execute(Trampoline.scala:43)
at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:44)
at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:133)
at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:120)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$2.apply(Deferred.scala:205)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$2.apply(Deferred.scala:205)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply$mcV$sp(Deferred.scala:241)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply(Deferred.scala:241)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply(Deferred.scala:241)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "geotrellis-default-io-17" java.lang.NoClassDefFoundError: fs2/Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169$$anonfun$apply$170
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2003)
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2001)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:139)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:127)
at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:70)
at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:36)
at cats.effect.internals.TrampolineEC$JVMTrampoline.cats$effect$internals$TrampolineEC$JVMTrampoline$$super$startLoop(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply$mcV$sp(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:92)
at cats.effect.internals.Trampoline.execute(Trampoline.scala:43)
at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:44)
at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:133)
at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:120)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$2.apply(Deferred.scala:205)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$2.apply(Deferred.scala:205)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply$mcV$sp(Deferred.scala:241)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply(Deferred.scala:241)
at cats.effect.concurrent.Deferred$ConcurrentDeferred$$anonfun$1.apply(Deferred.scala:241)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "geotrellis-default-io-1" java.lang.NoClassDefFoundError: fs2/Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169$$anonfun$apply$170
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2003)
at fs2.Stream$$anonfun$parJoin$extension$2$$anonfun$apply$159$$anonfun$apply$160$$anonfun$apply$161$$anonfun$fs2$Stream$$anonfun$$anonfun$$anonfun$$anonfun$$runInner$2$1$$anonfun$apply$165$$anonfun$apply$168$$anonfun$apply$169.apply(Stream.scala:2001)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:139)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:127)
at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:70)
at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:36)
at cats.effect.internals.TrampolineEC$JVMTrampoline.cats$effect$internals$TrampolineEC$JVMTrampoline$$super$startLoop(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply$mcV$sp(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at cats.effect.internals.TrampolineEC$JVMTrampoline$$anonfun$startLoop$1.apply(TrampolineEC.scala:93)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:92)
at cats.effect.internals.Trampoline.
Try upgrading sbt via build.properties
to 1.3.x
?
@philvarner is this the same error you were having?
See the Travis build runs and passes https://travis-ci.org/locationtech/rasterframes/jobs/656294160#L1369-L1377
We see Circle passing.
Signed-off-by: Jason T. Brown jason@astraea.earth