Lockup with high varianceThreshold (plus valgrind issue)

[paulmelis]

I'm seeing an issue with my network-driven render server where a very high varianceThreshold (in the range of 10-15) causes spurious lockups (render_future.isReady(OSP_TASK_FINISHED) never returns true). Those threshold values aren't very realistic, but might still be set by a user, so I'd like to figure out if the lockup is caused by my code somehow or stems from OSPRay itself. I've not been able to reproduce the issue with, say, a hacked version of ospTutorial, so I can only describe my own code at a high level.

There's a main loop that handles incoming network messages and manages the rendering of each sample if a render is active:

bool
handle_connection(TCPSocket *sock)
{
    while (true)
    {
        usleep(1000);

        // Handle incoming network messages
        ...

        if (cancel_rendering)
        {
            render_future.cancel();
            render_future.wait(OSP_TASK_FINISHED);
            continue;
        }

        // Check if frame render has finished
        if (!render_future.isReady(OSP_TASK_FINISHED))
            continue;        

        variance = framebuffer.variance();
        float *color = (float*)framebuffer.map(OSP_FB_COLOR);
        ...
        framebuffer.unmap(color);

        if (current_sample < num_samples)
        {
            // Kick off render of next sample
            render_future = framebuffer.renderFrame(ospray_renderer, ospray_camera, ospray_world);
        }
    }
}

There's no multi-threading going on in my own code, and all OSPRay calls are driven by incoming network messages. As mentioned render_future.isReady(OSP_TASK_FINISHED) in certain cases never returns true, although I don't see any CPU activity (indicating rendering is still going on) at that point. These lockups happen only when the FB variance has dropped below the threshold, e.g. for a threshold of 15:

32 samples per pixel (final)
Sample |  Frame   | Variance |  Memory   | Peak Mem  | FB Compr | FB pixels | Send FB  |  Total
     1 |   27.4ms |      inf |   91.7 MB |   91.7 MB |   18.9ms |    0.1 MB |    0.1ms |   46.5ms
     2 |   31.9ms |   13.876 |   99.0 MB |   99.0 MB |   14.4ms |    0.0 MB |    0.1ms |   46.4ms
     3 |    1.1ms |   13.876 |  127.4 MB |  127.4 MB |   16.5ms |    0.0 MB |    0.1ms |   17.7ms
     4 |    2.1ms |   13.876 |  127.4 MB |  127.4 MB |   15.7ms |    0.0 MB |    0.1ms |   17.9ms
     5 |    2.3ms |   13.876 |  127.4 MB |  127.4 MB |   16.3ms |    0.0 MB |    0.1ms |   18.7ms
     6 |    2.1ms |   13.876 |  127.4 MB |  127.4 MB |   15.6ms |    0.0 MB |    0.1ms |   17.7ms
     7 |    2.3ms |   13.876 |  127.4 MB |  127.4 MB |   16.7ms |    0.0 MB |    0.1ms |   19.1ms
     8 |    3.0ms |   13.876 |  127.4 MB |  127.4 MB |   18.1ms |    0.0 MB |    0.1ms |   21.2ms
     9 <stuck>

This seems to suggest that the future returned by renderFrame() at that point should signal it's done, but doesn't do so. I don't see how my own code could cause this behaviour, but like I said I'd like to know for sure. However, using Valgrind so far doesn't seem to work due to this issue:

==115411== 
VEX temporary storage exhausted.
Pool = TEMP,  start 0x597c8528 curr 0x59c85c20 end 0x59c8d067 (size 5000000)

vex: the `impossible' happened:
   VEX temporary storage exhausted.
Increase N_{TEMPORARY,PERMANENT}_BYTES and recompile.
vex storage: T total 2163584912 bytes allocated
vex storage: P total 512 bytes allocated

valgrind: the 'impossible' happened:
   LibVEX called failure_exit().

host stacktrace:
==115411==    at 0x58042F5A: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x58043087: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x580432EF: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x58043310: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x580584FA: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x5812E450: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x5812E4E1: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x581D343C: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x581BD9E0: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x5812C520: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x5805ADCB: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x58098BFB: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x5809B4E8: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x580E4050: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x580E4330: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0x580AC1DD: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
==115411==    by 0xDEADBEEFDEADBEEE: ???
==115411==    by 0xDEADBEEFDEADBEEE: ???
==115411==    by 0xDEADBEEFDEADBEEE: ???

sched status:
  running_tid=2

Thread 1: status = VgTs_WaitSys syscall 230 (lwpid 115411)
==115411==    at 0x6774FD5: clock_nanosleep@@GLIBC_2.17 (clock_nanosleep.c:48)
==115411==    by 0x6779B96: nanosleep (nanosleep.c:25)
==115411==    by 0x67A5D58: usleep (usleep.c:31)
==115411==    by 0x1462E6: handle_connection(TCPSocket*) (main.cpp:3401)
==115411==    by 0x148111: main (main.cpp:3860)
client stack range: [0x1FFEFFA000 0x1FFF000FFF] client SP: 0x1FFEFFF500
valgrind stack range: [0x10030B6000 0x10031B5FFF] top usage: 17872 of 1048576

Is this a know issue? This is with OSPRay 2.10.0 on an Arch Linux system.

[paulmelis]

Okay, so I did manage to observe the same behaviour with a sample based on ospTutorial. See ospTutorial.cpp.gz. This only gets stuck in the while (!f.isReady(OSP_TASK_FINISHED)) loop in roughly 1 out of 10 runs. It seems f.progess() returns 1.0 in those cases, but isReady() returns false:

melis@juggle 15:18:~/c/ospray-git/apps/ospTutorial$ ./ospTutorial
[openvkl] application requested ISPC device width 8via device name cpu_8
[openvkl] trying to look up device type 'cpu_8' for the first time
[openvkl] CPU device instantiated with width: 8, ISA: AVX2
Rendering sample 1 ... 0.0 0.0 0.0 0.1 0.1 0.1 0.2 0.2 0.2 0.2 0.3 0.3 0.3 0.3 0.3 0.4 0.4 0.4 0.5 0.5 0.5 0.5 0.5 0.6 0.6 0.7 0.7 0.7 0.8 0.8 0.8 0.9 0.9 0.9 0.9 0.9 1.0 1.0 done, variance = inf
Rendering sample 2 ... 0.0 0.0 0.1 0.1 0.1 0.2 0.2 0.3 0.3 0.3 0.4 0.4 0.4 0.5 0.5 0.5 0.5 0.6 0.6 0.6 0.6 0.6 0.6 0.7 0.7 0.7 0.8 0.8 0.8 0.8 0.9 0.9 0.9 0.9 1.0 done, variance = 17.9
Rendering sample 3 ... 0.0 0.0 0.1 0.2 0.2 0.3 0.3 0.3 0.4 0.4 0.4 0.4 0.4 0.5 0.5 0.5 0.5 0.5 0.5 0.6 0.6 0.6 0.6 0.6 0.7 0.7 0.7 0.8 0.8 0.9 0.9 0.9 0.9 1.0 done, variance = 12.8
Rendering sample 4 ... 0.0 0.0 0.1 0.1 0.2 0.2 0.2 0.3 0.3 0.4 0.4 0.4 0.4 0.4 0.5 0.5 0.5 0.5 0.5 0.6 0.6 0.6 0.7 0.7 0.8 0.8 0.8 0.9 0.9 0.9 0.9 0.9 0.9 1.0 done, variance = 15.0
Rendering sample 5 ... 0.0 0.0 0.1 0.1 0.2 0.2 0.3 0.3 0.4 0.4 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.6 0.6 0.6 0.6 0.6 0.7 0.7 0.7 0.8 0.8 0.8 0.8 0.8 0.8 0.9 0.9 1.0 done, variance = 12.3
Rendering sample 6 ... 0.0 0.1 0.1 0.1 0.2 0.2 0.2 0.3 0.3 0.3 0.3 0.3 0.4 0.4 0.4 0.5 0.5 0.5 0.6 0.6 0.6 0.7 0.7 0.7 0.8 0.8 0.9 0.9 0.9 0.9 0.9 0.9 1.0 1.0 done, variance = 13.2
Rendering sample 7 ... 0.0 0.0 0.0 0.1 0.1 0.1 0.1 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.3 0.3 0.3 0.4 0.4 0.4 0.4 done, variance = 12.2
Rendering sample 8 ... 0.0 0.0 0.0 0.1 0.1 0.1 0.1 0.1 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.3 0.3 0.3 0.3 0.3 0.4 0.4 0.4 done, variance = 11.8
Rendering sample 9 ... 0.0 done, variance = 11.8
Rendering sample 10 ... 0.0 done, variance = 11.8
Rendering sample 11 ... 0.0 done, variance = 11.8
Rendering sample 12 ... 0.0 done, variance = 11.8
Rendering sample 13 ... 0.0 done, variance = 11.8
Rendering sample 14 ... 0.0 done, variance = 11.8
Rendering sample 15 ... 0.0 done, variance = 11.8
Rendering sample 16 ... 0.0 done, variance = 11.8
Rendering sample 17 ... 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0...

Also interesting, when run with --osp:debug it gets stuck on the first sample with the progress staying at 0.0? I also see no CPU activity in that case (other than a bit for the check-progress loop):

melis@juggle 15:23:~/c/ospray-git/apps/ospTutorial$ ./ospTutorial --osp:debug

Embree Ray Tracing Kernels 3.13.3 (0835461a5dd1659696524303dc9cb2f2300ae156)
  Compiler  : Intel Compiler 19.10.0
  Build     : Release 
  Platform  : Linux (64bit)
  CPU       : Core Haswell (GenuineIntel)
   Threads  : 4
   ISA      : XMM YMM SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 POPCNT AVX F16C RDRAND AVX2 FMA3 LZCNT BMI1 BMI2 
   Targets  : SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 AVX AVXI AVX2 
   MXCSR    : FTZ=1, DAZ=1
  Config
    Threads : default
    ISA     : XMM YMM SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 POPCNT AVX F16C RDRAND AVX2 FMA3 LZCNT BMI1 BMI2 
    Targets : SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 AVX AVXI AVX2  (supported)
              SSE2 SSE4.2 AVX AVX2 AVX512  (compile time enabled)
    Features: intersection_filter 
    Tasking : TBB2021.5 TBB_header_interface_12050 TBB_lib_interface_12050 

general:
  build threads      = 0
  build user threads = 0
  start_threads      = 0
  affinity           = 0
  frequency_level    = simd256
  hugepages          = enabled
  verbosity          = 2
  cache_size         = 134.218 MB
  max_spatial_split_replications = 1.2
triangles:
  accel              = default
  builder            = default
  traverser          = default
motion blur triangles:
  accel              = default
  builder            = default
  traverser          = default
quads:
  accel              = default
  builder            = default
  traverser          = default
motion blur quads:
  accel              = default
  builder            = default
  traverser          = default
line segments:
  accel              = default
  builder            = default
  traverser          = default
motion blur line segments:
  accel              = default
  builder            = default
  traverser          = default
hair:
  accel              = default
  builder            = default
  traverser          = default
motion blur hair:
  accel              = default
  builder            = default
  traverser          = default
subdivision surfaces:
  accel              = default
grids:
  accel              = default
  builder            = default
motion blur grids:
  accel              = default
  builder            = default
object_accel:
  min_leaf_size      = 1
  max_leaf_size      = 1
object_accel_mb:
  min_leaf_size      = 1
  max_leaf_size      = 1
[openvkl] CPU device instantiated with width: 8, ISA: AVX2

Using ISPC device with AVX2 instruction set...
Using ISPC device with AVX2 instruction set...
ospray::Mesh created: #primitives=2, #vertices=4
=======================================================
Finalizing instance, which has 1 geometries, 0 volumes, 0 clipping geometries and 0 lights
               segments:          0
-----------------------------------
      flat_linear_curve:          0
     round_linear_curve:          0
  oriented_linear_curve:          0
      flat_bezier_curve:          0
     round_bezier_curve:          0
  oriented_bezier_curve:          0
     flat_bspline_curve:          0
    round_bspline_curve:          0
 oriented_bspline_curve:          0
     flat_hermite_curve:          0
    round_hermite_curve:          0
 oriented_hermite_curve:          0
 flat_catmull_rom_curve:          0
round_catmull_rom_curve:          0
oriented_catmull_rom_curve:          0
              triangles:          2
                  quads:          0
                   grid:          0
                subdivs:          0
                 sphere:          0
                   disc:          0
          oriented_disc:          0
               usergeom:          0
         instance_cheap:          0
     instance_expensive:          0
building BVH8<triangle4> using avx::BVH8BuilderSAH ...
finished BVH8<triangle4> : 0.0269413ms, 0.0742355 Mprim/s, 0.00653272 GB/s
  primitives = 2, vertices = 0, depth = 0
  total            : sah =   1.000 (100.00%), #bytes =    0.00 MB (100.00%), #nodes =       1 ( 50.00% filled), #bytes/prim =  88.00
  leaves           : sah =   1.000 (100.00%), #bytes =    0.00 MB (100.00%), #nodes =       1 ( 50.00% filled), #bytes/prim =  88.00
    histogram      : 100.00%   0.00%   0.00%   0.00%   0.00%   0.00%   0.00%   0.00% 
  alloc : used =   0.000 MB,                                                             #bytes/prim =  88.00
  alloc : used =   0.000 MB, free =   0.001 MB, wasted =   0.000 MB, total =   0.001 MB, #bytes/prim = 512.00
  total : used =   0.001 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.001 MB, #bytes/prim = 576.00
  4K    : used =   0.000 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.000 MB, #bytes/prim =   0.00
  2M    : used =   0.000 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.000 MB, #bytes/prim =   0.00
  malloc: used =   0.001 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.001 MB, #bytes/prim = 576.00
  shared: used =   0.000 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.000 MB, #bytes/prim =   0.00
created scene intersector
  accels[0]
    intersector1  = avx2::BVH8Triangle4Intersector1Moeller
    intersector4  = avx2::BVH8Triangle4Intersector4HybridMoellerNoFilter
    intersector8  = avx2::BVH8Triangle4Intersector8HybridMoellerNoFilter
    intersectorN = avx2::BVH8Triangle4IntersectorStreamMoellerNoFilter
selected scene intersector
  intersector1  = avx2::BVH8Triangle4Intersector1Moeller
  intersector4  = avx2::BVH8Triangle4Intersector4HybridMoellerNoFilter
  intersector8  = avx2::BVH8Triangle4Intersector8HybridMoellerNoFilter
  intersectorN = avx2::BVH8Triangle4IntersectorStreamMoellerNoFilter
=======================================================
Committing world, which has 1 instances and 2 lights
               segments:          0
-----------------------------------
      flat_linear_curve:          0
     round_linear_curve:          0
  oriented_linear_curve:          0
      flat_bezier_curve:          0
     round_bezier_curve:          0
  oriented_bezier_curve:          0
     flat_bspline_curve:          0
    round_bspline_curve:          0
 oriented_bspline_curve:          0
     flat_hermite_curve:          0
    round_hermite_curve:          0
 oriented_hermite_curve:          0
 flat_catmull_rom_curve:          0
round_catmull_rom_curve:          0
oriented_catmull_rom_curve:          0
              triangles:          0
                  quads:          0
                   grid:          0
                subdivs:          0
                 sphere:          0
                   disc:          0
          oriented_disc:          0
               usergeom:          0
         instance_cheap:          1
     instance_expensive:          0
building BVH8<instance> using avx::BVH8BuilderSAH ...
finished BVH8<instance> : 0.013113ms, 0.0762601 Mprim/s, 0.00122016 GB/s
  primitives = 1, vertices = 0, depth = 0
  total            : sah =   1.000 (100.00%), #bytes =    0.00 MB (100.00%), #nodes =       1 (100.00% filled), #bytes/prim =  16.00
  leaves           : sah =   1.000 (100.00%), #bytes =    0.00 MB (100.00%), #nodes =       1 (100.00% filled), #bytes/prim =  16.00
    histogram      : 100.00%   0.00%   0.00%   0.00%   0.00%   0.00%   0.00%   0.00% 
  alloc : used =   0.000 MB,                                                             #bytes/prim =  16.00
  alloc : used =   0.000 MB, free =   0.001 MB, wasted =   0.000 MB, total =   0.001 MB, #bytes/prim = 1024.00
  total : used =   0.001 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.001 MB, #bytes/prim = 1152.00
  4K    : used =   0.000 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.000 MB, #bytes/prim =   0.00
  2M    : used =   0.000 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.000 MB, #bytes/prim =   0.00
  malloc: used =   0.001 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.001 MB, #bytes/prim = 1152.00
  shared: used =   0.000 MB, free =   0.000 MB, wasted =   0.000 MB, total =   0.000 MB, #bytes/prim =   0.00
created scene intersector
  accels[0]
    intersector1  = avx2::BVH8InstanceIntersector1
    intersector4  = avx2::BVH8InstanceIntersector4Chunk
    intersector8  = avx2::BVH8InstanceIntersector8Chunk
    intersectorN = avx2::BVH8InstanceIntersectorStream
selected scene intersector
  intersector1  = avx2::BVH8InstanceIntersector1
  intersector4  = avx2::BVH8InstanceIntersector4Chunk
  intersector8  = avx2::BVH8InstanceIntersector8Chunk
  intersectorN = avx2::BVH8InstanceIntersectorStream
Rendering sample 1 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ^C

Note: even though this is the ospTutorial.cpp example from current git master, it's compiled and running against the 2.10 Linux binaries from ospray.org.

[BruceCherniak]

Hi @paulmelis, at one point in time we had a very sporadic problem in OSPRay Studio that feels similar. As a simple test, can you change the OSP_TASK_FINISHED to OSP_FRAME_FINISHED? See if that affects the lockup.

[paulmelis]

Well, just tried about 50 runs with OSP_FRAME_FINISHED, did not see any lockups, so perhaps that indeed is a fix.

[BruceCherniak]

Interesting. I have been using this flag in OSPRay Studio for several months now, after experiencing a similar sporadic lockup, but only on several older architectures. I have not seen any artifacts from waiting on OSP_FRAME_FINISHED, however I am not running other tasks at the same time. YMMV.

[chegarty3]

I'm encountering the same issue.

I'm running with a variance of 1.0. I am running in a loop with a 20ms sleep in between iterations, and when a frame is done (by checking Future::IsReady with OSP_TASK_FINISHED), it starts a new frame, very similar to the previously posted example. Sporadically, when the rate of new renders becomes very high (like 20 / s), this issue pops up.

Debugging, I see that RenderTask::finished (modules\cpu\render\RenderTask) returns false. Even if I leave it for several minutes, it remains false though CPU usage is 0 and the task is very clearly finished. Interestingly, RenderTask::wait(OSP_TASK_FINISHED) returns immediately.

Using OSP_FRAME_FINISHED does avoid the issue. Is it safe to start a new RenderFrame call at that time?

[johguenther]

The issue will be fixed in the next release. In the meantime either use OSP_FRAME_FINISHED (it is safe to immediately start a new frame) or apply the following patch to rkcommon/tasking/AsyncTask.h to initialize jobFinished before taskImpl:

diff --git a/rkcommon/tasking/AsyncTask.h b/rkcommon/tasking/AsyncTask.h
--- a/rkcommon/tasking/AsyncTask.h
+++ b/rkcommon/tasking/AsyncTask.h
@@ -48,8 +48,8 @@ namespace rkcommon {
       }

      private:
-      detail::AsyncTaskImpl<std::function<void()>> taskImpl;
       std::atomic<bool> jobFinished{false};
+      detail::AsyncTaskImpl<std::function<void()>> taskImpl;
       T retValue;
     };