effects compatible with parallel_for loop wrapping?

[nilsbecker]

I tried to implement coarse-grained parallelism in the following way (not actual code)

let pool = Domainslib.Task.setup_pool ~num_domains () in
let body _i =
    let rng_state = RNG.split rng_state in
    let sequence_of_populations =
      Smc_inference.run ~nthreads:1 ~npart resampling initial_state model rng_state in
    let l = Seq.to_rev_list sequence_of_populations in      (* force *)
    [extract_results (List.hd l)] in
let task () =
    Domainslib.Task.parallel_for_reduce ~start:1 ~finish:nn ~body pool (@) [[]] in
let results = Domainslib.Task.run pool task in
results

When num_domains > 0 i get an uncaught exception saying that a Stdlib.Queue is empty. With num_domains=0 the program runs fine in a single domain. Could there be an incompabitility of the way effects are used in dagger with the use of effects in Domainslib?

[nilsbecker]

wild guess: is this thread relevant?

[nilsbecker]

hmm, now i have a backtrace. the fact that the Logs library appears in there suggests that the problem may be coming from logging? could be a red herring.

Fatal error: exception Stdlib.Queue.Empty
Raised at Stdlib__Queue.take in file "queue.ml", line 73, characters 11-22
Called from Stdlib__Format.advance_left in file "format.ml", line 436, characters 6-31
Called from Stdlib__Format.pp_flush_queue in file "format.ml" (inlined), line 613, characters 2-20
Called from Stdlib__Format.pp_print_newline in file "format.ml" (inlined), line 671, characters 2-45
Called from Stdlib__Format.output_formatting_lit in file "format.ml", line 1335, characters 33-56
Called from Stdlib__Format.kfprintf.(fun) in file "format.ml", line 1416, characters 16-34
Called from Logs.report in file "src/logs.ml" (inlined), line 197, characters 2-42
Called from Logs.kmsg in file "src/logs.ml" (inlined), line 254, characters 4-33
Called from Logs.msg in file "src/logs.ml" (inlined), line 257, characters 26-52
Called from Logs.src_log.Log.msg in file "src/logs.ml" (inlined), line 300, characters 25-44
Called from Logs.src_log.Log.info in file "src/logs.ml" (inlined), line 305, characters 20-33
Called from Evidence.Estimation.evidence_resampling in file "lib/evidence.ml", line 300, characters 4-641
Called from Dagger__Smc_inference.Make.Population_state.resample in file "src/smc_inference.ml", line 521, characters 10-77
Called from Dagger__Smc_inference.Make.Runner.perform_resampling in file "src/smc_inference.ml" (inlined), line 680, characters 12-62
Called from Dagger__Smc_inference.Make.Runner.resume_importance_sampling in file "src/smc_inference.ml", line 746, characters 6-49
Called from Dagger__Smc_inference.Make.Runner.run_custom.iter in file "src/smc_inference.ml", line 817, characters 15-70
Re-raised at Domainslib__Task.run.loop in file "lib/task.ml", line 128, characters 23-57
Called from Stdlib__Fun.protect in file "fun.ml" (inlined), line 33, characters 8-15
Called from Domain_local_await.using in file "src/Domain_local_await.ml", line 104, characters 6-94
Re-raised at Stdlib__Fun.protect in file "fun.ml" (inlined), line 38, characters 6-52
Called from Domain_local_await.using in file "src/Domain_local_await.ml", line 104, characters 6-94
Called from Dune__exe__Gaussian_mean.kernel_stepping_variance_test in file "bin/gaussian_mean.ml", line 305, characters 16-45
Called from Dune__exe__Gaussian_mean.main.(fun) in file "bin/gaussian_mean.ml" (inlined), line 332, characters 17-98
Called from Dune__exe__Gaussian_mean.timeit in file "bin/gaussian_mean.ml" (inlined), line 317, characters 11-15
Called from Dune__exe__Gaussian_mean.main in file "bin/gaussian_mean.ml", line 331, characters 11-142

[nilsbecker]

I reran with a logging level that disables Log.info. This does not raise the exception! So then I called Logs_threaded.enable () in the main program before spawning. This allowed also enabling Log.info without raising an exception. Problem solved! Sorry for the noise.