search

rdicosmo / parmap

Parmap is a minimalistic library allowing to exploit multicore architecture for OCaml programs with minimal modifications.
http://rdicosmo.github.io/parmap/
Other
94 stars 20 forks source link

Data marshaling problem #22

Closed johanmazel closed 10 years ago

johanmazel commented 10 years ago

I am currently getting some errors "Fatal error: exception Failure("output_value: object too big")". I am not absolutely sure about this, but I strongly suspect the Marshal.to_string call in the function marshal at line 114 of the parmap.ml file to be the cause of this.

Do you have any idea how to fix this ? Can bin_prot/biniou help in this case ? If yes, it would be nice to be able to use these alternative marshaling methods.

UnixJunkie commented 10 years ago

I personally have no idea, not even a workaround.

rdicosmo commented 10 years ago

Are you really running parmap on an array of more than 4 gigabytes? That's huge! :-) But it's true that with modern machines this can happen. There is no way I can see of bypassing this limitation currently (biniou will not help here), so the (unsatisfactory, but easy) workaround is to just split your array/list in several chunks, run parmap on each of them, and then join the results.

UnixJunkie commented 10 years ago

The solution proposed by Roberto is good but reduces the parallel section of the program. Another way is to design your program so that you don't serialize back to the parent process (if that's doable): use a pariter instead of a fold or map.