search

scala / pickling

Fast, customizable, boilerplate-free pickling support for Scala
lampwww.epfl.ch/~hmiller/pickling
BSD 3-Clause "New" or "Revised" License
831 stars 79 forks source link

Pickling in parallel is slow #323

Open dirkgr opened 9 years ago

dirkgr commented 9 years ago

When pickling a lot of small objects in parallel, I see a lot of contention on internal.GRL, especially in functions.scala:29.

heathermiller commented 9 years ago

GRL is a global reflection lock because Scala reelection is unfortunately not threadsafe. We had to choose determinism and correctness over performance here.

What version of Scala and Scala Pickling are you using?

phaller commented 9 years ago

To improve this situation on Scala 2.11.x and 2.12.x, the plan is to create a 2.11.x branch where this lock can be removed, since reflection is thread-safe on 2.11.x. Moreover, we could investigate ways to remove this lock also in "static-only" mode.

heathermiller commented 9 years ago

Clarification: reflection is thread-safe on 2.11.x in the form of another global reflection lock. So Philipp's alternative 2 above would likely be the best bet, if you're sure that what you're trying to pickle can be statically serialized.

jsuereth commented 9 years ago

Additionally, we're working on removing this lock in any place where reflection isn't used. That should speed things up now.

terkhorn commented 9 years ago

Hey @jsuereth, if you have any WIP for this, do you mind posting it? Just a curious party here :-D

jsuereth commented 9 years ago

@terkhorn Check out the 0.11.x branch, specifically: https://github.com/scala/pickling/blob/0.11.x/core/src/main/scala/scala/pickling/spi/PicklingRuntime.scala

Sorry it took so long to get to this. In any case, The next TODOs in 0.11.x are:

  1. to make GRL be a pure interface, rather than a specific thing.
  2. Instantiate the GRL to a null lock so that there is no locking/contention in 2.11.x

Additionally, there should be less need for the GRL even in the 2.10.x case on the 0.11.x branch.