search

mfp / ocaml-sqlexpr

Minimalistic syntax extension for type-safe, convenient execution of SQL statements.
Other
86 stars 17 forks source link

Excessive GC load related to use of Sqlexpr_sqlite_lwt #13

Closed mfp closed 8 years ago

mfp commented 8 years ago

The Sqlexpr_sqlite_lwt worker loop uses the Event module to receive tasks across threads boundaries. As it turns out, Event.sync creates a new Condition.t.

This means in particular that a new condition variable (two in fact, also one for the sending side) is being created per returned row(!).

caml_condition_new allocates the custom blocks with used=1, max = Max_condition_number. The latter was raised to 5000 in 2010, after being set to 1000 back in 1996.

The end effect is that a full major GC run is performed when at most 5000 rows have been read from the SQLite3 DB.

This raises two questions:

  1. are condition variables expensive enough to care about their timely collection instead of using used=0, max = 1
  2. are such static limits reasonable and future-proof, and should they be collected in some visible place so that they can be raised in time? (I ran into a similar limit in the past with ocaml-pcre, which allowed no more than 500 compiled regexps to be unreclaimed)

Now, back to Sqlexpr: time to ditch the Event module and replace it with a single condition variable + manual handling.

Note to self: send issue to ocaml project regarding the performance impact of Event.sync.

Edit: also file Lwt issue, for it also uses Event.sync in Lwt_preemptive (so 1 full major GC run every 2500 Lwt_preemptive.detach at most)