Open ben1222 opened 2 years ago
record parser was meant to be used exclusively from event loop, that's why we do have this corner case.
we need try to fix this and have a concurrent version or find another strategy that could be in vertx-rx
it seems that using an atomic long for the demand and a CAS for the parsing boolean could work. we would likely also need to replace the current buffer by a list of buffer instead so a new buffer can be concurrently added while another thread is parsing.
@tsegismont made this code not reentrant and instead have thread cooperate and I think it helps to have a proper concurrent version
do you have an opinion you can share @jponge ?
Looking at the code it sounds like we should explore a drain-loop design as one can find in RxJava / Mutiny, etc.
handle
AtomcInteger
to make sure handleParsing
is called for just 1 thread when fetch
is being calledAtomicLong
in fetch
(and make sure overflowing is not an option)This way concurrent calls to:
handle
: safe, but ordering is up in the air,fetch
: safe, ensures serial emissionsWe'd also need to come up with an even smaller reproducer that's only based on Vert.x core.
Version
vertx 4.1.7
Context
I encountered some issues when using
RecordParser
withconcatMapCompletable
andobserveOn
like the following code to process records one by one on worker thread:The issues including:
RecordParser
will suddenly stop emitting record, there's no error and not reached end of file and it is not disposed... it just stuck there.MissingBackpressureException
is thrownAfter investigated and tried create a unit test for this issue, it looks like a race condition on
RecordParserImpl
: Usually the record is emitted fromRecordParserImpl
on event loop thread. (RecordParserImpl.handle
) However, when backpressure exists (concatMapCompletable
here), it can be the thread running on downstream to request item from upstream. (RecordParserImpl.fetch
) In my case, the inner stream ofconcatMapCompletable
is switched to a worker thread usingobserveOn
, so it will be the worker thread requesting item fromRecordParserImpl
during backpressure.When the
RecordParserImpl.handle
running on event loop thread and theRecordParserImpl.fetch
running on worker thread are called at same time, race condition happens becauseRecordParserImpl
is not written in thread-safe way. The race condition includes but not limited to:RecordParserImpl.handleParsing()
, both thread may passed theparsing
check and do the parsing concurrentlydemand
could be modified concurrently and result in unexpected valueRecordParserImpl.handle
has filled alldemand
and paused upstream, but beforeparsing
is set tofalse
, theRecordParserImpl.fetch
could adddemand
and exit quickly due toparsing
istrue
, and then it will stuck - upstream is paused soRecordParserImpl.handle
will not be called again, downstream has requested item and is waiting for next item.Steps to reproduce
Here's a unit test that could reproduce the issue:
In the printed log, we can see the
xx: Read record yy
log can sometime be printed on event loop thread (RecordParserImpl.handle
) and sometime be printed on shared worker thread (due to backpressure,RecordParserImpl.fetch
)The backtrace for issue 1 looks like:
The backtrace for issue 2 looks like:
Not sure if it is a problem in
RecordParser
orconcatMapCompletable
(is it expected for upstream to be requested on worker thread in this case?) or maybe it is not a desired to use them in this way?