ParsingThread blocked by jgroups

[guillaumepitel]

Hi, so as often, I've had this problem where the crawl slows down until nothing happens anymore. A threadDump shows this : `"ParsingThread-63" #41669 daemon prio=3 os_prio=0 tid=0x00007f71c03bb800 nid=0x7a97 runnable [0x00007f7514b09000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@9.0.1/Native Method)

• parking to wait for <0x00007f7ea346b1b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9.0.1/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@9.0.1/AbstractQueuedSynchronizer.java:2192) at org.jgroups.protocols.FC.handleDownMessage(FC.java:567) at org.jgroups.protocols.FC.down(FC.java:420) at org.jgroups.protocols.FRAG2.down(FRAG2.java:136) at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:202) at org.jgroups.protocols.pbcast.FLUSH.down(FLUSH.java:277) at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1038) at org.jgroups.JChannel.down(JChannel.java:791) at org.jgroups.JChannel.send(JChannel.java:426) at it.unimi.dsi.jai4j.jgroups.JGroupsRemoteJobManager.process(JGroupsRemoteJobManager.java:144) at it.unimi.dsi.jai4j.jgroups.JGroupsJobManager.submit(JGroupsJobManager.java:469) at it.unimi.di.law.bubing.frontier.Frontier.enqueue(Frontier.java:627) at it.unimi.di.law.bubing.frontier.ParsingThread$FrontierEnqueuer.enqueue(ParsingThread.java:209) at it.unimi.di.law.bubing.frontier.ParsingThread.run(ParsingThread.java:425) `

I will try to identify if this is happening as soon as the crawl slows down, since the slowdown started before I noticed jgroups-related issues, but it may be at least interesting to have a non-blocking file-backed queue for the URL submission process.

[vigna]

It happened to us too, but the best we could do was to submit the stack trace to the JGroup author, who didn't do much with it. After the third-from-bottom line, it's no longer us: at some point JGroup gets stuck.

[guillaumepitel]

After a little digging, FC is the flow control mecanism in JGroups that handle backpressure to avoid keeping sending message faster than the receiver can handle them. So I guess it boils down to JGroups configuration. My configuration is probably not optimal. However, I still think it's a bad idea to make the ParsingThread potentially blocking. Especially since just one overwhelmed receiver can easily block all ParsingThreads, which can have cascading effects for other receivers.

[vigna]

Mmmhhhh... I'm thinking. Maybe the receiver is flushing. So it locks for minutes. And then it never recovers. Do you see something like this from the logs?