search

apache / incubator-livy

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
https://livy.apache.org/
Apache License 2.0
890 stars 602 forks source link

[LIVY-1003][RSC] Interactive session - Setting large value of rsc.ser… #449

Closed wangdengshan closed 2 months ago

wangdengshan commented 5 months ago

…ver.connect.timeout blocks other tasks

What changes were proposed in this pull request?

The main adjustment here is the thread pool used when creating and closing sessions asynchronously. Scala's default thread pool size is limited, which will cause the waiting thread to be blocked.

https://issues.apache.org/jira/browse/LIVY-1003

How was this patch tested?

How to reproduce:

  1. Set livy.rsc.server.connect.timeout to something high like 24h.
  2. Create enough interactive livy sessions in YARN so that they are queued in ACCEPTED state. The number of sessions that are stuck in ACCEPTED state should be equal to global execution context thread pool size (Runtime.availableProcessors)
  3. Try to delete a session using DELETE /sessions/ {sessionId} and it should not be hang until one of the sessions is no longer stuck in ACCEPTED state.
gyogal commented 3 months ago

Thank you for your contribution @wangdengshan (and thanks also for reviewing @askhatri @nileshrathi345)!