apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.29k stars 1.16k forks source link

[BUG] when submitting tasks in batches, em resources may exceed the threshold #4917

Closed sjgllgh closed 11 months ago

sjgllgh commented 11 months ago

What is the purpose of the change

Fix bug: when submitting tasks in batches, em resources may exceed the threshold.

Related issues/PRs

Related issues: #4916

Brief change log

Checklist

peacewong commented 11 months ago

The logical judgment here is considered as follows:

  1. The ECM tag is not locked first for resource judgment. If the resource is judged, it will continue. Therefore, there is a certain probability of negative resources, but there will not be too many situations.
  2. The reason for this consideration is to reduce lock waiting, because here is a concurrent lock grab, so the lock granularity should be as small as possible to reduce concurrency pressure.
  3. If the usercreator lock is placed at a higher level, it will cause more lock waits.

这里的逻辑判断是这样考虑的:

  1. ECM标签先不锁进行资源判断如果资源判断过就继续,所以存在一定概率出现负资源,但是不会出现太多情况
  2. 之所以这样考虑是为了减少锁的等待,因为这里是并发抢锁的,所以锁的粒度尽量小减少并发压力
  3. 如果把usercreator的锁放到更上一层,会导致更多的锁等待
sjgllgh commented 11 months ago

Thank you for your answer. Different scenarios may have different considerations.

The logical judgment here is considered as follows:

  1. The ECM tag is not locked first for resource judgment. If the resource is judged, it will continue. Therefore, there is a certain probability of negative resources, but there will not be too many situations.
  2. The reason for this consideration is to reduce lock waiting, because here is a concurrent lock grab, so the lock granularity should be as small as possible to reduce concurrency pressure.
  3. If the usercreator lock is placed at a higher level, it will cause more lock waits.

这里的逻辑判断是这样考虑的:

  1. ECM标签先不锁进行资源判断如果资源判断过就继续,所以存在一定概率出现负资源,但是不会出现太多情况
  2. 之所以这样考虑是为了减少锁的等待,因为这里是并发抢锁的,所以锁的粒度尽量小减少并发压力
  3. 如果把usercreator的锁放到更上一层,会导致更多的锁等待