search

volcengine / SecureUnionID

Secure ECC-based DID intersection in Go, Java and C.
Apache License 2.0
25 stars 12 forks source link

Spark集群大批量加盲task阻塞 #35

Open jiafeiwang opened 4 months ago

jiafeiwang commented 4 months ago

编译打包后，提交到Spark集群注册udf进行大批量Blind，udf中调用randomSeed() API会造成task阻塞，是不是因为int genRandSeed(char *rnd)实现中：open("/dev/random")... 在yarn容器中多个task调用会形成死锁?

bearzlh commented 2 months ago

java编译打包后在线程池并发执行，genRandSeed方法会造成阻塞，线程耗尽，这个问题如何处理呢，应该和楼主一个问题

"Threaded-256" #626 daemon prio=5 os_prio=0 tid=0x00007fe5f8821000 nid=0x4c77 runnable [0x00007fe561f66000] java.lang.Thread.State: RUNNABLE at com.volcengine.secureunionid.SecureUnionIDJNI.genRandSeed(Native Method) at com.volcengine.secureunionid.SecureUnionID.randomSeed(SecureUnionID.java:147)

阿里云实例规格：ecs.sn2ne.2xlarge 镜像名称：centos_7_02_64_20G_alibase_20170818.vhd

jiafeiwang commented 2 months ago

java编译打包后在线程池并发执行，genRandSeed方法会造成阻塞，线程耗尽，这个问题如何处理呢，应该和楼主一个问题

"Threaded-256" #626 daemon prio=5 os_prio=0 tid=0x00007fe5f8821000 nid=0x4c77 runnable [0x00007fe561f66000] java.lang.Thread.State: RUNNABLE at com.volcengine.secureunionid.SecureUnionIDJNI.genRandSeed(Native Method) at com.volcengine.secureunionid.SecureUnionID.randomSeed(SecureUnionID.java:147)

阿里云实例规格：ecs.sn2ne.2xlarge 镜像名称：centos_7_02_64_20G_alibase_20170818.vhd

把 encrypotion.c里面的 int fd = open("/dev/random", O_RDONLY) 换成 int fd = open("/dev/urandom", O_RDONLY) 重新编译打包试试。

参考：https://cloud.tencent.com/developer/article/1915057

bearzlh commented 2 months ago

@jiafeiwang 我们上线后测试是没有出现卡死的问题了，基本能用了，就是接口响应稍微慢点，cpu使用率高了不少