search

apache / iotdb

Apache IoTDB
https://iotdb.apache.org/
Apache License 2.0
4.34k stars 980 forks source link

[Bug] Error WALBuffer....WAL-Recover Fail to allocate wal node....OutOfMemoryError: Cannot reserve 16777216 bytes of direct buffer memory (allocated: 469771426, limit: 471859200) #12714

Open automatous opened 1 month ago

automatous commented 1 month ago

Search before asking

Version

1.3.0 3C3D

Describe the bug and provide the minimal reproduce step

image

We upped the JVM memory and still reported this error

What did you expect to see?

Cluster running properly

What did you see instead?

We can expand the memory

Anything else?

No response

Are you willing to submit a PR?

HTHou commented 1 month ago

It may happen when too many databases created. A limitation added in v1.3.2 can avoid this issue. https://github.com/apache/iotdb/pull/12431

FutureHai commented 3 weeks ago

升级到1.3.2最新版本datanode还是报错，这个问题有修复吗？

image
FutureHai commented 3 weeks ago

@HTHou @automatous @wallezhang

HTHou commented 3 weeks ago

升级到1.3.2最新版本datanode还是报错，这个问题有修复吗？ image

1.3.2 从新部署的情况下可以避免这个问题，但是有数据的情况下升级是不行的。database 和 data region 的数量太多导致的，建议调整建模，不要创建过多的 database。

FutureHai commented 3 weeks ago

如果从新部署的情况下，数据库database太多的话还会不会出现上面的问题？ @HTHou

HTHou commented 3 weeks ago

如果从新部署的情况下，数据库database太多的话还会不会出现上面的问题？ @HTHou

新部署的话 database太多时会拒绝掉新的 data region 的创建，从而避免这个问题。

FutureHai commented 3 weeks ago

麻烦再问一下，这个数据库数量到什么值时会拒绝掉新的 data region 的创建，大致能创建多少个数据库。 @HTHou