apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.3k stars 1.17k forks source link

[Bug] the method of `log` in Entrance can be lost log info #3742

Open guoshupei opened 1 year ago

guoshupei commented 1 year ago

Search before asking

Linkis Component

linkis-computation-governance

Steps to reproduce

If the script is executed for more than 5 minutes, and the log is actively added, the previous log will be lost.

Expected behavior

Call the log method, we can get the log normally

Your environment

Anything else

More than 5 minutes, the cachedLogs will write to hdfs . And execute fakeClear method that it set all element to null, but not reset the Attributes of flag, tail, realSize, so when realLog, it still read log from cachedLogs, but it empty. (默认超过5分钟,增加新日志会执行清除缓存并刷hdfs,但是清除缓存不会重置cachedLogs的flag, tail, realSize,所以当获取日志时,!sharedCache.cachedLogs.nonEmpty 不会为空,也就是不会从hdfs读取,还是从内存读取)

image

image

Are you willing to submit a PR?

github-actions[bot] commented 1 year ago

:blush: Welcome to the Apache Linkis (incubating) community!!

We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context. We will be here shortly.

If you are interested in contributing to our website project, please let us know! You can check out our contributing guide on :point_right: How to Participate in Project Contribution.

Community

WeChat Assistant WeChat Public Account

Mailing Lists

name description Subscribe Unsubscribe archive
dev@linkis.apache.org community activity information subscribe unsubscribe archive
guoshupei commented 1 year ago

How to understand the Attributes of flag, realSize in org.apache.linkis.entrance.log.LoopArray ?

peacewong commented 1 year ago

Hi guoshupei RealSize records the true size after the first ring, and if you want to read data that is less than realSize fromLine<realSize, you read it from the file

flag is used in conjunction with realSize to get the real position of index, for example, when index=9 and maxCapacity=5

array(4) value stores 4 more

tail=0 The next data that can be inserted is 0

flag 1 realSize-flag is always a multiple of 5 and is used to get the true index

realSize 6