Open flaming-archer opened 1 year ago
Data monitored at the beginning:
I fixed with 3 prs : https://github.com/Alluxio/alluxio/pull/17006 https://github.com/Alluxio/alluxio/pull/16893 https://github.com/Alluxio/alluxio/pull/16944
Then the test results on our side:
I think the 3 prs could be merged. 😄 。@HelloHorizon @elega @jiacheliu3
The idea is that presto calls listStatus and getFileInfo do not need mountinfo information. When a directory has thousands of files and 500 mount points, it can reduce the number of calls to mount point resolution by at least hundreds of thousands.
Seen from the phenomenon, it has improved a lot of rpc processing capacity compared with the previous one.
But the performance is still not enough, especially when Alluxio is just started. Because I found that alliuxio will synchronize the inode after restarting. This process is particularly slow, and it will also parse the mount point.
Hi I can help the review.
Hi I can help the review. Thx. I found that the reason why the master started to synchronize is that we configured to synchronize once a day.
alluxio.user.file.metadata.sync.interval=1d
. When the master is started, the cache has no data, and it will be synchronized according to this.
FYI we are also working on a better mount point resolution using Trie. Also you can also try setting the sync interval to -1 and sync on your own on demand.
@elega @jiacheliu3 This https://github.com/Alluxio/alluxio/pull/16944 also helps me to review. Those test cases failed. It should not be my problem.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.
Is your feature request related to a problem? Please describe. When using Alluxio to accelerate Presto, we found that the Alluxio master rpc was heavily stacked.
Describe the solution you'd like
Describe alternatives you've considered Try to fix it myself and see where the performance bottleneck is full.
Urgency urgent
Additional context Alluxio cluster deployment, Speed up presto query.