search

Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.87k stars 2.94k forks source link

alluxio metadata sync issue #16194

Open flyhighzy opened 2 years ago

flyhighzy commented 2 years ago

Alluxio Version: 2.9.0-SNAPSHOT-5f981271cc825ddf0cf130caea330de0df5a4bca

Describe the bug our tasks use alluxio as data cache, and have met some empty files in alluxio but file on UFS is not empty. we found some logs about the empty file in alluxio master: image

when we use alluxio fs ls broken file's dir, found it's strange too: image

As we've set alluxio.user.file.metadata.sync.interval to default value "-1", so if some error happened during first metadata sync, the wrong state last forever?

To Reproduce Maybe large directories with about up to 30000 files and multiple process read can easily cause this problem

Expected behavior If some files' metadata sync failed, just left it unsynced and can have the chance to resync again.

Additional context Some other commands output about this broken file:

  1. no update executing alluxio fs ls -f image

  2. no node for the file executing alluxio fs location image

  3. when we delete this file from alluxio only(not changed in UFS), and do ls again, it's back to normal

LuQQiu commented 2 years ago

@HelloHorizon This looks like a high priority issue, cannot read correctly from Alluxio because of failed to sync

HelloHorizon commented 2 years ago

@flyhighzy do you mind trying the latest master with #16241 to see if it fixes the issue? If the issue insists, feel free to ping @elega /me. You need to set 'alluxio.master.metadata.concurrent.sync.dedup=true' to enable this feature.

SeananXu commented 2 years ago

I didn't reproduce the above problem according to your solution, When will an official version be released?

HelloHorizon commented 2 years ago

@SeananXu Thank you for confirmation. We will release 2.9 in one/two weeks