search

juicedata / juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.
https://juicefs.com
Apache License 2.0
10.95k stars 967 forks source link

Perform object storage test at startup #5281

Closed polyrabbit closed 2 weeks ago

polyrabbit commented 2 weeks ago

Test object storage at startup to fail fast instead of throwing EIO in the middle of user's workload.

davies commented 2 weeks ago

It's slow to test, which will slow down the mount.

The AK/SK is verified in format/config, which should be enough.

polyrabbit commented 2 weeks ago

format only verifies the AK/SK but cannot check user's environment. A typical scenario is when a node cannot connect to OSS for various reasons, such as DNS issues or VPC network isolation problems. In such cases, users may encounter EIO errors in the middle of his task. If we can perform some cheap self-checks at startup to expose errors early, we can avoid expensive user-side errors.

davies commented 2 weeks ago

It could be useful in some cases, should be enabled by a flag.

polyrabbit commented 2 weeks ago

Do you have a suggestion for this flag's name?

davies commented 2 weeks ago

--check-storage?

polyrabbit commented 2 weeks ago

Updated. Please review, thanks!

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 29.41176% with 12 lines in your changes missing coverage. Please review.

Project coverage is 53.87%. Comparing base (c2b21f4) to head (c20ebe7). Report is 70 commits behind head on main.

Files with missing lines Patch % Lines
cmd/mount.go 0.00% 10 Missing :warning:
cmd/format.go 0.00% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #5281 +/- ## ========================================== - Coverage 54.27% 53.87% -0.40% ========================================== Files 161 162 +1 Lines 43776 44409 +633 ========================================== + Hits 23758 23924 +166 - Misses 17336 17772 +436 - Partials 2682 2713 +31 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.