feat: add retry mechanism in read blob metadata

[Desiki-high]

Relevant Issue (if applicable)

https://github.com/dragonflyoss/nydus/issues/1613

Details

When reading blob size from blob metadata, we should retry reading from the remote if an error occurs.

• Handle the BackendError.
• Set the max retry times to 3.

Types of changes

What types of changes does your PullRequest introduce? Put an x in all the boxes that apply:

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Documentation Update (if none of the other choices apply)

Checklist

Go over all the following points, and put an x in all the boxes that apply.

• [x] I have updated the documentation accordingly.
• [x] I have added tests to cover my changes.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 27.27273% with 16 lines in your changes missing coverage. Please review.

Project coverage is 61.27%. Comparing base (7fc9ede) to head (e04b5ca). Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
storage/src/meta/mod.rs	27.27%	15 Missing and 1 partial :warning:

Additional details and impacted files

[](https://app.codecov.io/gh/dragonflyoss/nydus/pull/1617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) ```diff @@ Coverage Diff @@ ## master #1617 +/- ## ========================================== - Coverage 61.29% 61.27% -0.02% ========================================== Files 146 146 Lines 48143 48156 +13 Branches 46110 46123 +13 ========================================== + Hits 29507 29509 +2 - Misses 17075 17085 +10 - Partials 1561 1562 +1 ``` | [Files with missing lines](https://app.codecov.io/gh/dragonflyoss/nydus/pull/1617?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) | Coverage Δ | | |---|---|---| | [storage/src/meta/mod.rs](https://app.codecov.io/gh/dragonflyoss/nydus/pull/1617?src=pr&el=tree&filepath=storage%2Fsrc%2Fmeta%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-c3RvcmFnZS9zcmMvbWV0YS9tb2QucnM=) | `72.21% <27.27%> (-0.31%)` | :arrow_down: | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/dragonflyoss/nydus/pull/1617/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)