search

kopia / kopia

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
https://kopia.io
Apache License 2.0
7.57k stars 384 forks source link

DownloadFileRangeByName: internal_error: internal incident #3854

Open santo74 opened 4 months ago

santo74 commented 4 months ago

Until yesterday my backups were running fine, but now I'm suddenly getting the following error when connecting to the repository:

DEBUG Creating cache directory '/root/.cache/kopia/a8d98c124c954dca' with max size 5242880000
DEBUG throttling limits from connection info    {"limits":{}}
DEBUG finished initial cache scan   {"cache":"contents","duration":"72.777µs","totalRetainedSize":0,"tooRecentBytes":0,"tooRecentCount":0,"maxSizeBytes":5242880000,"limitBytes":0,"inUsePercent":0}
DEBUG finished initial cache scan   {"cache":"metadata","duration":"39.023µs","totalRetainedSize":0,"tooRecentBytes":0,"tooRecentCount":0,"maxSizeBytes":5242880000,"limitBytes":0,"inUsePercent":0}
DEBUG finished initial cache scan   {"cache":"index-blobs","duration":"38.412µs","totalRetainedSize":0,"tooRecentBytes":0,"tooRecentCount":0,"maxSizeBytes":5242880000,"limitBytes":0,"inUsePercent":0}
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#0), sleeping for 100ms before retrying
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#1), sleeping for 150ms before retrying
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#2), sleeping for 225ms before retrying
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#3), sleeping for 337.5ms before retrying
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#4), sleeping for 506.25ms before retrying
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#5), sleeping for 759.375ms before retrying
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#6), sleeping for 1.1390625s before retrying
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#7), sleeping for 1.70859375s before retrying
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#8), sleeping for 2.562890625s before retrying
DEBUG got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#9), sleeping for 3.844335937s before retrying
ERROR failed to open repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1: getContent: unable to complete GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) despite 10 retries: DownloadFileRangeByName: internal_error: internal incident
ERROR error connecting to repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1: getContent: unable to complete GetBlob(xn53_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) despite 10 retries: DownloadFileRangeByName: internal_error: internal incident

The repository is on b2.

What does this error mean and how can I get this fixed? I already looked at this information but that doesn't resolve the issue.

ArjonBu commented 4 months ago

I am also having this problem. backups started failing a few days ago. 0.17.0 build: 89c8eb47af2e1d5c1d14fe299a0cf7eaac095abf from: kopia/kopia

Any idea what happened?

r3nor commented 4 months ago

Same here

r3nor commented 3 months ago

is there anything we can do? I am having constant errors:

[ERROR] unable to add xn188_527be6c148fc644adee20e191d98175a-s7f7dfaea3c41696b12a-c1 to index-blobs: context canceled
[ERROR] failed to open repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading

retrying does sometimes work, but it would be great if this was solved... I had no errors until this latest version...

julio-lopez commented 3 months ago

How is kopia configured to access B2 in this case? Is it via the B2 storage provider or the S3 one?

Note that the B2 storage provider is currently unmaintained.

r3nor commented 3 months ago

I see... I am using b2 format, I will test with s3 format then

r3nor commented 3 months ago

UPDATE: I was able to delete all conflicting xn blobs, and the repo is now working fine again

is there any way to solve the got error DownloadFileRangeByName: internal_error: internal incident when GetBlob(xn187_fc790a49a926e57377a8f0666526589f-sa2cc73bc2fd4a114128-c1,0,-1) (#0), sleeping for 100ms before retrying? I am getting this and I can't access my repo neither can I backup to it