Randomly getting the following error while uploading CAR file to web3.storage

[chaitanyaprem]

We use web3.storage as an archival storage and regularly upload CAR files once in 10 hours via the HTTP API /car. We have a rate-limiter in place to ensure we don't hit rate-limits for the API's.

These file sizes should be around few MB's in size (1-10MB) mostly.

Randomly we have been noticing following errors returned from web3.storage. For eg: In the process of uploading 356 CAR files, 342 have been successfuly and 14 have failed with 500 errors along with below messages. We have a retry mechanism on failure for 3 times, and first time we receive the message Failed to write satnav to R2, and on retry we receive Could not decode varint for the same file.

{Name: Message:Failed to write satnav index to R2}
{Name: Message:Could not decode varint}

Would be great to know what could be causing this and if so, is there any workaround to address the same.

[chaitanyaprem]

Today, have noticed this error even with the file upload API at endpoint /uploadas well. The files we are trying to upload are of small size in this case, probably 1MB or so. {Name: Message:Failed to write satnav index to R2}

[yusefnapora]

@chaitanyaprem is it always the same CARs that are failing to upload? If so, it would be great if you could link us to a "known bad" CAR, so we can inspect it.

@alanshaw or @olizilla might know more about the satnav error?

[chaitanyaprem]

Not necessarily, have noticed this happening with different CAR files at different times randomly. Also noticed that they ended up getting uploaded in next run in some cases.

Here is one of the CAR file where we faced this error for your reference. Let me know if any further information would help resolve this.

This is the root CID of the DAG that is part of one of the CAR's that failed upload multiple times. We do a dag export of this CID from ipfs node and then upload it to web3.storage.

bafyreicpz2rcpop6ao5uo3uu3pe6tlxredq457kypir7vm2p2laxgooqga