"Missing MD5 property in Manifest and Azure Blob" error for 16GB file using Migration API

[Lewin8687]

Target SharePoint environment

SharePoint Online

What SharePoint development model, framework, SDK or API is this about?

other (enter in the "Additional environment details" area below)

Developer environment

Windows

What browser(s) / client(s) have you tested

• [ ] 💥 Internet Explorer
• [ ] 💥 Microsoft Edge
• [ ] 💥 Google Chrome
• [ ] 💥 FireFox
• [ ] 💥 Safari
• [ ] mobile (iOS/iPadOS)
• [ ] mobile (Android)
• [X] not applicable
• [ ] other (enter in the "Additional environment details" area below)

Additional environment details

• browser version
• SPFx version
• Node.js version
• etc

Describe the bug / error

When migrating a 16GB size file to SPO library using Migration API, it throws me error "Missing MD5 property in Manifest and Azure Blob."

Here is the full stack trace: Missing MD5 property in Manifest and Azure Blob. File: 1317_ CallStack -- at Microsoft.SharePoint.AzureHelperObj.VerifyAndPickMD5HashOnManifestAndAzBlob(String manifestMD5, String azureMD5, String blobName, SPReliabilityMonitor monitor) at Microsoft.SharePoint.AzureHelperObj.DownloadToStream(SPWeb web, String containerSas, String blobName, String manifestMD5, String manifestIV, String quickXorHash, SPWorkItem workItem, Stream& stm, String& contentMD5, Int64& bytesInAzure, DecryptionInfo& decryptionInfo) at Microsoft.SharePoint.Deployment.FileSerializer.GetTargetStream(SPImportSettings settings, FileInfo fileInfo, String fileValue, SPWeb web, SPLoggerObject loggerObject, SerializationInfoHelper infoHelper, Int64& bytesInAzure, String& quickXorHash, DecryptionInfo& decryptionInfo)

But based on the documentation here, MD5 is only required for files smaller than 2GB,https://docs.microsoft.com/en-us/sharepoint/dev/apis/migration-api-overview#api-documention

Did I misunderstand it? Is MD5 required for pretty much all size files?

Steps to reproduce

1. Prepare a 16GB file
2. Create azure container to contain the file
3. Create azure container to contain all needed manifest files
4. Start the job using migration api

Expected behavior

The file larger than 15GB should only require XorHash in order to be migrated.

[ghost]

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

[JakeStanger]

Hey, I hit a similar issue, mostly in case anybody stumbles across this error as the only result as I did.

The error was occurring when downloading the manifest files:

[5/11/2023 3:34:11 AM] Start Time: 5/11/2023 3:34:11 AM.
[5/11/2023 3:34:11 AM] Correlation Id: cfc8b1a0-e071-6000-97b1-65e3f4215352 
[5/11/2023 3:34:11 AM] [Progress] Initializing Import.
[5/11/2023 3:34:11 AM] [Error] Unable to download SystemData.xml with exception 'Missing MD5 property in Manifest and Azure Blob.'
[5/11/2023 3:34:11 AM] [Error] Unable to download Requirements.xml with exception 'Missing MD5 property in Manifest and Azure Blob.'
[5/11/2023 3:34:11 AM] [Warning] Import requirement file Requirements.xml was not found no verifications ran.
[5/11/2023 3:34:11 AM] [Error] Unable to download ExportSettings.xml with exception 'Missing MD5 property in Manifest and Azure Blob.'
[5/11/2023 3:34:11 AM] [FatalError] [-2147024894] [System.IO.FileNotFoundException] [Missing Export Settings file stream] 
[5/11/2023 3:34:11 AM] [Debug]    at Microsoft.SharePoint.Deployment.SPImport.DeserializeExportSettings()
   at Microsoft.SharePoint.Deployment.SPImport.Run()
[5/11/2023 3:34:11 AM] [Progress] Import did not complete.
[5/11/2023 3:34:11 AM] Finish Time: 5/11/2023 3:34:11 AM.
[5/11/2023 3:34:11 AM] Duration: 00:00:00
[5/11/2023 3:34:11 AM] Byte imported: 0.00 at 0.00 bytes per second
[5/11/2023 3:34:11 AM] Total Processed Objects: 0
[5/11/2023 3:34:11 AM] Finished with 1 warnings.
[5/11/2023 3:34:11 AM] Finished with 4 errors.

The docs do allude to it, although it is burried. Every file under 2GB, including the XML package, must have the Content-MD5 property set on their blob. In the case of content files, I believe you can also set this in the Manifest XML.

These MD5s have to be base64 encoded. The following sheds some light on doing that: https://powers-hell.com/2021/12/31/calculate-validate-md5-hashes-on-azure-blob-storage-files-with-powershell/

I've not tested for files > 15GB yet. If you have only tested with a 16GB file, this may be a discrepancy between GB and GiB and it is actually still under the limit.