search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.22k stars 3.83k forks source link

File corruption issue when downloading very large blobs #10298

Closed paulmrozowski closed 4 years ago

paulmrozowski commented 4 years ago

Description

We are using Get-AzStorageBlobContent in a Powershell script. It seemed to work consistently well for a while but now that one of the blobs we download is over approx 95GB we are seeing that the file downloaded is corrupt - it appears that around #16 3300 000 (hex) suddenly the file is just filled with 00 values. We don't see any errors in $DownloadError after it completes. As far as it's concerned, it was successful.

If we manually download the file using AzCopy instead, it downloads successfully.

Steps to reproduce

Get-AzStorageBlobContent -Container $StorageContainerName -Blob $FileToDownload.FileName -Destination $DownloadPath -Context $StorageContext -ErrorAction SilentlyContinue -ErrorVariable DownloadError

Environment data

Name                           Value                                                                                                         
----                           -----                                                                                                         
PSVersion                      5.1.14393.3053                                                                                                
PSEdition                      Desktop                                                                                                       
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                       
BuildVersion                   10.0.14393.3053                                                                                               
CLRVersion                     4.0.30319.42000                                                                                               
WSManStackVersion              3.0                                                                                                           
PSRemotingProtocolVersion      2.3                                                                                                           
SerializationVersion           1.1.0.1

Module versions


    Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version    Name                                ExportedCommands                                                                   
---------- -------    ----                                ----------------                                                                   
Script     1.6.3      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, E...
Script     1.6.1      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, E...
Script     1.5.2      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, E...
Script     1.5.1      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, E...
Script     1.0.1      Az.Advisor                          {Get-AzAdvisorRecommendation, Enable-AzAdvisorRecommendation, Disable-AzAdvisorR...
Script     1.0.2      Az.Aks                              {Get-AzAks, New-AzAks, Remove-AzAks, Import-AzAksCredential...}                    
Script     1.1.1      Az.AnalysisServices                 {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAnalys...
Script     1.3.2      Az.ApiManagement                    {Add-AzApiManagementApiToProduct, Add-AzApiManagementProductToGroup, Add-AzApiMa...
Script     1.0.2      Az.ApplicationInsights              {Get-AzApplicationInsights, New-AzApplicationInsights, Remove-AzApplicationInsig...
Script     1.3.4      Az.Automation                       {Get-AzAutomationHybridWorkerGroup, Remove-AzAutomationHybridWorkerGroup, Get-Az...
Script     1.1.2      Az.Batch                            {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAccountKey, New-AzBatchAc...
Script     1.0.1      Az.Billing                          {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollmentAccount, Get-AzConsu...
Script     1.3.1      Az.Cdn                              {Get-AzCdnProfile, Get-AzCdnProfileSsoUrl, New-AzCdnProfile, Remove-AzCdnProfile...
Script     1.2.1      Az.CognitiveServices                {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCognit...
Script     2.7.0      Az.Compute                          {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Update-...
Script     1.0.1      Az.ContainerInstance                {New-AzContainerGroup, Get-AzContainerGroup, Remove-AzContainerGroup, Get-AzCont...
Script     1.1.0      Az.ContainerRegistry                {New-AzContainerRegistry, Get-AzContainerRegistry, Update-AzContainerRegistry, R...
Script     1.4.0      Az.DataFactory                      {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFactoryV2, Remove-AzData...
Script     1.0.1      Az.DataLakeAnalytics                {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalyticsCatalogCredential, Re...
Script     1.2.3      Az.DataLakeStore                    {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreTrustedIdProvider, ...
Script     1.0.1      Az.DeploymentManager                {Get-AzDeploymentManagerArtifactSource, New-AzDeploymentManagerArtifactSource, S...
Script     1.0.0      Az.DevTestLabs                      {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartP...
Script     1.1.1      Az.Dns                              {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRecordSet, Set-AzDnsReco...
Script     1.2.2      Az.EventGrid                        {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGridTopic, New-AzEventGr...
Script     1.4.0      Az.EventHub                         {New-AzEventHubNamespace, Get-AzEventHubNamespace, Set-AzEventHubNamespace, Remo...
Script     1.1.1      Az.FrontDoor                        {New-AzFrontDoor, Get-AzFrontDoor, Set-AzFrontDoor, Remove-AzFrontDoor...}         
Script     2.0.2      Az.HDInsight                        {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wait-AzHDInsightJob, New...
Script     1.0.0      Az.HealthcareApis                   {New-AzHealthcareApisService, Remove-AzHealthcareApisService, Set-AzHealthcareAp...
Script     1.3.1      Az.IotHub                           {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-AzIotHubConnectionStrin...
Script     1.3.1      Az.KeyVault                         {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCertifi...
Script     1.3.1      Az.LogicApp                         {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-AzInte...
Script     1.1.1      Az.MachineLearning                  {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssociation, Get-AzMlCommitme...
Script     1.0.1      Az.ManagedServices                  {Get-AzManagedServicesAssignment, New-AzManagedServicesAssignment, Remove-AzMana...
Script     1.0.1      Az.MarketplaceOrdering              {Get-AzMarketplaceTerms, Set-AzMarketplaceTerms}                                   
Script     1.1.0      Az.Media                            {Sync-AzMediaServiceStorageKey, Set-AzMediaServiceKey, Get-AzMediaServiceKey, Ge...
Script     1.4.0      Az.Monitor                          {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile...}   
Script     1.15.0     Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGatewayAuth...
Script     1.1.0      Az.NotificationHubs                 {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRule, Get-AzNotificati...
Script     1.3.3      Az.OperationalInsights              {New-AzOperationalInsightsAzureActivityLogDataSource, New-AzOperationalInsightsC...
Script     1.1.3      Az.PolicyInsights                   {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSummary, Get-AzPolicyRem...
Script     1.1.0      Az.PowerBIEmbedded                  {Remove-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollection, Get-AzPo...
Script     1.4.5      Az.RecoveryServices                 {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRecove...
Script     1.1.1      Az.RedisCache                       {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheScheduleEntry, Get-AzRedisCac...
Script     1.0.2      Az.Relay                            {New-AzRelayNamespace, Get-AzRelayNamespace, Set-AzRelayNamespace, Remove-AzRela...
Script     1.7.0      Az.Resources                        {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzR...
Script     1.6.1      Az.Resources                        {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzR...
Script     1.3.1      Az.Resources                        {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzR...
Script     1.4.0      Az.ServiceBus                       {New-AzServiceBusNamespace, Get-AzServiceBusNamespace, Set-AzServiceBusNamespace...
Script     1.2.0      Az.ServiceFabric                    {Add-AzServiceFabricApplicationCertificate, Add-AzServiceFabricClientCertificate...
Script     1.1.0      Az.SignalR                          {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSignalRKey...}              
Script     1.15.0     Az.Sql                              {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseTransparentDataEnc...
Script     1.8.0      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStor...
Script     1.5.1      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStor...
Script     1.3.0      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStor...
Script     1.2.0      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStor...
Script     1.2.1      Az.StorageSync                      {Invoke-AzStorageSyncCompatibilityCheck, New-AzStorageSyncService, Get-AzStorage...
Script     1.0.0      Az.StreamAnalytics                  {Get-AzStreamAnalyticsFunction, Get-AzStreamAnalyticsDefaultFunctionDefinition, ...
Script     1.0.2      Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTrafficManagerCustomHeader...
Script     1.5.0      Az.Websites                         {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-AzAppS...
Manifest   2.0.2      AzTable                             {Add-AzTableRow, Get-AzTableRow, Get-AzTableRowAll, Get-AzTableRowByPartitionKey...
Script     2.7.2      Logging                             {Add-LoggingLevel, Add-LoggingTarget, Get-LoggingCallerScope, Get-LoggingDefault...
Script     1.0.1      Microsoft.PowerShell.Operation.V... {Get-OperationValidation, Invoke-OperationValidation}                              
Binary     1.0.0.1    PackageManagement                   {Find-Package, Get-Package, Get-PackageProvider, Get-PackageSource...}             
Script     3.4.0      Pester                              {Describe, Context, It, Should...}                                                 
Script     1.0.0.1    PowerShellGet                       {Install-Module, Find-Module, Save-Module, Update-Module...}                       
Script     1.2        PSReadline                          {Get-PSReadlineKeyHandler, Set-PSReadlineKeyHandler, Remove-PSReadlineKeyHandler...
Script     21.1.18102 SqlServer                           {Add-RoleMember, Add-SqlAvailabilityDatabase, Add-SqlAvailabilityGroupListenerSt...

    Directory: C:\Windows\system32\WindowsPowerShell\v1.0\Modules

ModuleType Version    Name                                ExportedCommands                                                                   
---------- -------    ----                                ----------------                                                                   
Manifest   1.0.0.0    AppBackgroundTask                   {Disable-AppBackgroundTaskDiagnosticLog, Enable-AppBackgroundTaskDiagnosticLog, ...
Manifest   2.0.0.0    AppLocker                           {Get-AppLockerFileInformation, Get-AppLockerPolicy, New-AppLockerPolicy, Set-App...
Manifest   1.0.0.0    AppvClient                          {Add-AppvClientConnectionGroup, Add-AppvClientPackage, Add-AppvPublishingServer,...
Manifest   2.0.0.0    Appx                                {Add-AppxPackage, Get-AppxPackage, Get-AppxPackageManifest, Remove-AppxPackage...} 
Script     1.0.0.0    AssignedAccess                      {Clear-AssignedAccess, Get-AssignedAccess, Set-AssignedAccess}                     
Manifest   1.0        BestPractices                       {Get-BpaModel, Get-BpaResult, Invoke-BpaModel, Set-BpaResult}                      
Manifest   2.0.0.0    BitsTransfer                        {Add-BitsFile, Complete-BitsTransfer, Get-BitsTransfer, Remove-BitsTransfer...}    
Manifest   1.0.0.0    BranchCache                         {Add-BCDataCacheExtension, Clear-BCCache, Disable-BC, Disable-BCDowngrading...}    
Manifest   1.0.0.0    CimCmdlets                          {Get-CimAssociatedInstance, Get-CimClass, Get-CimInstance, Get-CimSession...}      
Manifest   1.0        ConfigCI                            {Get-SystemDriver, New-CIPolicyRule, New-CIPolicy, Get-CIPolicy...}                
Manifest   1.0        Defender                            {Get-MpPreference, Set-MpPreference, Add-MpPreference, Remove-MpPreference...}     
Binary     2.0.0.0    DFSR                                {New-DfsReplicationGroup, Get-DfsReplicationGroup, Set-DfsReplicationGroup, Remo...
Manifest   1.0.0.0    DirectAccessClientComponents        {Disable-DAManualEntryPointSelection, Enable-DAManualEntryPointSelection, Get-DA...
Script     3.0        Dism                                {Add-AppxProvisionedPackage, Add-WindowsDriver, Add-WindowsCapability, Add-Windo...
Manifest   1.0.0.0    DnsClient                           {Resolve-DnsName, Clear-DnsClientCache, Get-DnsClient, Get-DnsClientCache...}      
Manifest   1.0.0.0    EventTracingManagement              {New-EtwTraceSession, Get-EtwTraceSession, Set-EtwTraceSession, Send-EtwTraceSes...
Manifest   2.0.0.0    International                       {Get-WinDefaultInputMethodOverride, Set-WinDefaultInputMethodOverride, Get-WinHo...
Manifest   1.0.0.0    iSCSI                               {Get-IscsiTargetPortal, New-IscsiTargetPortal, Remove-IscsiTargetPortal, Update-...
Manifest   2.0.0.0    IscsiTarget                         {Add-ClusteriSCSITargetServerRole, Add-IscsiVirtualDiskTargetMapping, Checkpoint...
Script     1.0.0.0    ISE                                 {New-IseSnippet, Import-IseSnippet, Get-IseSnippet}                                
Manifest   1.0.0.0    Kds                                 {Add-KdsRootKey, Get-KdsRootKey, Test-KdsRootKey, Set-KdsConfiguration...}         
Manifest   1.0.1.0    Microsoft.PowerShell.Archive        {Compress-Archive, Expand-Archive}                                                 
Manifest   3.0.0.0    Microsoft.PowerShell.Diagnostics    {Get-WinEvent, Get-Counter, Import-Counter, Export-Counter...}                     
Manifest   3.0.0.0    Microsoft.PowerShell.Host           {Start-Transcript, Stop-Transcript}                                                
Manifest   1.0.0.0    Microsoft.PowerShell.LocalAccounts  {Add-LocalGroupMember, Disable-LocalUser, Enable-LocalUser, Get-LocalGroup...}     
Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Content, Clear-Content, Clear-ItemProperty, Join-Path...}                     
Script     1.0        Microsoft.PowerShell.ODataUtils     Export-ODataEndpointProxy                                                          
Manifest   3.0.0.0    Microsoft.PowerShell.Security       {Get-Acl, Set-Acl, Get-PfxCertificate, Get-Credential...}                          
Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Format-List, Format-Custom, Format-Table, Format-Wide...}                         
Manifest   3.0.0.0    Microsoft.WSMan.Management          {Disable-WSManCredSSP, Enable-WSManCredSSP, Get-WSManCredSSP, Set-WSManQuickConf...
Manifest   1.0        MMAgent                             {Disable-MMAgent, Enable-MMAgent, Set-MMAgent, Get-MMAgent...}                     
Manifest   1.0.0.0    MsDtc                               {New-DtcDiagnosticTransaction, Complete-DtcDiagnosticTransaction, Join-DtcDiagno...
Manifest   2.0.0.0    NetAdapter                          {Disable-NetAdapter, Disable-NetAdapterBinding, Disable-NetAdapterChecksumOffloa...
Manifest   1.0.0.0    NetConnection                       {Get-NetConnectionProfile, Set-NetConnectionProfile}                               
Manifest   1.0.0.0    NetEventPacketCapture               {New-NetEventSession, Remove-NetEventSession, Get-NetEventSession, Set-NetEventS...
Manifest   2.0.0.0    NetLbfo                             {Add-NetLbfoTeamMember, Add-NetLbfoTeamNic, Get-NetLbfoTeam, Get-NetLbfoTeamMemb...
Manifest   1.0.0.0    NetNat                              {Get-NetNat, Get-NetNatExternalAddress, Get-NetNatStaticMapping, Get-NetNatSessi...
Manifest   2.0.0.0    NetQos                              {Get-NetQosPolicy, Set-NetQosPolicy, Remove-NetQosPolicy, New-NetQosPolicy}        
Manifest   2.0.0.0    NetSecurity                         {Get-DAPolicyChange, New-NetIPsecAuthProposal, New-NetIPsecMainModeCryptoProposa...
Manifest   1.0.0.0    NetSwitchTeam                       {New-NetSwitchTeam, Remove-NetSwitchTeam, Get-NetSwitchTeam, Rename-NetSwitchTea...
Manifest   1.0.0.0    NetTCPIP                            {Get-NetIPAddress, Get-NetIPInterface, Get-NetIPv4Protocol, Get-NetIPv6Protocol...}
Manifest   1.0.0.0    NetworkConnectivityStatus           {Get-DAConnectionStatus, Get-NCSIPolicyConfiguration, Reset-NCSIPolicyConfigurat...
Manifest   1.0.0.0    NetworkSwitchManager                {Disable-NetworkSwitchEthernetPort, Enable-NetworkSwitchEthernetPort, Get-Networ...
Manifest   1.0.0.0    NetworkTransition                   {Add-NetIPHttpsCertBinding, Disable-NetDnsTransitionConfiguration, Disable-NetIP...
Manifest   1.0        NFS                                 {Get-NfsMappedIdentity, Get-NfsNetgroup, Install-NfsMappingStore, New-NfsMappedI...
Manifest   1.0.0.0    PcsvDevice                          {Get-PcsvDevice, Start-PcsvDevice, Stop-PcsvDevice, Restart-PcsvDevice...}         
Manifest   1.0.0.0    PKI                                 {Add-CertificateEnrollmentPolicyServer, Export-Certificate, Export-PfxCertificat...
Manifest   1.0.0.0    PlatformIdentifier                  Get-PlatformIdentifier                                                             
Manifest   1.0.0.0    PnpDevice                           {Get-PnpDevice, Get-PnpDeviceProperty, Enable-PnpDevice, Disable-PnpDevice}        
Manifest   1.1        PrintManagement                     {Add-Printer, Add-PrinterDriver, Add-PrinterPort, Get-PrintConfiguration...}       
Manifest   1.1        PSDesiredStateConfiguration         {Set-DscLocalConfigurationManager, Start-DscConfiguration, Test-DscConfiguration...
Script     1.0.0.0    PSDiagnostics                       {Disable-PSTrace, Disable-PSWSManCombinedTrace, Disable-WSManTrace, Enable-PSTra...
Binary     1.1.0.0    PSScheduledJob                      {New-JobTrigger, Add-JobTrigger, Remove-JobTrigger, Get-JobTrigger...}             
Manifest   2.0.0.0    PSWorkflow                          {New-PSWorkflowExecutionOption, New-PSWorkflowSession, nwsn}                       
Manifest   1.0.0.0    PSWorkflowUtility                   Invoke-AsWorkflow                                                                  
Manifest   2.0.0.0    RemoteDesktop                       {Get-RDCertificate, Set-RDCertificate, New-RDCertificate, New-RDVirtualDesktopDe...
Manifest   1.0.0.0    ScheduledTasks                      {Get-ScheduledTask, Set-ScheduledTask, Register-ScheduledTask, Unregister-Schedu...
Manifest   2.0.0.0    SecureBoot                          {Confirm-SecureBootUEFI, Set-SecureBootUEFI, Get-SecureBootUEFI, Format-SecureBo...
Manifest   1.0.0.0    SecurityCmdlets                     {Backup-SecurityPolicy, Restore-SecurityPolicy, Backup-AuditPolicy, Restore-Audi...
Script     1.0.0.0    ServerCore                          {Get-DisplayResolution, Set-DisplayResolution}                                     
Script     2.0.0.0    ServerManager                       {Get-WindowsFeature, Install-WindowsFeature, Uninstall-WindowsFeature, Enable-Se...
Cim        1.0.0.0    ServerManagerTasks                  {Get-SMCounterSample, Get-SMPerformanceCollector, Start-SMPerformanceCollector, ...
Manifest   2.0.0.0    SmbShare                            {Get-SmbShare, Remove-SmbShare, Set-SmbShare, Block-SmbShareAccess...}             
Manifest   2.0.0.0    SmbWitness                          {Get-SmbWitnessClient, Move-SmbWitnessClient, gsmbw, msmbw...}                     
Manifest   2.0.0.0    SoftwareInventoryLogging            {Get-SilComputer, Get-SilComputerIdentity, Get-SilSoftware, Get-SilWindowsUpdate...
Manifest   1.0.0.0    StartLayout                         {Export-StartLayout, Import-StartLayout, Get-StartApps}                            
Manifest   2.0.0.0    Storage                             {Add-InitiatorIdToMaskingSet, Add-PartitionAccessPath, Add-PhysicalDisk, Add-Tar...
Manifest   2.0.0.0    TLS                                 {New-TlsSessionTicketKey, Enable-TlsSessionTicketKey, Disable-TlsSessionTicketKe...
Manifest   1.0.0.0    TroubleshootingPack                 {Get-TroubleshootingPack, Invoke-TroubleshootingPack}                              
Manifest   2.0.0.0    TrustedPlatformModule               {Get-Tpm, Initialize-Tpm, Clear-Tpm, Unblock-Tpm...}                               
Binary     2.1.639.0  UEV                                 {Clear-UevConfiguration, Clear-UevAppxPackage, Restore-UevBackup, Set-UevTemplat...
Manifest   1.0.0.0    UserAccessLogging                   {Enable-Ual, Disable-Ual, Get-Ual, Get-UalDns...}                                  
Manifest   2.0.0.0    VpnClient                           {Add-VpnConnection, Set-VpnConnection, Remove-VpnConnection, Get-VpnConnection...} 
Manifest   1.0.0.0    Wdac                                {Get-OdbcDriver, Set-OdbcDriver, Get-OdbcDsn, Add-OdbcDsn...}                      
Manifest   2.0.0.0    Whea                                {Get-WheaMemoryPolicy, Set-WheaMemoryPolicy}                                       
Manifest   1.0.0.0    WindowsDeveloperLicense             {Get-WindowsDeveloperLicense, Unregister-WindowsDeveloperLicense, Show-WindowsDe...
Script     1.0        WindowsErrorReporting               {Enable-WindowsErrorReporting, Disable-WindowsErrorReporting, Get-WindowsErrorRe...
Manifest   1.0.0.0    WindowsSearch                       {Get-WindowsSearchSetting, Set-WindowsSearchSetting}                               
Manifest   1.0.0.0    WindowsUpdate                       Get-WindowsUpdateLog                                                               

    Directory: C:\Program Files (x86)\Microsoft SQL Server\130\Tools\PowerShell\Modules

ModuleType Version    Name                                ExportedCommands                                                                   
---------- -------    ----                                ----------------                                                                   
Manifest   1.0        SQLPS                               {Backup-SqlDatabase, Save-SqlMigrationReport, Add-SqlAvailabilityDatabase, Add-S...

    Directory: C:\Program Files (x86)\Microsoft SQL Server\140\Tools\PowerShell\Modules

ModuleType Version    Name                                ExportedCommands                                                                   
---------- -------    ----                                ----------------                                                                   
Manifest   14.0       SQLPS                               {Backup-SqlDatabase, Save-SqlMigrationReport, Add-SqlAvailabilityDatabase, Add-S...

Debug output

This didn't show anything beyond "Yep, starting 20 remote calls", then "Finished 20 remote calls" along with an operation ID. No errors.

Error output

This is not generating any errors.

blueww commented 4 years ago

@paulmrozowski

Which version of AzCopy do you use? Does this consistently repro? (It seems the download is not finished and fail/stop in the middle.)

Would you please run it with following command and see if any error happens , if the file still not download correctly? Get-AzStorageBlobContent -Container $StorageContainerName -Blob $FileToDownload.FileName -Destination $DownloadPath -Context $StorageContext

Is it possible that the $DownloadError is overwritten by other command?

Besides that , do you mind to share the source file URI and download time, so I can see the server log to investigate it.

paulmrozowski commented 4 years ago

AzCopy works fine (v10.2.1), but the script uses Get-AzStorageBlobContent and that's where the problem is. Yes, this now consistently fails after running successfully for a few months.

We know the download is corrupt (and NEVER successfully downloads anymore) because I've tried this manually - the script does an MD5 verification against the file. I also manually downloaded the file via Get-AzStorageBlobContent and another copy via AzCopy and did a file comparison while trying to fix this.

I don't believe there is any other place where $DownloadError would be changed. This download is wrapped in a do...while to automatically attempt 1 retry on the download. We push this info into a log and into Slack.

do 
{
   # Code to download file here
} while ($DownloadError[0].Exception.Message -like "*The client could not finish the operation within specified timeout*")

    if (-not($DownloadError[0]) -and -not($DownloadTimeout))
    {
      Write-Log -Level 'SLACK' -Message "*Nightly Restore:* Successfully Downloaded [ *{0}* ]"`
        -Arguments $FileToDownload.FileName
      return $True
      # Write-Log -Level 'SLACK' -Message "*Nightly Restore:* Successfully Downloaded [ *{0}* ], Time Taken: *{1}* Minutes"`
      #   -Arguments $FileToDownload.FileName, $([Math]::Round($Script:DownloadTime.TotalMinutes, 2))
      # return $True
    }
    elseif ($DownloadTimeout)
    {
      Write-Log -Level 'SLACK' -Message "*Nightly Restore:* Download timeout after two attempts of [ {0} ]. ``ERROR:`` *{1}*"`
        -Arguments $FileToDownload.FileName, $DownloadError[0].Exception.Message
      return $False
    }
    else
    {
      Write-Log -Level 'SLACK' -Message "*Nightly Restore:* There was an error downloading [ {0} ]. ``ERROR:`` *{1}*"`
        -Arguments $FileToDownload.FileName, $DownloadError[0].Exception.Message
      return $False
    }

Is there a way I can pull the server log for this vs. sharing the URL?

blueww commented 4 years ago

@paulmrozowski I have tested Get-AzStorageBlobContent with Az.Storage module 1.8.0 on my machine, and not see this issue repro by download one 100GB blob. After the download fail with "The client could not finish the operation within specified timeout.", it will put the error to $DownloadError, and per your code, it should retry.

And I don't see you add "-Force" parameter to "Get-AzStorageBlobContent". If so, retry will get a pops up to ask for overwrite the existing file, and the script can't be run with none interactive.

Could you please share the real script you run for "# Code to download file here", please hide any credential?

And for the server log, if you can give the account name, container name, blob name, and the repro time, I can search the server log. If you can't past it here, you can send to my mail weiwei@microsoft.com.

paulmrozowski commented 4 years ago

This is the full method. We delete the file in the real code before the retry.

function Get-DownloadFiles
{
  param ($FileToDownload)

  $CurrentFiles = Get-ChildItem -Path $DownloadPath -File -Filter *.bak
  Write-Log -Level "INFO" -Message "Processing {0} file(s) for Download" -Arguments @($FileToDownload).Count
  if ($CurrentFiles | Where-Object { $_.Name -eq $FileToDownload.FileName })
  {
    Write-Log -Level 'INFO' -Message "File found locally so will skip"
    return $True
  }
  else
  {
    Write-Log -Level 'INFO' -Message "Downloading: {0}" -Arguments $FileToDownload.FileName
    # TODO: Wrap this in a DoWHile loop for the Client Timeout error but keep track and maybe only try like 3 times at most, then throw an error
    # Need to also cleanup the failed file else there won't be enough spae and it is corrupt anyway
    $DownloadCounter = 0
    $DownloadTimeout = $False
    do
    {
      $DownloadCounter ++
      if ($DownloadCounter -gt 1)
      {
        Write-Log -Level 'INFO' -Message "Error Counter Value: [$DownloadCounter of 2]. Deleting existing download to retry"
        Remove-BackupFiles -FileNameToDelete $FileToDownload.FileName -CallingFunction "Get-DownloadFiles"
      }
      if ($DownloadCounter -gt 2)
      {
        Write-Log -Level 'INFO' -Message "Error Counter Value: [$DownloadCounter of 2].  Download Operation Timeout"
        $DownloadTimeout = $True
        Break
      }
      else
      {        
        Get-AzStorageBlobContent -Container $StorageContainerName -Blob $FileToDownload.FileName -Destination $DownloadPath -Context $StorageContext -ErrorAction SilentlyContinue -ErrorVariable DownloadError

      }
    } while ($DownloadError[0].Exception.Message -like "*The client could not finish the operation within specified timeout*")

    if (-not($DownloadError[0]) -and -not($DownloadTimeout))
    {
      Write-Log -Level 'SLACK' -Message "*Nightly Restore:* Successfully Downloaded [ *{0}* ]"`
        -Arguments $FileToDownload.FileName
      return $True
      Write-Log -Level 'SLACK' -Message "*Nightly Restore:* Successfully Downloaded [ *{0}* ], Time Taken: *{1}* Minutes"`
         -Arguments $FileToDownload.FileName, $([Math]::Round($Script:DownloadTime.TotalMinutes, 2))
      return $True
    }
    elseif ($DownloadTimeout)
    {
      Write-Log -Level 'SLACK' -Message "*Nightly Restore:* Download timeout after two attempts of [ {0} ]. ``ERROR:`` *{1}*"`
        -Arguments $FileToDownload.FileName, $DownloadError[0].Exception.Message
      return $False
    }
    else
    {
      Write-Log -Level 'SLACK' -Message "*Nightly Restore:* There was an error downloading [ {0} ]. ``ERROR:`` *{1}*"`
        -Arguments $FileToDownload.FileName, $DownloadError[0].Exception.Message
      return $False
    }
  }
}
blueww commented 4 years ago

Will follow up by mail, and update the issue later.

blueww commented 4 years ago

Discussed with @paulmrozowski in mail:

  1. Paul don't have time to repro it, and switch to AzCopy instead
  2. Powershell team can't repro the issue

So we will close the issue now.

ghost commented 4 years ago

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.