olahallengren / sql-server-maintenance-solution

SQL Server Maintenance Solution
https://ola.hallengren.com
MIT License
2.92k stars 757 forks source link

Data Domain - Backup Promotion #256

Open TreyHawes opened 5 years ago

TreyHawes commented 5 years ago

Data Domain has a feature that will change a log backup to a full backup when it doesn't find a full backup, when this happens, a message like this will show in the results - _No full backup of database wss_teamsHR1 was found. Promoting the backup level to full database backup.

I have found this functionality isn't very good because not only does it promote to a full backup when there is a backup, the "log" backup will get thrown off on its timing. EMC recommends using one stream when log backup and n-streams when doing a full backup. So if the log backup get promoted full, the full backup gets dropped down to one stream and it will take much longer to backup. Also, I have other alerts that tell me if a database hasn't been backed up, so I don't want Data Domain to decide on its own with to do a log or full backup.

The switch to turn off this feature is (-a "BACKUP_PROMOTION=NONE") . I would like to suggest this switch be added to the stored procedure and/or as one of your parameters.

olahallengren commented 5 years ago

Data Domain has a feature that will change a log backup to a full backup when it doesn't find a full backup, when this happens, a message like this will show in the results - No full backup of database wss_teams_HR1 was found. Promoting the backup level to full database backup.

@TreyHawes, do you know why this is getting triggered?

Do you have an output - file when this has happened?

TreyHawes commented 5 years ago

I will send you part of the DD log in an email as I don't want it on a "public" forum. Unfortunately, it doesn't give more information and I don't know how DD determines if there is a full backup or not. If you have more questions, please let me know.

SQL-Matt commented 5 years ago

We have recently implemented Dell EMC Data Domain Boost and have had similar issues where, in the case of a log backup being done, it gets escalated to a full backup with 1 stream.

This was happening on a clustered environment and DDB (at our site) would record the backup 'directory' as using the cluster name rather than the instance name (of the node). When this occurred, the log backup was the first to happen after failover of the AG.

As far as I can figure - and we have sent the question to Dell to answer - is that the DDBoost check looks for a full backup to have been taken from that node in that cluster, even though I had the understanding that the backup of blocks would be used by all nodes of the cluster (but this appears to not be the case).

My question to you ( @TreyHawes ) in this is: Are you operating the DDBoost in a clustered environment and was this the first backup into DDBoost taken on that node?

I only ask this as it this appears to be more a failing of DDBoost rather than the Hallengren job - although the option to prevent the escalation would be handy.

TreyHawes commented 5 years ago

Hi @SQL-Matt, We experienced the same thing when we started implementing DD Boost over a year ago and since there is a switch to bypass it, we didn't contact EMC about it. In hind-sight, perhaps we should have asked about the logic that determines what promotes it or not. All of our log backup jobs have this switch since we have other processes that will tell us if a full backup failed.

I wholeheartedly agree with you, the issue is with DDBoost and not Ola scripts. His script simply creates/executes the DDBoost command line. I have hard coded the switch into the backup stored procedure on my servers, I just thought other folks might like the option, thus the suggestion.

The environment where this came up is a Sharepoint AoG. The "primary" has been on that node for several weeks and once I realized the log backup was being promoted, I made sure there was a full backup and there was. It was only happening on 1 out of 51 databases on the server so there was something happen at the database level not the server level where DDBoost thinks there isn't a full backup.

We had issues on Window Clustered environments when it was first implemented and noticed the same cluster/instance issue, they are working good now. AoG's have been running pretty smooth as well, however, we did need to add a "failover" job that would kick off a full backup when the AoG failed over.

SQL-Matt commented 5 years ago

We're about to embark on a new problem with regards DDBoost.... Another part of the business has decided to create another 2 nodes in the cloud that will be asynchronous to the nodes on the premises... in SQL 2012 (we're stuck at this version thanks to a vendor who refuses to allow the upgrade). When on-premises, the Hallengren jobs are writing to DDBoost and, when in the cloud, they'll be writing to normal backups. Can anyone else see the problem here?

xkxhkmJNux79bdHyagiG commented 1 year ago

Just wanted to say "thank you" for this old thread saving our bacon tonight. First time running into DDBoost backup promotion, and found this thread containing the switch we needed in order to disable it and force a log backup. In fairness, we're also running an old version of dbo.DatabaseBackup, before this fix, so that's partly on us having to scramble.

Thanks, Randy