microsoft / navcontainerhelper

Official Microsoft repository for BcContainerHelper, a PowerShell module, which makes it easier to work with Business Central Containers on Docker.
MIT License
385 stars 247 forks source link

Export-BcContainerDatabasesAsBacpac cannot restore .ndf files to tempdb #2544

Closed michvllni closed 2 years ago

michvllni commented 2 years ago

PLEASE DO NOT INCLUDE ANY PASSWORDS OR TOKENS IN YOUR ISSUE!!!

Describe the issue When trying to export a bacpac from a database using Export-BcContainerDatabasesAsBacpac on a database that has an .ndf file it fails to correctly restore the database as temp database after backing it up.

Please not that my SQL server lies in a separate container that is connected to the BC container via docker compose:

Scripts used to create container and cause the issue

version: "3.3"
services:
  bc:
    volumes:
    - C:\data:c:\data
    - C:\ProgramData\BcContainerHelper:C:\BcContainerHelper
    container_name: bc
    mem_limit: 25G
    image: a prebuilt image from new-bcimage with nodatabase
    environment:
      - Accept_eula=Y
      - UseSSL=N
      - ACCEPT_OUTDATED=Y
      - REMOVEPASSWORDKEYFILE=N
      - auth=NavUserPassword
      - PasswordKeyFile=C:\Data\nav.key
      - databaseServer=sql,1433
      - databaseName=mydatabase
      - databaseUsername=sa
      - databaseSecurePassword=encryptedpassword
    depends_on:
      - sql

  sql:
    container_name: seminar-sql
    image: a pre built sql server developer image
    volumes:
      - C:\data:c:\data
    environment:
      - ACCEPT_EULA=Y
      #important: give access to bakpath to NETWORK SERVICE on Host
      - bakPath=C:\data\database.bak
      - SA_PASSWORD=the sql server password

Output:

docker-compose up --force-recreate -d;docker-compose logs -f
[+] Running 2/2
 - Container sql     Started                                                                              10.9s
 - Container bc  Started                                                                               6.0s
bc  | Configuring host.docker.internal
sql     | Configuring host.docker.internal
sql     | Starting SQL Server
sql     | Initializing
sql     | Changing SA login credentials
sql     | Restoring C:\data\database.bak
bc  | Configuring host.docker.internal
bc  | Initializing...
bc  | Starting Container
bc  | Hostname is b7c0ad73c11e
bc  | PublicDnsName is b7c0ad73c11e
bc  | Using NavUserPassword Authentication
bc  | Starting Internet Information Server
bc  | Import Encryption Key
sql     | Processed 269560 pages for database 'mydatabase', file 'newsystem_Data' on file 1.
sql     | Processed 549816 pages for database 'mydatabase', file 'newsystem_1_Data' on file 1.
sql     | Processed 1 pages for database 'mydatabase', file 'newsystem_Log' on file 1.
sql     | RESTORE DATABASE successfully processed 819377 pages in 84.263 seconds (75.969 MB/sec).
sql     | Started SQL Server.
sql     | Ready for connections!
sql     |
bc  | Creating Self Signed Certificate
bc  | Self Signed Certificate Thumbprint 1C34B75BC29EF3DB3AFDFBC48146C35427ADF99F
bc  | DNS identity b7c0ad73c11e
bc  | Modifying Service Tier Config File with Instance Specific Settings
bc  | Modifying Service Tier Config File with settings from environment variable
bc  | Setting PublicWebBaseUrl to http://localhost:1537/BC
bc  | Setting SqlLongRunningThreshold to 10000
bc  | Setting AllowSessionWhileSyncAndDataUpgrade to True
bc  | Setting EnableTaskScheduler to True
bc  | Setting ClientServicesReconnectPeriod to 01:00:00
bc  | Starting Service Tier
bc  | Registering event sources
bc  | Creating DotNetCore Web Server Instance
bc  | Creating http download site
bc  | Container IP Address: 172.31.166.23
bc  | Container Hostname  : b7c0ad73c11e
bc  | Container Dns Name  : b7c0ad73c11e
bc  | Web Client          : http://b7c0ad73c11e/BC/
bc  | Dev. Server         : http://b7c0ad73c11e
bc  | Dev. ServerInstance : BC
bc  |
bc  | Files:
bc  | http://b7c0ad73c11e:8080/ALLanguage.vsix
bc  |
bc  | WARNING: You are running a container which is 131 days old.
bc  | Microsoft recommends that you always run the latest version of our containers.
bc  |
bc  | Container Total Physical Memory is 64.0Gb
bc  | Container Free Physical Memory is 53.6Gb
bc  |
bc  | Initialization took 195 seconds
bc  | Ready for connections!
bc  | Starting EventLog Monitor
bc  | Monitoring EventSources from EventLog[Application]:
bc  | - MicrosoftDynamicsNAVClientClientService
bc  | - MicrosoftDynamicsNAVClientWebClient
bc  | - MicrosoftDynamicsNavServer$BC
bc  | - MSSQL$SQLEXPRESS
bc  |

Everything starts up fine, but when executing Export-BcContainerDatabasesAsBacpac -containerName seminar-master -sqlCredential $cred -bacpacFolder 'C:\data\' -diagnostics

I get the following output ($cred contains the login credentials for the sql server as defined in the compose): Full output of scripts

Downloading Dac Framework 19.0
Installing Dac Framework 19.0
Copying Database on sql,1433\SQLEXPRESS from mydatabase to tempmydatabase
EngineEdition 3
backup database [mydatabase] to disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with init, stats=10;
restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Warning, exception when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Waiting... - retrying
Warning, exception when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Waiting...... - retrying
Warning, exception when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Waiting......... - retrying
Warning, exception when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Waiting............ - retrying
Warning, exception when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Waiting............... - retrying
Warning, exception when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Waiting.................. - retrying
Warning, exception when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Waiting..................... - retrying
Warning, exception when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Waiting........................ - retrying
Warning, exception when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Waiting........................... - retrying
Error when running: restore database [tempmydatabase] from disk = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.bak' with stats=10, recovery, move 'newsystem_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.mdf', move 'newsystem_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\tempmydatabase.ldf', move 'newsystem_1_Data.ndf'
Incorrect syntax near 'newsystem_1_Data.ndf'.

Exception Script Stack Trace:
at Invoke-SqlCmdWithRetry, C:\Run\HelperFunctions.ps1: line 434
at Copy-NavDatabase, C:\Run\HelperFunctions.ps1: line 250
at <ScriptBlock>, <No file>: line 360

PowerShell Call Stack:
at Invoke-ScriptInBcContainer, C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\3.0.10\ContainerHandling\Invoke-ScriptInNavContainer.ps1: line 58
at Export-BcContainerDatabasesAsBacpac, C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\3.0.10\Bacpac\Export-NavContainerDatabasesAsBacpac.ps1: line 67
at <ScriptBlock>, <No file>: line 1

Container Free Physical Memory: 48.2Gb

Services in container bc:
- MicrosoftDynamicsNavServer$BC is Running
- MSSQL$SQLEXPRESS is Stopped

Export-BcContainerDatabasesAsBacpac Telemetry Correlation Id: 2cebc57e-140a-453f-a8c3-1bbb4ca90aea
Incorrect syntax near 'newsystem_1_Data.ndf'.
At C:\Program
Files\WindowsPowerShell\Modules\BcContainerHelper\3.0.10\ContainerHandling\Invoke-ScriptInNavContainer.ps1:99 char:13
+             throw $errorMessage
+             ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Incorrect synta...em_1_Data.ndf'.:String) [], RuntimeException
    + FullyQualifiedErrorId : Incorrect syntax near 'newsystem_1_Data.ndf'.

It appears that the ndf file fails to get a file path to which it can be restored and thus the SQL command is incomplete (line 6). Additionally, the script appears to add an instance SQLEXPRESS because the sql container does not use an instance (line 3).

The files in the database look like this:

sqlcmd -Q "RESTORE FILELISTONLY FROM 'C:\data\database.bak'"
LogicalName PhysicalName    Type    FileGroupName    Size MaxSize    FileId  CreateLSN    DropLSN UniqueId    ReadOnlyLSN ReadWriteLSN    BackupSizeInBytes    SourceBlockSize    FileGroupId    LogGroupGUID    DifferentialBaseLSN    DifferentialBaseGUID   IsReadOnly    IsPresent    TDEThumbprint    SnapshotUrl
-------------------------------------------------------------------------------------------------------------------------------- -----------------------------------------------------------------------------------------------------------------------------------------
newsystem_Data    D:\Data\database.mdf    D    PRIMARY  2731278336       35184372080640    1    0    0 9DE67251-C09E-4BC2-A9A0-0C792501A6EB    0    0    2207907840    512      1 NULL    7206000036878800077    AA4E83E3-042D-4B95-8011-8A50D4AA1CD5    0    1    NULL    NULL
newsystem_1_Data    D:\Data\database.ndf    D    Data Filegroup 1  5019992064       35184372080640    3  21000000050300002    0    4AB10FDF-674A-4443-BFAF-C37FACCF5AF5    0    0    4503896064  512    2 NULL    7206000036878800077 AA4E83E3-042D-4B95-8011-8A50D4AA1CD5    0    1    NULL    NULL
newsystem_Log    D:\Data\database.ldf    L    NULL    39133184    2199023255552    2    0    0 E4C5600B-82BB-4B29-A789-1B1AE3DC1218  0    0     0   512    0    NULL    0    00000000-0000-0000-0000-000000000000    0    1    NULL     NULL

(3 rows affected)
freddydk commented 2 years ago

You should set an environment variable databaseInstance to an empty string. On the .ndf - I assume that the SQL image is generating this - isn't there a way to just use ldf and mdf ?

michvllni commented 2 years ago

Setting the databaseInstance to an empty string resolved the issue with the instance.

Unfortunately all our databases are structured like that (I don't know why). Meaning all databases we use consist of an mdf, an ndf and a ldf file

michvllni commented 2 years ago

I've been using something similar as you use it here when restoring the backup in the sql container...maybe it could help you resolve the issue?

$sqlversion = 15
$bakPathCleaned = $env:bakPath.TrimStart('\\').TrimEnd('\\')
if (Test-Path "$bakPathCleaned") {
    Write-Host "Restoring $bakPathCleaned"
    $sqlcmd = "RESTORE FILELISTONLY FROM DISK = '$bakPathCleaned'" 
    $files = sqlcmd -Q $sqlcmd -s "," -W
    $files = $files[2..($files.length - 3)] #remove header and footer
    $importcmd = "RESTORE DATABASE mydatabase FROM DISK = '$bakPathCleaned'"
    if ($files.Count -gt 0) {
        $importcmd += "WITH "
        foreach ($file in $files) {
            $fileRow = $file -split ","
            $logicalName = $fileRow[0]
            $physicalName = Join-Path "C:\Program Files\Microsoft SQL Server\MSSQL$sqlVersion.MSSQLSERVER\MSSQL\DATA" (Split-Path $fileRow[1] -Leaf)
            if ($importcmd -like "*MOVE*TO*") {
                $importcmd += ","
            }
            $importcmd += "MOVE '$logicalName' TO '$physicalName'"
        }
    }
    sqlcmd -Q "$importcmd"
    if (!$?) {
        exit 1
    }
}
else {
    Write-Host "Could not find $bakPathCleaned"
    Exit 1
}

Edit: Using Invoke-SQLCmd would make things a lot easier, but I'm not in the mood of touching the images at the moment

freddydk commented 2 years ago

The Export function was created to allow container databases to be exported as bacpac and container databases do not have this problem and I haven't heard this problem from other partners.

it isn't something I will have the time to look at before my summer vacation - meaning that at the earliest, it will be august or september.

Feel free to submit a PR on this https://github.com/microsoft/navcontainerhelper/blob/master/Bacpac/Export-NavContainerDatabasesAsBacpac.ps1 if you cannot wait

freddydk commented 2 years ago

Maybe the .ndf file can just be ignored

michvllni commented 2 years ago

Thank you for the hint. It appears it is not an issue with the export function but rather with the copy-navdatabase function in nav-docker. I will take a closer look at it tomorrow and send a pull request your way if that's okay for you

freddydk commented 2 years ago

nav-docker is the generic image. To test, you can place a new helperfunctions in the c:\run\my folder of the container when you create the container. Then that will override the existing.

michvllni commented 2 years ago

Fixing the helper function in nav-docker did the trick for me. I've created a PR there.