lordmilko / PrtgSensors

Miscellaneous sensors for PRTG Network Monitor
MIT License
9 stars 3 forks source link

An exception occurred while trying to process backups: Access to the path 'Veeam\Backup' is denied. #3

Open ghost opened 2 years ago

ghost commented 2 years ago

Hi,

I read the guide, installed the B&R console and put the Get-VeeamBackupStatus.ps1 script and corresponding lookup into place. A Windows user who has the Veeam Backup Administrator role is saved in the device which I added the sensor to and the sensor is set to "use Windows credentials of upstream device". Now when the sensor does a query, I get the following result:

DEBUG: Entered Main
DEBUG: Entered [VeeamBackupStatus]::GetStatus
DEBUG: Entered [VeeamBackupStatus]::GetStatusInternal
DEBUG: -Loading Veeam PSSnapIn
DEBUG: -Connecting to Veeam Backup Server
<Prtg>
    <Error>1</Error>
    <Text>An exception occurred while trying to process backups: Access to the path 'Veeam\Backup' is denied.</Text>
</Prtg>

If I run a powershell on the probe system as the above-mentioned user, and execute the script with the same parameters as entered in the PRTG sensor, it returns the backup data as intended. Also, in the same PS session, Connect-VBRServer works fine.

The path in the error message ('Veeam\Backup') looks like a substring of the path to the Veeam PS SnapIn ("C:\Program Files**Veeam\Backup** and Replication\Console\Veeam.Backup.PowerShell\Veeam.Backup.PowerShell.psd1"), although, according to the DEBUG entry above, the SnapIn was already loaded when the error is returned.

I am at a loss here. Any suggestions? Thanks in advance.

thwgha

lordmilko commented 2 years ago

After generating the error can you please provide the output of

$error[0].Exception.StackTrace

In the code, we can see the following

hidden [void]GetStatusInternal()
{
    [Logger]::Log("Entered [VeeamBackupStatus]::GetStatusInternal")

    $this.Init()

    $backup = $this.GetBackupsToProcess()

    ...
}

hidden [void]Init()
{
    [Logger]::Log("-Loading Veeam PSSnapIn")
    #Add-PSSnapin -Name VeeamPSSnapIn -ErrorAction Stop
    ipmo "C:\Program Files\Veeam\Backup and Replication\Console\Veeam.Backup.PowerShell\Veeam.Backup.PowerShell.psd1" -DisableNameChecking

    [Logger]::Log("-Connecting to Veeam Backup Server")
    Disconnect-VBRServer
    Connect-VBRServer -Server $this.server -Timeout 20
}

hidden [object]GetBackupsToProcess()
{
    [Logger]::Log("Entered [VeeamBackupStatus]::GetBackupsToProcess")

    return CBackupJob::Get($this.jobName)
}
  1. In GetStatusInternal we call Init, which prints -Loading Veeam PSSnapIn and loads the Veeam PowerShell Module (formerly a PSSnapIn)
  2. Init prints -Connecting to Veeam Backup Server
  3. Init disconnects from any existing servers, and then connects to the specified server
  4. GetStatusInternal calls GetBackupsToProcess, which prints Entered [VeeamBackupStatus]::GetBackupsToProcess

Based on the fact a status message is printed in steps 2 and 4, the only stage at which an error could occur would appear to be step 3. Maybe check that the server you're trying to connect to is correct; the first argument is the hostname of your VBR server e.g. veeam-1. You can also try and run the Connect-VBRServer -Server <yourServer> command manually to see if that reproduces the issue

ghost commented 2 years ago

Hi, thanks for your switft feedback.

You wrote: "After generating the error can you please provide the output of $error[0].Exception.StackTrace" - how can I accomplish that when the skript is run by PRTG on the probe server? As I wrote, when I run the script manally in the context of the intended user, it works fine.

I came to the same conclusion, that the error must be caused by the two lines after [Logger]::Log("-Connecting to Veeam Backup Server"). Since, according to the error text, it seems to be some sort of permissions problem, I suspected the script is not run in the intended users context. Therefore I added [Logger]::Log("Running as User: $($env:USERDOMAIN)\$($env:USERNAME)") right after [Logger]::Log("Entered Main"). The output is DEBUG: Running as User: DOMAIN\PRTGPROBESERVERNAME$. So it is not using the Windows credentials I supplied in the upstream device, although it is configured to do so. Any ideas on that?

Some googling braught up this entry for another Veeam PRTG sensor script, where people seem to have the same issues: https://github.com/vMarkusK/Advanced-PRTG-Sensors/issues/80 There, the issue could be solved by putting another skript in PRTG which just calls the Get-VeeamBackupStatus.ps1, called Run-Get-VeeamBackupStatus.ps1:

if ($args.Length -gt 0) { 
    Invoke-Command -ComputerName PRTGPROBESERVERNAME.domain.com -FilePath "C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML\Get-VeeamBackupStatus.ps1" -ArgumentList @args
} else { 
    echo "no parameters" 
}
echo "Username: $($env:userdomain)\$($env:username)" | out-file C:\users\public\documents\log.txt

When using that in PRTG, the error changes to Invalid JSON. In C:\users\public\documents\log.txt it says Username: DOMAIN\PRTGPROBESERVERNAME$. The Result of Sensor 3431.Data.txt contains the following:

Data['linuxloginpassword'].asString := '';
Data['notonpod'].asString := '0';
Data['fastcount'].asString := '0';
Data['lastmsg'].asString := '#Y2 @#O233 @#O231[Invalid JSON.]';
Data['resultfile'].asString := 'Result of Sensor 3431.txt';
Data['windowsloginusername'].asString := 'veeam.backupadminuser';
Data['hostv6'].asString := '';
Data['exefile'].asString := 'Run-Get-VeeamBackupStatus.ps1';
Data['lastuptime'].asString := '0';
Data['writeresult'].asString := '2';
Data['blockedsens'].asString := '';
Data['reqmsginterval'].asString := '60';
Data['windowslogindomain'].asString := 'domain';
Data['tlsexplicit_imap'].asString := '';
Data['channelnames'].asString := '';
Data['tlsexplicit_default'].asString := '';
Data['canlinux'].asString := '0';
Data['isexesensor'].asString := '1';
Data['windowsloginpassword'].asString := '***';
Data['environment'].asString := '';
Data['mutexname'].asString := '';
Data['channelinfos'].asString := '{}';
Data['uptimecount'].asString := '0';
Data['vmwareparams'].asString := '1';
Data['reboot'].asString := '44755.4792620833';
Data['usednstime'].asString := '0';
Data['newpowershell'].asString := '1';
Data['linuxlogindomain'].asString := '';
Data['tlsexplicit_port'].asString := '';
Data['monitorchange'].asString := '';
Data['inerror'].asString := '1';
Data['sensorid'].asString := '3431';
Data['ipversion'].asString := '0';
Data['tlsexplicit_smtp'].asString := '';
Data['host'].asString := 'veeambackupserver.domain.com';
Data['usewindowsauthentication'].asString := '1';
Data['simulate'].asString := '0';
Data['tlsexplicit_ftp'].asString := '';
Data['timeout'].asString := '300';
Data['exeparams'].asString := 'veeambackupserver.domain.com BackupJobName';
Data['momopersistent'].asString := '';
Data['tlsexplicit_pop3'].asString := '';

We do not need PSX64.exe with this script, correct? I tried both just hostname and FQDN of the Veeam server, followed by the backup job to check. As I mentioned, Connect-VBRserver works fine, too.

lordmilko commented 2 years ago

If it says your username is the name of your probe server, this indicates it is not in fact running using Windows Authentication, it is running as SYSTEM - which is the account the PRTG Probe Service runs as.

I can see that your debug output says Data['usewindowsauthentication'].asString := '1'; in it which contradicts the above statement

The PRTG Probe Service is a 32-bit process, however the Veeam Backup & Replication PowerShell module is 64-bit only. As such it must be run within a 64-bit process. 32-bit processes will load PowerShell from SYSWOW64. I wrote this script before I wrote PSx64, so this script includes its own built-in logic for self elevating to a 64-bit process

I would suggest running Process Monitor and force a refresh of the script in the PRTG UI. Add the Command Line column, and you'll be able to see exactly what commands are being executed, whether it's self elevating, and you'll also be able to inspect whether any processes are running as SYSTEM or not

ghost commented 2 years ago

Thanks again for your quick reply! What I see after digging through 600k+ lines after recording just for a few seconds, is that the script file is opened by PRTG Probe.exe and then PowerShellScriptRunner.exe is started, Impersonating: DOMAIN\veeam.backupadminuser. Nevertheless, the username logged by the script itself is DOMAIN\PRTGPROBESERVERNAME$. It is all contradicting.

Since after some hours of troubleshooting I still don't know what to do from here, I think I will try my luck with a different script. Thanks for your support!