search

vMarkusK / Advanced-PRTG-Sensors

Collection of my Advanced PRTG Sensors
https://mycloudrevolution.com/
Apache License 2.0
69 stars 29 forks source link

XML/JSON issue. (Code: PE233) (Code: PE231) #81

Closed FynnKotlorz closed 11 months ago

FynnKotlorz commented 3 years ago

Good evening,

i just updated my veeam server to version 11. after i updated the console aswell and checked all steps needed and updated the script with your new one i have the following issue:

PRTG shows up this failure:

(sorry that it is in german) XML: Das zurückgelieferte XML entspricht nicht dem erwarteten Schema. (Code: PE233) -- JSON: Das zurückgelieferte JSON entspricht nicht der erwarteten Struktur (Invalid JSON.). (Code: PE231)

and of course a do not get any data.

i just reviewed the logfiles in which i found the following message:

Disconnect-VBRServer : Der Zugriff auf den Pfad "Veeam\Backup" wurde verweigert. In C:\Program Files (x86)\PRTG Network Monitor\custom sensors\EXEXML\PRTG-VeeamBRStats.ps1:66 Zeichen:5

i don't really know how to fix this issue.

Greetings from germany,

Fynn

vMarkusK commented 3 years ago

Duplicate of #77

vMarkusK commented 3 years ago

Duplicate of #80

vMarkusK commented 3 years ago

As you can see in the linked issues this seems to be caused by an changed security behavior of Veeam v11.

Have you configured the PRTG sensor to use the Security context of the Probe service? If so, is the Security context of the Probe service "Local System" or a named user?

Some additional details about your environment:

vMarkusK commented 3 years ago

Hello, I just want to update you about the current state of the analysis.

I have tested a few Concepts to reproduce this issue:

As I was not able to reproduce this issue I need more details details to dig deeper:

FynnKotlorz commented 3 years ago

Hey,

thanks for your reply and great help.

Some informations i should gave you earlier, my bad im sorry!

The PRTG Probe is not installed on the same server as Veeam.

Because the Veeam Server is not a Domain Member (and we can not change it in the future because of some safety rules we have to follow), i created a service account user localy on both servers with the same name and password (in the past it worked like that and we‘re still using it on another older version).

username is constructed like this:

sa--srv-vbs-01

for example:

sa-abc-srv-01

This is a normal Windows User. (Tested it with Admin rights already) On the VBR Server this user has viewer rights in Veeam.

Both logins worked in the past.

OS on both systems is Windows Server (i dont know the specific version, i can tell you tomorrow! :>)

i hope these informations will help you :o

regards,

Fynn

Markus Kraus @.***> schrieb am Di. 30. März 2021 um 23:35:

Hello, I just want to update you about the current state of the analysis.

I have tested a few Concepts to reproduce this issue:

  • Probe on VBR Server (Service as "local System" and "Named User")
  • Probe on dedicated Server (Service as "Named User")

As I was not able to reproduce this issue I need more details details to dig deeper:

  • OS of VBR Server
  • OS of PRTG Probe
  • Domain Members?
  • Is the Probe installed on VBR or dedicated Server?
  • Is probe Server Started as Named user?
  • Local permission of the Probe Service user on the Probe?
  • Local permission of the Probe Service user on the VBR Server?
  • VBR Permissions of the Probe Service User?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/vMarkusK/Advanced-PRTG-Sensors/issues/81#issuecomment-810593129, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATO2ASYLODDGYBVKJ4TOHMLTGI7ZZANCNFSM42CFBM7Q .

RonSharcott commented 2 years ago

Spent a lot of time trying to get this to work. Running the ps1 script from the probe server works. Capturing that output and turning it into a ps1 script using a lot of "Write-Output" statements works. Calling it with or without the PSx64.exe fails.

I get the feeling that it works when it is hard-coded XML but the sensor is interpreting the output as JSON only when it takes it from the script directly even though the output is in XML.

This is a fantastic script and I hope this issue get's worked out. The only work around I am considering is to have your script write to a file then have another script read and put that result out. I tried somethign similar but might just have to schedule your script then have another act as a filter for the output. Pretty ugly and really don't want to go down that path.

vMarkusK commented 2 years ago

Hi @RonSharcott ,

let's get things sorted:

Manually running Script on Probe Works fine, right?

Calling it with or without the PSx64.exe fails.

Have you enabled the Logs for the Custom Sensor (please with PSx64.exe)?

Might you please run the script run with -Debug switch and share the output?

I don't think that your Workaround is required. In the case the script execution works manually fine, we will figure out what's going wrong.

RonSharcott commented 2 years ago

Manually running Script on Probe Works fine, right? Correct. Load the script into the ISE then execute. I get an XML output which I have taken an edited into a series of Write-Output statements then called that script instead. That works fine so the output looks correct.

Do you use any parameters to execute the script (-BRHost etc.)? Calling it with or without the PSx64.exe fails. I have used the -f (script path - both relational and full path) and -p="BRHost - (Veeam host - both by hostname and FQDN)

Calling from PRTG as Custom Sensor, right? Have you enabled the Logs for the Custom Sensor (please with PSx64.exe)? I have turned on the logs but all I get is the data file and a blank txt file. (blank actually just has spaces on one line)

https://www.paessler.com/manuals/prtg/exe_script_sensor - Result Handling Please share the output in the Log-File. Pasted after all questions

Might you please run the script run with -Debug switch and share the output? I assume -Debug is a switch on the file? maybe in the -p string? Trying to find where it writes to.

Result of Sensor 22499.Data.txt Edited for security

Data['linuxloginpassword'].asString := ''; Data['notonpod'].asString := '0'; Data['fastcount'].asString := '0'; Data['lastmsg'].asString := '#Y2 @#O233 @#O231[Invalid JSON.]'; Data['resultfile'].asString := 'Result of Sensor 22491.txt'; Data['windowsloginusername'].asString := 'PRTGAdmin'; Data['hostv6'].asString := ''; Data['exefile'].asString := 'PSx64.exe'; Data['lastuptime'].asString := '0'; Data['writeresult'].asString := '2'; Data['blockedsens'].asString := ''; Data['reqmsginterval'].asString := '300'; Data['windowslogindomain'].asString := ''; Data['tlsexplicit_imap'].asString := ''; Data['channelnames'].asString := ''; Data['tlsexplicit_default'].asString := ''; Data['canlinux'].asString := '0'; Data['isexesensor'].asString := '1'; Data['windowsloginpassword'].asString := ''; Data['environment'].asString := ''; Data['mutexname'].asString := ''; Data['channelinfos'].asString := '{}'; Data['uptimecount'].asString := '0'; Data['vmwareparams'].asString := '1'; Data['reboot'].asString := '44593.7838533681'; Data['usednstime'].asString := '0'; Data['newpowershell'].asString := '0'; Data['linuxlogindomain'].asString := ''; Data['tlsexplicit_port'].asString := ''; Data['monitorchange'].asString := ''; Data['inerror'].asString := '1'; Data['sensorid'].asString := '22491'; Data['ipversion'].asString := '0'; Data['tlsexplicit_smtp'].asString := ''; Data['host'].asString := ''; Data['exeparamslist'].asString := ''; Data['usewindowsauthentication'].asString := '0'; Data['simulate'].asString := '0'; Data['tlsexplicit_ftp'].asString := ''; Data['timeout'].asString := '60'; Data['exeparams'].asString := '-f=".\PRTG-VeeamBRStats.ps1" -p="BRHost -debug"'; Data['momopersistent'].asString := ''; Data['tlsexplicit_pop3'].asString := '';

RonSharcott commented 2 years ago

Manual run with -debug parameter Edited for security

PS C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML> .\PRTG-VeeamBRStats.ps1 -debug DEBUG: Starting to Process Connection to 'veeamlan1' with user '' ... DEBUG: Connecting BRHost '' with user ''...

Successful-Backups 14 1 1 Warning-Backups 0 1 1 0 1 Failes-Backups 0 1 1 0 1 Failed-Backups 0 1 1 0 1 Running-Backups 0 1 1 TotalBackupRead 85 Custom GB 1 1 TotalBackupTransfer 59 Custom GB 1 1 Successful-BackupCopys 0 1 1 Warning-BackupCopys 0 1 1 0 1 Failes-BackupCopys 0 1 1 0 1 Failed-BackupCopys 0 1 1 0 1 Running-BackupCopys 0 1 1 Idle-BackupCopys 4 1 1 Successful-Replications 2 1 1 Warning-Replications 0 1 1 0 1 Failes-Replications 0 1 1 0 1 Failed-Replications 0 1 1 0 1 Running-Replications 0 1 1 Successful-Endpoints 0 1 1 Warning-Endpoints 0 1 1 0 1 Failes-Endpoints 0 1 1 0 1 Running-Endpoints 0 1 1 DEBUG: Veeam.Backup.Model.CBackupRepositoryInfo DEBUG: Veeam.Backup.Model.CBackupRepositoryInfo DEBUG: Veeam.Backup.Model.CBackupRepositoryInfo DEBUG: Veeam.Backup.Model.CBackupRepositoryInfo DEBUG: Veeam.Backup.Model.CBackupRepositoryInfo DEBUG: Veeam.Backup.Model.CBackupRepositoryInfo DEBUG: Veeam.Backup.Model.CBackupRepositoryInfo REPO - Local Veeam server Backup Repository 21 Percent 1 1 20 10 1 REPO - BCNET Data Domain 48 Percent 1 1 20 10 1 REPO - veeambackuprespository storageint 41 Percent 1 1 20 10 1 REPO - veeambackuprepository1 storageint 95 Percent 1 1 20 10 1 REPO - veeambackuprepository1 nas2 24 Percent 1 1 20 10 1 REPO - Backup Repository lansdowne EMC 28 Percent 1 1 20 10 1 REPO - veeambackuprepository nas2 46 Percent 1 1 20 10 1

Repository Name Free (GB) Total (GB) Free (%) Status

Local Veeam server Backup Repository 4.18 20.00 21 OK
BCNET Data Domain 4911.09 10240 48 OK
veeambackuprespository storageint 10196.62 24576 41 OK
veeambackuprepository1 storageint 9720.32 10240 95 OK
veeambackuprepository1 nas2 1218.22 5120 24 OK
Backup Repository lansdowne EMC 4526.28 16126.12 28 OK
veeambackuprepository nas2 5615.21 12288 46 OK

Successful Backups : 14 Warning Backups : 0 Failes Backups : 0 Failed Backups : 0 Running Backups : 0 Total Backup Transfer : 59 Total Backup Read : 85 Warning BackupCopys : 0 Failes BackupCopys : 0 Failed BackupCopys : 0 Running BackupCopys : 0 Idle BackupCopys : 4 Successful Replications : 2 Warning Replications : 0 Failes Replications : 0 Failed Replications : 0 Running Replications : 0 Seccessful Endpoints : 0 Warning Endpoints : 0 Failes Endpoints : 0 Running Endpoints : 0

vMarkusK commented 2 years ago

Thanks. Please try the following steps:

RonSharcott commented 2 years ago

It is starting to look like a permissions issue. Trying to find the correct account to use. I've tried changing the Windows credentials on the device and setting up the Veeam credentials. Not quite getting the correct balance.

Thanks to that -debug flag for pointing out the credential issue.

PS C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML> .\PRTG-VeeamBRStats.ps1 -p="veeamlan1" -debug DEBUG: Starting to Process Connection to '-p=veeamlan1' with user '' ... DEBUG: Connecting BRHost '-p=veeamlan1' with user ''... C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML\PRTG-VeeamBRStats.ps1 : Failed to connect to Veeam BR Host '-p=veeamlan1' with user '' At line:1 char:1

C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML\PRTG-VeeamBRStats.ps1 : at , C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML\PRTG-VeeamBRStats.ps1: line 311 at , : line 1 At line:1 char:1

1 Failed to connect to Veeam BR Host '-p=veeamlan1' with user ''
vMarkusK commented 2 years ago

Please change the default BRHOST in the script and do not use the parameter to pass the name. There seems to be a syntax error in your call.

RonSharcott commented 2 years ago

Done. That failure may have come from repeated failures locking me out.

It's odd. I can run the script, even in -debug mode, and get XML back. But the sensor complains it's not in JSON format. I would say something is tripping the sensor to think the returned data is supposed to be in JSON format.

From what I can tell, from the the original description on setup, there are 2 accounts at play. The Windows Credentials and the Veeam Credentials. I can use my own System Account to remote and to get a debug or XML return. It still won't work with the sensor.

vMarkusK commented 2 years ago

Is the probe service started as a valid windows user which has access to veeam (please verify this via vbr console connection)?

In most cases, the XML error returned by the PRTG Sensor is based on an unknown error during script execution. This seems the case because also the log file is empty in your case.

RonSharcott commented 2 years ago

Although I have tried this before I just now:

RonSharcott commented 2 years ago

Double checked the service and it was running as Local System Changed it to run as the PRTG account and the error on the sensor changed to: A required privilege is not held by the client. (0x522)

vMarkusK commented 2 years ago

Is the PRTG Core server the only Probe in your environment? Service is restarted after the change?

I am not sure what's the reason for this message (maybe the local security policy for this account).

A reboot of the PRTG Core service might also help.

An other workaround is installing a probe service on the veeam Server itself and move the sensor to this probe.

RonSharcott commented 2 years ago

I have a few remote probes. Generally I would expect the same result from all of them. I originally moved this device off a remote probe and onto the core probe to try and get better logging.

For the service, I entered the credentials and restarted the service.

I have a few hoops to jump through before considering turning off UAC on the Veeam server. From other threads here that has been suggested.

RonSharcott commented 2 years ago

Tried setting up the sensor account in Veeam with Administrator rights and that didn't work.

Not sure if anything else can be done.

vMarkusK commented 2 years ago

Sorry, bus this sounds definitely like a permission issue.

An other workaround is installing a probe service on the veeam Server itself and move the sensor to this probe. Have you tried this workaround?

RonSharcott commented 2 years ago

It does look like permissions. I tried a probe on the Veeam server with no success. Not sure how much more time I can throw at this. I did try setting up accounts in Veeam and also using the accounts that were already listed. My guess at this point is that there may be something prohibiting the use of the APIs either in Veeam or on the server.

vMarkusK commented 2 years ago

Please have a look at this new and enhanced Version: https://github.com/vMarkusK/Advanced-PRTG-Sensors/blob/master/Veeam/PRTG-VeeamBRStats-v3.ps1

IcyPunch commented 2 years ago

We also did the veeam upgrade from 10 to 11. Since then the script is broken for us as well. Nothing else changed. image

vMarkusK commented 2 years ago

@IcyPunch have you upgraded to vbr console on the probe?

Please have a look at the v2 version of the script in the repo. This might solve a lot of issues.

IcyPunch commented 2 years ago

@vMarkusK I have used this one today, is it incorrect?: image

vMarkusK commented 2 years ago

@vMarkusK

I have used this one today, is it incorrect?:

image

That's the right one. With that version you do not need the PSX64.exe anymore.

Please try to connect to the vbr server from the probe device. Missing upgrade of the console is most common issue.

IcyPunch commented 2 years ago

@vMarkusK You were absolutely correct with your hint of not having upgraded the veeam console on the prtg server. I feel stupid saying / admitting this, but I'm very happy to confirm it's working again now after i did exactly that. First time around I remembered it and now I didn't :-P Thank you for your patience and advice!