Response status code does not indicate success: 556

bajayb4u commented 2 years ago

Describe the bug

Hello,

Using PrtgAPI module v0.9.17, I am trying to add devices and sensors. No issues when adding device but only with add-sensors throwing this error. This error is not consistent even on the same device when added multiple times. Though most of times script is throwing this error sometimes it is working just fine.

Error: Failed CMD Line: $getDevice | Add-Sensor $sensorparams Error message: Response status code does not indicate success: 556 (Unknown Response Code).

Steps to reproduce

$PRTGGroupID = <PRTGGroupID> 
$deviceName = <deviceName> 
$ServerIP = <ServerIP> 
Connect-PrtgServer $prtgURL (New-Credential $prtgUserName $prtgPass) 
$addDevice = Get-Group -id $PRTGGroupID | Add-Device -Name $deviceName -Host $ServerIP 
$addDeviceId = $addDevice.Id 
$getDevice = Get-Device -Id $addDeviceId 

$SatSensor = "snmpcpu" 

$sensorparams = $getDevice | New-SensorParameters -RawType $SatSensor 
$cpusensor = $getDevice | Add-Sensor $sensorparams 
$cpuchannel = Get-Channel -Sensor $cpusensor 
$cputotal = $cpuchannel | Where {$_.Name -eq "Total"} 
Set-ChannelProperty -Channel $cputotal -Property UpperErrorLimit -Value 90 
Set-ChannelProperty -Channel $cputotal -Property UpperWarningLimit -Value 75

What is the output of 'Get-PrtgClient -Diagnostic'?

PSVersion      : 5.1.14393.4583
PSEdition      : Desktop
OS             : Microsoft Windows Server 2016 Standard
PrtgAPIVersion : 0.9.17
Culture        : en-US
CLRVersion     : .NET Framework 4.7.2 (461814)
PrtgVersion    : 22.1.74.1869
PrtgLanguage   : english.lng

Additional context

Above sample code has only snmpcpu sensor in it but we are also encountering same with ping and other SNMP sensors. Please let me know if I can provide any further information. Thanks for the help!

lordmilko commented 2 years ago

Did this issue only start happening recently? Or did you only start using PrtgAPI and/or this feature recently and so this has always occurred?

bajayb4u commented 2 years ago

We have recently upgraded Prtg to v22.1.74.1869. Before the upgrade we had v20.4.64.1402 and used PrtgAPI PS module v0.9.16 to add devices and sensors without any issues. After the upgrade we start seeing error "Could not authenticate to PRTG; the specified username and password were invalid", so we now using module v0.9.17. Now the authentication issue was resolved but getting this new error.

carlbeechvuw commented 2 years ago

Hi Don't know whether this helps, but here's the full exception that I'm seeing when getting the same problem (i.e. randomly when adding sensors - and the sensors are present, even though there's an exception reported:

System.Net.Http.HttpRequestException: Response status code does not indicate success: 556 (Unknown Response Code). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at PrtgAPI.Request.RequestEngine.ValidateHttpResponse(HttpResponseMessage responseMessage, PrtgResponse response) at PrtgAPI.Request.RequestEngine.ExecuteRequest(PrtgRequestMessage request, CancellationToken token, Func2 responseParser) at PrtgAPI.Request.RequestEngine.ExecuteRequest(ICommandParameters parameters, Func2 responseParser, CancellationToken token) at PrtgAPI.Request.VersionClient18_1.AddSensorInternal(ICommandParameters internalParams, Int32 index, CancellationToken token) at PrtgAPI.PrtgClient.AddObjectInternal(Either2 parent, NewObjectParameters parameters, CancellationToken token) at PrtgAPI.PrtgClient.<>c__DisplayClass151_01.b0(CancellationToken t) at PrtgAPI.PrtgClient.ResolveWithDiff[T](Action1 createObject, Func2 getObjects, Func3 exceptFunc, CancellationToken token, Action2 errorCallback, Func1 shouldStop, Boolean allowMultiple, String enhancedResolutionError) at PrtgAPI.PrtgClient.AddObject[T](Either2 parent, NewObjectParameters parameters, Func3 getObjects, Boolean resolve, CancellationToken token, Action2 errorCallback, Func`1 shouldStop, Boolean allowMultiple) at PrtgAPI.PowerShell.Base.NewObjectCmdlet.<>cDisplayClass6_01.<AddAndResolveObject>b__0() at PrtgAPI.PowerShell.Base.NewObjectCmdlet.AddAndResolveRunner[T](Func1 addObject) at PrtgAPI.PowerShell.Base.AddObject2.ExecuteOperationAction(Int32 destinationId) at PrtgAPI.PowerShell.Base.PrtgOperationCmdlet.ExecuteOperation(Action action, String progressMessage, Boolean complete, Boolean incrementRecord) at PrtgAPI.PowerShell.Base.AddObject2.AddObjectInternal(Int32 destinationId) at PrtgAPI.PowerShell.Cmdlets.AddSensor.ProcessRecordEx() at PrtgAPI.PowerShell.Base.PrtgCmdlet.ExecuteWithCoreState(Action action) at System.Management.Automation.CommandProcessor.ProcessRecord()

PrtgAPI 0.9.17 PRTG: 22.1.74.1869

lordmilko commented 2 years ago

I haven't been able to reproduce this issue myself, however are you able to advise

Whether upgrading to PRTG 22.1.75 resolves the issue?
Whether the sensor is actually created when you get this error?

bajayb4u commented 2 years ago

We just upgraded our PRTG to v22.1.74.1869 and it might a take while for one more upgrade
Yes, sensor is getting created however it is failing to get channel for the sensor and add any sensor properties. Also, as we add more sensors through a loop, script won't get pass the failed sensor.

We are not seeing full exception that was reported by @carlbeechvuw above, but based on that can you can any suggestion that we can try?

lordmilko commented 2 years ago

@bajayb4u,

I have no idea what the issue is; when trying to repro this you should be running one liner commands to narrow down what the source of the issue is. I tried reproducing the issue using your repro steps above; there is a mistake in your code: $DeviceId is never declared, however even if I manually fixup the code I'm unable to replicate the issue

lordmilko commented 2 years ago

@carlbeechvuw,

What sort of sensors are you trying to add? Can you provide the output of specifying -Verbose to the Add-Sensor cmdlet when it crashes (please omit the server, username and passhash)

carlbeechvuw commented 2 years ago

Hi

Its SNMP->Memory and SNMP->disk sensors that I'm adding, and it appears to be 'random' - so I'm guessing there's a possibility that it's a timing issue in relation to something going on inside PRTG possibly?

I'll see if I can reproduce and get output to you.

Thanks

Carl.

From: lordmilko @.> Sent: Friday, 1 April 2022 10:51 AM To: lordmilko/PrtgAPI @.> Cc: Carl Beech @.>; Mention @.> Subject: Re: [lordmilko/PrtgAPI] Response status code does not indicate success: 556 (Issue #280)

@carlbeechvuwhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcarlbeechvuw&data=04%7C01%7Ccarl.beech%40vuw.ac.nz%7Ccbad5be4f9c24974cca508da13608275%7Ccfe63e236951427e8683bb84dcf1d20c%7C0%7C0%7C637843602530291281%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FOTYLaPUolySWPf6%2B%2Be8KXU0%2BaZuPKkoqM4UaE730Vk%3D&reserved=0,

What sort of sensors are you trying to add? Can you provide the output of specifying -Verbose to the Add-Sensor cmdlet when it crashes (please omit the server, username and passhash)

- Reply to this email directly, view it on GitHubhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flordmilko%2FPrtgAPI%2Fissues%2F280%23issuecomment-1085159649&data=04%7C01%7Ccarl.beech%40vuw.ac.nz%7Ccbad5be4f9c24974cca508da13608275%7Ccfe63e236951427e8683bb84dcf1d20c%7C0%7C0%7C637843602530291281%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jAKotvfP96M8V4ducFBB9Qmye6KFzXxtgoheJSdIxKU%3D&reserved=0, or unsubscribehttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQBB65RM3JLBK74MJZKUWJTVCYM3HANCNFSM5R32HKSA&data=04%7C01%7Ccarl.beech%40vuw.ac.nz%7Ccbad5be4f9c24974cca508da13608275%7Ccfe63e236951427e8683bb84dcf1d20c%7C0%7C0%7C637843602530291281%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=gPMs2qhT0yXe9fz13QfHdPCBFm%2FbQ%2Fel8rR3CjWQKII%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.**@.>>

bajayb4u commented 2 years ago

Hi @lordmilko, here is the verbose output when adding the snmpcpu sensor; Reproduced using the same code I provided above (fixed the $DeviceId, it was a copy paste mistake). Script is throwing error at Add-sensor command.

Is it possible to ass some wait time to add-sensor?

VERBOSE: Performing the operation "Add-Sensor" on target "SNMP CPU Load 70 (Destination:2454566-pragtwb4 (XXX.XXX) (ID: 22222))".
VERBOSE: Add-Sensor: Synchronously executing request https://prtg.hostname.org/api/table.xml?content=sensors&columns=objid,name,probe,group,favorite,lastvalue,device,downtime,downtimetime,downtimesince,uptime,uptimetime,uptimesince,knowntime,cumsince,lastcheck,lastup,lastdown,minigraph,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=55555&filter_type=snmpcpu&username=**********&passhash=**********
VERBOSE: Add-Sensor: Synchronously executing request https://prtg.hostname.org/controls/addsensor2.htm?id=98452&sensortype=snmpcpu&username=**********&passhash=**********
VERBOSE: Add-Sensor: Synchronously executing request https://prtg.hostname.org/addsensor5.htm?name_=SNMPCPULoad70&priority_=3&inherittriggers=1&intervalgroup=1&interval_=60%7C60seconds&errorintervalsdown_=1&tags_=snmpcpucpuloadsensor&anti-csrf-token=8567bfq385b696    98fe5bn0MmEyNGQ2OGE4ZTY4MWVjYjc1OA%3D%3D&sensortype=snmpcpu&id=98452&tmpid=129

Add-Sensor : Response status code does not indicate success: 556 (Unknown Response Code).
At C:\work\Powershell\PRTG_AddDevice.ps1:53 char:49
 ...           $cpusensor = $getDevice | Add-Sensor $sensorparams -Verbose
     CategoryInfo          : NotSpecified: (:) [Add-Sensor], HttpRequestException
     FullyQualifiedErrorId : System.Net.Http.HttpRequestException,PrtgAPI.PowerShell.Cmdlets.AddSensor

   Get-Channel : PRTG was unable to complete the request. The server responded with the following error: There is no object with the specified ID.
At C:\work\Powershell\PRTG_AddDevice.ps1:57 char:37
                       $cpuchannel = Get-Channel -Sensor $cpusensor
     CategoryInfo          : InvalidOperation: (:) [Get-Channel], PrtgRequestException
     FullyQualifiedErrorId : PrtgRequestException,PrtgAPI.PowerShell.Cmdlets.GetChannel

    SNMP CPU Sensor channel:
Set-ChannelProperty : Cannot bind argument to parameter 'Channel' because it is null.
At C:\work\Powershell\PRTG_AddDevice.ps1:61 char:52
                       Set-ChannelProperty -Channel $cputotal -Propert ...
     CategoryInfo          : InvalidData: (:) [Set-ChannelProperty], ParameterBinding ValidationException
     FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,PrtgAPI.P owerShell.Cmdlets.SetChannelProperty

Set-ChannelProperty : Cannot bind argument to parameter 'Channel' because it is null.
At C:\work\Powershell\PRTG_AddDevice.ps1:62 char:52
                       Set-ChannelProperty -Channel $cputotal -Propert ...
     CategoryInfo          : InvalidData: (:) [Set-ChannelProperty], ParameterBinding ValidationException
     FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,PrtgAPI.PowerShell.Cmdlets.SetChannelProperty

lordmilko commented 2 years ago

Can you see if you can replicate this issue with HTTP, WMI Service or Advanced EXE/Script sensors? I would like to see whether you can reproduce the issue when using dynamic sensor parameters for these 3 (New-SensorParameters with a -RawType of http, exexml or wmiservice)

If you can reproduce the issue with one of these sensor types, can you then see if you can reproduce the issue using a "normal" set of sensor parameters for them instead (i.e. New-SensorParameters Http rather than New-SensorParameters -RawType http)

This will help tell us whether it's got something to do with something in the dynamic sensor parameters. I note that the anti-csrf-token in the URL above appears to have a tab in it. Something else worth doing might be validating that is in fact the case and clearing out this property and seeing if that has an effect

$sensorparams = $getDevice | New-SensorParameters -RawType $SatSensor 

# What is the output of this?
$sensorparams ."anti-csrf-token".contains("`t")

# What happens if you do this then try and add the sensor?
$sensorparams ."anti-csrf-token" = $null

bajayb4u commented 2 years ago

There is no tab in anti-csrf-token, kindly ignore that; I have edited it. But I will try to reproduce with dynamic and normal sensors as you suggested.

bajayb4u commented 2 years ago

@lordmilko, this error is also reproducible with adding ping sensor; Here also I am using raw type,

$SatSensor = "ping" $sensorparams = $getDevice | New-SensorParameters -RawType $SatSensor $getDevice | Add-Sensor $sensorparams

lordmilko commented 2 years ago

Please see my post above and advise what occurs if you test using raw HTTP/WMI Service/EXE Script Advanced sensors using raw + non-raw types

bajayb4u commented 2 years ago

@lordmilko,

I have tried http sensor few times with raw and non-raw types and in both the case the issue reproducible. Below is what I am getting,

Add-Sensor : Response status code does not indicate success: 556 (Unknown Response Code). At line:17 char:14 $getDevice | Add-Sensor $sensorparams


    + CategoryInfo          : NotSpecified: (:) [Add-Sensor], HttpRequestException
    + FullyQualifiedErrorId : System.Net.Http.HttpRequestException,PrtgAPI.PowerShell.Cmdlets.AddSensor

bajayb4u commented 2 years ago

I was able to reproduce the same error for WMI Service sensor with both raw type and non-raw type.

bajayb4u commented 2 years ago

Update, we had a call with Paessler, and they suspect the issue may relate to system resources. we have thousands of sensors in our PRTG, as system is busy taking care of other tasks it might creating some read and write locks which causing this issue. They made some recommendation to offload sensors on core server local probe to remote probe to free up the load on core server. I will check the script again once we made the recommended changes.

Thank you @lordmilko, for your help

lordmilko commented 2 years ago

PrtgAPI should be able to handle any scenario regardless of load; the common scenario is that API requests time out (as opposed to returning random error codes). PrtgAPI has special logic for retrying API requests that fail due to a suspected timeout, so if that is indeed the case here an appropriate error handler needs to be implemented

@carlbeechvuw how many sensors do you have in your PRTG server?

bajayb4u commented 2 years ago

Our PRTG server has 22300 sensors.

Silex commented 2 years ago

We have the same issue (add-sensor failing randomly), our PRTG is huge and has way too many devices/sensors, maybe that's the issue.

    Write-Output ('Processing "{0}"' -f $sensor.Device)
    $device = Get-Device -Probe $probe -Name $sensor.Device -Group $sensor.Group

    $params = New-SensorParameters ExeXml 'My Sensor'
    $params.ExeFile = 'script.bat'
    $params.ExeParameters = '%host'
    $params.UseWindowsAuthentication = $true
    $params.Timeout = 120
    $ErrorActionPreference = 'Continue'
    Add-Sensor -ErrorAction Continue -Destination $device -Parameters $params | Out-Null
    $ErrorActionPreference = 'Stop'

Output:

Processing "DEVICE1"
Processing "DEVICE2"
Processing "DEVICE3"
Add-Sensor : Response status code does not indicate success: 556 (Unknown Response Code).
At C:\CCTV\Scripts\PRTG\PrtgReconfigureStorages.ps1:26 char:5
+     Add-Sensor -ErrorAction Continue -Destination $device -Parameters ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Add-Sensor], HttpRequestException
    + FullyQualifiedErrorId : System.Net.Http.HttpRequestException,PrtgAPI.PowerShell.Cmdlets.AddSensor

What makes NO SENSE to me is that despite all my attemps to tell it "PLEASE CONTINUE" it stops at the first error. As if $ErrorActionPreference and -ErrorAction was ignored.

EDIT: ok, it makes sense because it's actually an exception. Also it can be ignored because in PRTG the sensor is added (even tho it's reported as an error HTTP 556).

EDIT2: question, if you only have a sensor, is $device = Get-Device -Probe $probe -Name $sensor.Device -Group $sensor.Group the best way to get the device?

Silex commented 2 years ago

Hello,

I wrapped the Add-Sensor in a try/catch which warns "Supposedly cannot add sensor" and here's the output:

Processing "DEVICE1"
Processing "DEVICE2"
WARNING: Supposedly cannot add sensor
Processing "DEVICE3"
WARNING: 'Add-Sensor' failed to resolve sensor: object is still being created. Retries remaining: 4
WARNING: 'Add-Sensor' failed to resolve sensor: object is still being created. Retries remaining: 3
Processing "DEVICE4"
Processing "DEVICE5"

It's very likely that this bug is triggered by an overloaded PRTG server, and HTTP 556 means something like "Ok but wait".

lordmilko commented 2 years ago

Based on the reports made in this thread it sounds to me as if HTTP status code 556 can indeed be safely ignored as objects are actually created properly, however I've raised a case with Paessler to get their input on this as well. Users need to be confident their objects were indeed created successfully; PrtgAPI suppressing an error because it thinks its "probably" fine is not really an ideal outcome, and implementing the necessary logic to "double check" seems like it would be very painful

lordmilko commented 2 years ago

@Silex

In regards to your question, if you have a sensor then the best way to get the device is to do Get-Device -Id $sensor.ParentId

bajayb4u commented 2 years ago

Hello @lordmilko,

How to ignore only status code 556 and continue with the rest of the script? It appears add-sensor is ignoring "-ErrorAction Continue". In my case I have to set error limits to the sensor once it added. but script errors out at add-sensor with status code 556. Even if we are able to ignore the 556, will get-channel works for that sensor?

lordmilko commented 2 years ago

Hi @bajayb4u,

-ErrorAction only applies to non-terminating exceptions (i.e. emitted ErrorRecord objects). To ignore an exception you need to wrap the relevant code in a try/catch block

try
{
    # do stuff
}
catch
{
    Write-Host $_.Exception.Message
}

lordmilko commented 2 years ago

Hi all,

Please be advised I have been able to reproduce this issue and have released a new pre-release build of PrtgAPI which should hopefully include a fix for this issue

Can you please follow the manual installation instructions and advise whether the issue is now resolved? If you've successfully followed the manual installation instructions, Get-PrtgClient -Diagnostic should show 0.9.18-preview.4

bajayb4u commented 2 years ago

Hi @lordmilko,

I have tested the 0.9.18-preview.4. It appears status code 556 is no longer reproducible but getting timeout error (which may not related to the status code 556) when adding the sensors. At the end prtgapi is adding the snmp sensors but adding it TWICE and failing to get the channel to further apply sensor limits. Here is the script and error I am getting. Is there any workaorund to fix these issues?

$SatSensor = "snmpcpu" $getDevice = Get-Device -Id $addDeviceId $sensorparams = $getDevice | New-SensorParameters -RawType $SatSensor $cpusensor = $getDevice | Add-Sensor $sensorparams $cpuchannel = Get-Channel -Sensor $cpusensor $cputotal = $cpuchannel | Where {$_.Name -eq "Total"} Set-ChannelProperty -Channel $cputotal -Property UpperErrorLimit -Value 90 Set-ChannelProperty -Channel $cputotal -Property UpperWarningLimit -Value 75

Adding Sensor: snmpcpu WARNING: 'Add-Sensor' timed out: The server timed out while executing request. Retries remaining: 6

following error returned by "Get-Channel -Sensor $cpusensor" cmd

Error message: Cannot convert 'System.Object[]' to the type 'PrtgAPI.Sensor' required by parameter 'Sensor'. Specified method is not supported. 
PositionMessage: At line: char:
+ ...                  $cpuchannel = Get-Channel -Sensor $cpusensor

lordmilko commented 2 years ago

Hi @bajayb4u,

You should not be doing

$cpuchannel = Get-Channel -Sensor $cpusensor

If $cpusensor contains multiple objects you'll get the error you got above. This is an error the PowerShell Engine throws due to how you've written your script, as opposed to being anything to do with PrtgAPI specifically.

The recommended way of using the Get-Channel cmdlet - and indeed any cmdlet in PowerShell, is as so

$cpusensor | Get-Channel

By piping the $cpusensor variable, the PowerShell engine will enumerate through each object in the variable, assigning each one to the -Sensor parameter one at a time

I can't see any reason why the sensor would be created twice; are you potentially able to test with another sensor type such as HTTP and see if it is added twice as well?

bajayb4u commented 2 years ago

Hi @lordmilko, This issue has been resolved after we have upgraded our PRTG version. But it is only working with 0.9.18-preview.4 version. Thanks for your help,

lordmilko / PrtgAPI