Open xasz opened 2 years ago
I'm unable to reproduce this issue. Are you able to confirm what version of PrtgAPI you're running?
Has the above screenshot been manipulated, or is it the output of a script? (as opposed to raw commands being typed in a PowerShell prompt)
Can you also please provide the output of Get-PrtgClient -Diagnostic
The screenshot is not manipulated in any form. It was just a half output of the get-prtgclient.
Sure, here we go with some additional informations:
Get-PrtgClient -Diagnostic
PSVersion : 5.1.17763.2803
PSEdition : Desktop
OS : Microsoft Windows Server 2019 Standard
PrtgAPIVersion : 0.9.17
Culture : de-DE
CLRVersion : .NET Framework 4.7.2 (461814)
PrtgVersion : 22.2.76.1705
PrtgLanguage : german.lng
The "call" of the actual Add-Sensor/Add-Device Sensor is quite complex and i just can't copy paste it in three lines.
This is my Setup:
Import-Module prtghelper.psm1 -force # This is a custom helper module for us internal to share some methods between scripts
Connect-PrtgServer "https://$($PRTGServer):$($PRTGPort)" (New-Credential $PRTGUser $PRTGPassHash) -PassHash -RetryCount 50 -RetryDelay 50
# Gather some devices from the crm
foreach($erpdevice in $erpdevices){
$myDevice = Set-PRTGDevice -Parent $someParentGroup -ERPDevice $erpdevice
# now we add / replace some sensors with the "same" method as on the the device.
}
function Set-PRTGDevice{
param($Parent, $ERPDevice)
$device = $Parent | Get-Device -Name $DeviceName
# Sometimes this just return nothing even if there is a correct device
if(($device | Measure-Object).Count -gt 0){
if(($device | Measure-Object).Count -gt 1){
throw "$DeviceName in $($Parent.Name) not unique"
}
$device | Set-ObjectProperty Host $Hostname
}else{
$device = $Parent | Add-Device $DeviceName $($Hostname)
# THIS will "fail" and take aged on adding multiple devices. and i return a wrong result
}
return $device
}
I have the same problem 😞 With PrtgApi 0.9.19
RetryCount when using Add-Device
seems to be hardcoded at 5.
Variable is correctly set:
Connect-PrtgServer -Server $PrtgServer -Credential $credentials -PassHash -IgnoreSSL -RetryCount 10 -RetryDelay 5
Set-PrtgClient -RetryCount 10 -RetryDelay 30
Get-PrtgClient -Diagnostic | fl *
Get-PrtgClient | fl *
...
$device = Add-Device -Name $name -Host $_.album.url -Destination $group
Output:
PSVersion : 5.1.19041.4648
PSEdition : Desktop
OS : Microsoft Windows 10 Pro
PrtgAPIVersion : 0.9.19
Culture : en-US
CLRVersion : 533325
PrtgVersion : 24.2.96.1375
PrtgLanguage : english.lng
Targets : PrtgAPI.Request.PrtgTargetHelper
Server : localhost
UserName : prtgadmin
PassHash : 2154168053
RetryCount : 10
RetryDelay : 30
LogLevel : Trace, Request
Version : 24.2.96.1375
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 4
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 3
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 2
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 1
Add-Device : Could not resolve object: PRTG is taking too long to create the object. Confirm the object has been
created in the Web UI and then attempt resolution again manually.
At C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML\SyncWithArqivis.ps1:63 char:17
+ ... $device = Add-Device -Name $name -Host $_.album.url -Destination $g ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Add-Device], ObjectResolutionException
+ FullyQualifiedErrorId : PrtgAPI.ObjectResolutionException,PrtgAPI.PowerShell.Cmdlets.AddDevice
Ok I think I found the cause:
It looks hardcoded here.
And it it's not then probably that https://github.com/lordmilko/PrtgAPI/blob/e3fcb4072b3214486bc9618f72b39a9068fb68aa/src/PrtgAPI/Request/RequestEngine.cs#L244 is passed a copy of the PrtgClient instead of the modified one.
@lordmilko do you have a workaround, other than rolling our own sleep-then-check-in-a-loop?
Thanks @Silex,
I think I understand what's happening here. There are two levels of retries
RequestEngine
relate to when the PRTG server throws an exception saying that your request timed out. If that occurs, we allow retrying a set number of times with an increasing backoff delayRequestObject
method pertain to the resolution of the created object. The API request asking for the created object is succeeding...it's just that the object hasn't been created yet.I have effectively moved onto other projects from PrtgAPI, and am mostly providing basic level support for the product as-is until Paessler break API V1 compatibility. I think exposing something to allow specifying an object-creation-resolution-retry-count should be easy enough however, so I will work on doing this
Hi all,
Can you please try the latest pre-release build and advise how you go? PrtgAPI now takes the RetryCount
and RetryDelay
into consideration when resolving objects. I have updated the wiki to make note of this fact. Note that PrtgAPI uses a backoff algorithm when consuming the RetryDelay
, wherein it will wait longer and longer and longer with each failed attempt. As such, it's recommended to keep your retry delay small (i.e. 3 seconds) and add a few extra retry attempts
Thanks!
So, the countdown "works" but it never actually resolves...
Connect-PrtgServer -Server $PrtgServer -Credential $credentials -PassHash -IgnoreSSL -RetryCount 10 -RetryDelay 5
...
$device = Add-Device -Name $name -Host $_.album.url -Destination $group
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 10
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 9
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 8
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 7
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 6
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 5
The time it takes between tries is not linear, at first it takes 5 seconds before retrying... then 10... then 15... then 20... it looks like you don't just sleep $RetryDelay
, but something akin to $i * $RetryDelay
.
Then the other problem is that it never detects the device was actually added.
Ok, by replacing with the following workaround it works (notice the -Resolve:$false):
$device = Add-Device -Name $name -Host $_.album.url -Destination $group -Resolve:$false
while ($true) {
$device = $group | Get-Device | Where-Object Name -eq $name
if ($device) {
break
}
Write-Warning 'wait'
Start-Sleep 5
}
That said, I found a weird bug:
C:\> $name.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object
C:\> $name.length
53
C:\> $device = Get-Device -Name "$name*"
C:\> $device.name -eq $name
True
C:\> (Get-Device -Name $name).Count
0
C:\> (Get-Device | ? Name -eq $name).Count
1
I cannot show you the name because it's a client's name, but as you can see Get-Device
works only if I use -Name
with a wildcard or with Where-Object
, but using the plain string does not work.
The name has a value similar to this (maybe pipes/quotes cause troubles):
Foo "BAR-BAZ | BAK | Lorem Ipsum KEWK"
Maybe that bug is related to the Add-Device
bug that does not resolve objects, because it checks with exact name?
What type of sensor are you adding? If it's something to do with SNMP, the created type may differ from what you're adding (see the wiki for more info). You have to specify $params.DynamicTrue = $true
in that scenario
If $name
contains a character that is considered a wildcard by PowerShell, this can prevent Get-Device
from returning the expected results, because it attempts to filter the results returned by PRTG for those that match the wildcard expression. Wildcard characters include *
, ?
[
and ]
. So a [
in a wildcard pattern won't match a [
literal. If you experience issues with the -Name
parameter, you can use the New-SearchFilter
cmdlet (aliased as flt
) to be explicit
flt name contains "Hello [world]" | Get-Device
Here's a testcase:
$PrtgServer = 'localhost'
$PrtgUserName = 'admin@bla.com'
$PrtgUserHash = '123456789'
$credentials = New-Credential -Username $PrtgUsername -Password $PrtgPasshash
Connect-PrtgServer -Server $PrtgServer -Credential $credentials -PassHash -IgnoreSSL -RetryCount 10 -RetryDelay 5
$group = Get-Group -Name 'Test group'
if (!$group) {
throw 'Could not find group'
}
$name = 'Foo "BAR-BAZ | BAK | Lorem Ipsum KEWK"'
$device = Add-Device -Name $name -Host '1.2.3.4' -Destination $group
# This outputs this:
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 10
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 9
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 8
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 7
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 6
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 5
I don't mention the sensor because the error is when I create the device, so it's the Add-Device
call like my screenshot is showing.
As you see the name has no special characters besides quotes & pipes.
If you do Set-PrtgClient -LogLevel All
and then do
$device = Add-Device -Name $name -Host '1.2.3.4' -Destination $group -Verbose
what does it respond with? And while it's timing out creating the device, if you look in the PRTG UI has the device been created?
what does it respond with? And while it's timing out creating the device, if you look in the PRTG UI has the device been created?
While it's timeing out I see the device created in PRTG, but PrtgAPI is still waiting for it.
I'll report what the "-LogLevel All" says soonish.
The -Verbose
output will contain sensitive information, but the key thing we want to ascertain is to what extent the name in the URL it's querying for matches what you're specifying to the cmdlet, and whether the object is being included in the response and PrtgAPI is just ignoring it. I have a strong suspicion that the issue is related to your use of |
pipe characters; I suspect PRTG or PowerShell is interpreting these in a special way
Ok, I found out it's the quotes and not the pipes. It used to work. I guess latest updates from PRTG broke something, the good news is that I can use any string to trigger it, will post traces soon.
So, here are the trace of a working request and a failing request.
In both case, the device is created just fine, it's just the resolving of the new device that works in the first case and fails in the second case.
Here's the trace of a request working (name Test
):
PS C:\> $device = Add-Device -Name 'Test' -Host '1.2.3.4' -Destination $group -Verbose
VERBOSE: Performing the operation "Add-Device" on target "Test (Host: 1.2.3.4) (Destination: Test (ID: 5912))".
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="0" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
</devices>
VERBOSE: Add-Device: Synchronously executing request https://localhost/adddevice2.htm?name_=Test&host_=1.2.3.4&ipversion_=0&discoverytype_=0&discoveryschedule_=0&id=5912&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <!DOCTYPE html>
<html>
<!--
_____ _______ _______ _______ _______ _______ ______
|_____] |_____| |______ |______ |______ | |______ |_____/
| | | |______ ______| ______| |_____ |______ | \_
We are hiring! https://jobs.paessler.com
-->
<head>
<-- snip long HTML -->
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="1" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
<item>
<objid>7269</objid>
<name>Test</name>
<location><a href="/devices.htm?filter_location=@sub(RETRACTED)">RETRACTED.</a></location>
<location_raw>RETRACTED</location_raw>
<host>1.2.3.4</host>
<group>Test</group>
<probe>Local Probe</probe>
<favorite><span class="objectisnotfavorite icon-gray ui-icon ui-icon-flag" title="Flag/Unflag as
Favorite" id="fav-7269"
onclick="_Prtg.objectTools.faveObject.call(this,7269,'toggle');return
false;"></span></favorite>
<favorite_raw>0</favorite_raw>
<upsens/>
<upsens_raw>0000000000</upsens_raw>
<downsens/>
<downsens_raw>0000000000</downsens_raw>
<downacksens/>
<downacksens_raw>0000000000</downacksens_raw>
<partialdownsens/>
<partialdownsens_raw>0000000000</partialdownsens_raw>
<warnsens/>
<warnsens_raw>0000000000</warnsens_raw>
<pausedsens/>
<pausedsens_raw>0000000000</pausedsens_raw>
<unusualsens/>
<unusualsens_raw>0000000000</unusualsens_raw>
<undefinedsens/>
<undefinedsens_raw>0000000000</undefinedsens_raw>
<totalsens>0</totalsens>
<totalsens_raw>0000000000</totalsens_raw>
<schedule/>
<basetype>device</basetype>
<baselink>/device.htm?id=7269</baselink>
<baselink_raw>7269</baselink_raw>
<notifiesx>Inherited</notifiesx>
<intervalx>Inherited (600)</intervalx>
<intervalx_raw>0000000060</intervalx_raw>
<access>Full access</access>
<access_raw>0000000400</access_raw>
<dependency>Parent</dependency>
<dependency_raw>Test</dependency_raw>
<position>810</position>
<position_raw>0000000810</position_raw>
<status>Up </status>
<status_raw>3</status_raw>
<priority>3</priority>
<message><div class="status">OK<div
class="moreicon"></div></div></message>
<message_raw>OK</message_raw>
<parentid>5912</parentid>
<type>Device</type>
<type_raw>device</type_raw>
<active>True</active>
<active_raw>-1</active_raw>
</item>
</devices>
And here's the failing one (name Test "Foobar"
):
PS C:\> $device = Add-Device -Name 'Test "Foobar"' -Host '1.2.3.4' -Destination $group -Verbose
VERBOSE: Performing the operation "Add-Device" on target "Test "Foobar" (Host: 1.2.3.4) (Destination: Test (ID: 5912))".
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test+%22Foobar%22&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="0" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
</devices>
VERBOSE: Add-Device: Synchronously executing request https://localhost/adddevice2.htm?name_=Test+%22Foobar%22&host_=1.2.3.4&ipversion_=0&discoverytype_=0&discoveryschedule_=0&id=5912&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <!DOCTYPE html>
<html>
<!--
_____ _______ _______ _______ _______ _______ ______
|_____] |_____| |______ |______ |______ | |______ |_____/
| | | |______ ______| ______| |_____ |______ | \_
We are hiring! https://jobs.paessler.com
-->
<head>
<!-- snip long HTML -->
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test+%22Foobar%22&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="0" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
</devices>
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 4
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test+%22Foobar%22&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="0" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
</devices>
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 3
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test+%22Foobar%22&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="0" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
</devices>
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 2
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test+%22Foobar%22&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="0" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
</devices>
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 1
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test+%22Foobar%22&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="0" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
</devices>
Add-Device : Could not resolve object: PRTG is taking too long to create the object. Confirm the object has been created in the Web UI and then attempt resolution
again manually.
At line:1 char:11
+ $device = Add-Device -Name 'Test "Foobar"' -Host '1.2.3.4' -Destinati ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Add-Device], ObjectResolutionException
+ FullyQualifiedErrorId : PrtgAPI.ObjectResolutionException,PrtgAPI.PowerShell.Cmdlets.AddDevice
For the record I'm completely reinstalling PRTG because it behaves weirdly, will check if the bug is still happening in 24.3.98.1210
Thanks @Silex,
On the basis that I'm escaping the quotes correctly, I feel like PrtgAPI is doing the right thing.
If you paste this URL into your web browser it will also give the XML response. You can then potentially fiddle around specifying different values to the filter_name
parameter, which may enable you to tell Paessler what's wrong.
https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test+%22Foobar%22&username=prtgadmin&passhash=RETRACTED
Alternatively, if you find that names with single quotes work, potentially modifying your names to use single quotes instead of double quotes would work? This can easily be done with a one-liner with PrtgAPI
Yeah, it looks like a PRTG regression.
Still what's weird is that PRTG creates the node with quotes just fine. And my workaround where I query the name using Where-Object
on Get-Device
in a loop also work just fine.
So my guess is that the regression is subtle... like only filter_name
in the PRTG api regressed.
For the record single quotes don't have this problem.
Anyway, about the original issue I think your fix works, but I think you'd not make it so the delay increases with each tries.
Even with the default values it's like it takes 3 seconds for the first try, then 6, then 9...
About the issue with the quotes, I'll reopen another case if needed.
The purpose of the request retry feature is that in large PRTG installations (10,000 sensors) there can be quite a delay in attempting to perform certain actions, in particular creating new objects. Thus, when PrtgAPI repeatedly queries for the given object, and find again and again it hasn't been created yet, the assumption is that your sever must be under intense load; we want to do our best to guarantee that your request will succeed, hence why PrtgAPI will wait longer and longer and longer to maximize its chance at success.
The implication of this however is that you should not go too crazy modifying the default retry parameters; I believe the defaults might result in a total wait of around 30 seconds or so. The scaling behavior ensures that PrtgAPI's default retry rules will generally function well on both small and large PRTG installations, scaling up the delay as required for your system.
Understood. I'd maybe consider renaming RetryDelay
to RetryFactor
or something that implies it's exponential.
What's going on?
Hello,
we have a quite large PRTG Installation and sometimes it takes a long time that the prtg api does it thing ;) No i tried to set RetryCount and RetryDelay higher but it does not seem to work:
I set RetryCount and RetryDelay to 20:
Connect-PrtgServer "https://$($PRTGServer):$($PRTGPort)" (New-Credential $PRTGUser $PRTGPassHash) -PassHash -RetryCount 20 -RetryDelay 20
And now i expected that i see something like Retries remaining 20 instead of 4 ? Am i just doing it wrong?thank you, best regards xasz
Due Dilligance