lordmilko / PrtgAPI

C#/PowerShell interface for PRTG Network Monitor
MIT License
301 stars 37 forks source link

Sorry, the object is currently not valid, try again later. #200

Closed Z3nto closed 3 years ago

Z3nto commented 3 years ago

Hi @lordmilko ,

I've found a bug that I'm not sure if it is caused by PrtgAPI or by PRTG itself. Maybe you can have a look.

Sporadically my program crashes while creating an ICMP Ping Sensor, it does not occur every time. The error is

Unhandled Exception: PrtgAPI.PrtgRequestException: PRTG was unable to complete the request. The server responded with the following error: Sorry, the object is currently not valid, try again later.

Thats my code for creating the ping sensor:

DynamicSensorParameters dynamicParameters = this.PRTGclient.GetDynamicSensorParameters(device, "ping");
this.PRTGclient.AddSensor(device, dynamicParameters);

Here is some debugging output:

Synchronously executing request https://xyz/api/sensortypes.json?id=73323&username=xyz&passhash=xyz
Synchronously executing request https://xyz/controls/addsensor2.htm?id=73323&sensortype=ping&username=xyz&passhash=xyz
Synchronously executing request https://xyz/api/getaddsensorprogress.htm?id=73323&tmpid=127
Synchronously executing request https://xyz/addsensor4.htm?id=73323&tmpid=127
Synchronously executing request https://xyz/api/table.xml?content=sensors&columns=objid,name,probe,group,favorite,lastvalue,device,downtime,downtimetime,downtimesince,uptime,uptimetime,uptimesince,knowntime,cumsince,lastcheck,lastup,lastdown,minigraph,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=73323&filter_type=ping&username=xyz&passhash=xyz
Synchronously executing request https://xyz/controls/addsensor2.htm?id=73323&sensortype=ping&username=xyz&passhash=xyz
Synchronously executing request https://xyz/addsensor5.htm?name_=Ping&priority_=3&inherittriggers=1&intervalgroup=1&interval_=60%7C60+seconds&errorintervalsdown_=1&tags_=pingsensor&timeout_=5&size_=32&countmethod_=1&count_=5&delay_=5&autoacknowledge_=0&sensortype=ping&id=73323&tmpid=128

Unhandled Exception: PrtgAPI.PrtgRequestException: PRTG was unable to complete the request. The server responded with the following error: Sorry, the
object is currently not valid, try again later.
   at PrtgAPI.Request.RequestEngine.ValidateHttpResponse(HttpResponseMessage responseMessage, PrtgResponse response)
   at PrtgAPI.Request.RequestEngine.ValidateHttpResponse(PrtgClient client, PrtgRequestMessage request, HttpResponseMessage responseMessage, PrtgRespo
nse response)
   at PrtgAPI.Request.RequestEngine.ExecuteRequest(PrtgRequestMessage request, CancellationToken token, Func`2 responseParser)
   at PrtgAPI.Request.RequestEngine.ExecuteRequest(ICommandParameters parameters, Func`2 responseParser, CancellationToken token)
   at PrtgAPI.Request.VersionClient18_1.AddSensorInternal(ICommandParameters internalParams, Int32 index, CancellationToken token)
   at PrtgAPI.PrtgClient.AddObjectInternal(Either`2 parent, NewObjectParameters parameters, CancellationToken token)
   at PrtgAPI.PrtgClient.<>c__DisplayClass148_0`1.<AddObject>b__0(CancellationToken t)
   at PrtgAPI.PrtgClient.ResolveWithDiff[T](Action`1 createObject, Func`2 getObjects, Func`3 exceptFunc, CancellationToken token, Action`2 errorCallba
ck, Func`1 shouldStop, Boolean allowMultiple)
   at PrtgAPI.PrtgClient.AddObject[T](Either`2 parent, NewObjectParameters parameters, Func`3 getObjects, Boolean resolve, CancellationToken token, Ac
tion`2 errorCallback, Func`1 shouldStop, Boolean allowMultiple)
   at PrtgAPI.PrtgClient.AddSensor(Either`2 deviceOrId, NewSensorParameters parameters, Boolean resolve, CancellationToken token)
   at DMVRolloutLHD.DMVRolloutLHD.create()
   at DMVRolloutLHD.Program.Main(String[] args)

I'm using PrtgAPI 0.9.14 with PRTG 21.1.65.1767+.

Thanks Z3nto

lordmilko commented 3 years ago

Hi @Z3nto,

The tricky question here is whether PRTG is just doing this randomly (in which case you'd expect to be able to replicate this behavior sometimes when creating sensors in the PRTG UI) or whether there is some new behavior of PRTG that PrtgAPI needs to implement.

The first thing that comes to mind when looking at this error is whether the tmpid has expired or is no longer valid. In the API request to addsensor2.htm we establish an add sensor "session" (like you would get when you actually select the sensor to add in the UI). Since these sessions don't actually support the use of GET authentication we utilize a cookie for identifying our previously authenticated session instead. One interesting question could be whether you have a particularly large PRTG install - maybe it takes so long for addsensor5.htm to execute that the tmpid has now expired? (that seems unlikely)

I ran a loop of creating ping dynamic sensor parameters and adding them in PowerShell; I let this run 200 times and didn't experience any issues.

The question I would be most interested in is whether this error can actually be ignored - if you kept trying to make the API request, would it eventually succeed? If you can potentially download and compile PrtgAPI (easiest way is probably to open PrtgAPI.sln, restore NuGet packages then compile)., Then, if you can replace this function

internal void AddObjectInternalDefault(ICommandParameters internalParams, CancellationToken token) =>
            RequestEngine.ExecuteRequest(internalParams, token: token);

with something like

internal void AddObjectInternalDefault(ICommandParametersV1 internalParams, CancellationToken token)
{
    try
    {
        while (true)
        {
            RequestEngine.ExecuteRequest(internalParams, token: token);

            return;
        }
    }
    catch(Exception ex)
    {
        Log("Failed: " + ex.Message + ", retrying", LogLevel.Trace);
        Thread.Sleep(10);
    }
}

and then compile and run your program with the PrtgClient's LogLevel set to Request and Trace, hopefully we can somehow replicate the failure, but then also see how it eventually succeeds or maybe changes to a different exception message

lordmilko commented 3 years ago

In addition, if you look at the Core or Web Server logs on your PRTG Core server, I'd be interested to know whether they provide more information about what exactly the error was (or which object ID it objected to) when you look around the time period this API request was executed

lordmilko commented 3 years ago

Hi @Z3nto,

Are you able to advise how you went with this/whether you are still experiencing this issue?

Z3nto commented 3 years ago

Hi @lordmilko sorry for my late response.

I've implemented your loop solution, and since then it worked. Seems that is again some sort of PRTG busy bug. I've not yet found an reason in the log files but I will report it to paessler once i've found one.