Exporting a group's worth of objects and importing to new core

[SamuelmAsh]

Hi LordMilko! I know we spoke a while ago about the unforseen complexities of moving objects between cores and the delays surrounding that. I was just wondering if you have any pointers on how to perhaps export a group to a bunch of csv/xml/json that can then be fed back to PRTGAPI as new devices and sensors on a new core?

[lordmilko]

There are a few ways you can look at this problem. From my perspective, I am interested in implementing a complete tree traversal mechanism. In a tree consisting of probes, groups, devices and sensors, there are three "types" of objects

• Containers, containing groups and devices (probes, groups)
• Devices, containing sensors
• The sensors themselves

Based on this information, you can write a series of functions for recursively retrieving the children of an object. Once you have that, it is simply a matter of plugging in something to "do" to each of these elements (for example, by passing a ScriptBlock as a parameter to each function so that it may execute it for each object it processes). For example, you may have your script block record the name of the object, and the name of its parent. If it's a device, include its hostname/IP Address too.

Of course, if all you're doing is recording parent/child relationships, perhaps an easier solution could be to simply get all groups and all devices, record the pertinent details of the object and its parent relationship in a CSV, which you can convert into a tree upon importation via the use of the Group-Object cmdlet.

For example, consider the following CSV output

Name,Type,ParentName
"dc-1","Device","Servers"
"Servers","Group","Local Probe"
"Child Servers", "Group", "Servers"

When you process this in PowerShell, you get the following

PS Z:\> import-csv Z:\output.csv

Name          Type   ParentName
----          ----   ----------
dc-1          Device Servers
Servers       Group  Local Probe
Child Servers Group  Servers

PS Z:\> import-csv Z:\output.csv|group parentname

Count Name                      Group
----- ----                      -----
    2 Servers                   {@{Name=dc-1; Type=Device; ParentName=Servers}, @{...
    1 Local Probe               {@{Name=Servers; Type=Group; ParentName=Local Prob...

PS Z:\>

By grouping on the ParentName column, we group all of the devices and groups that go under a given group together. This then makes it easy to iterate over each of the groups, reconstructing its tree

foreach($group in $groups)
{
    $g = Get-Probe -Id 1 | Add-Group $group.Name

    foreach($child in $g.Group)
    {
        ProcessChild($child)
    }
}

In our hypothetical ProcessChild function, it will either determine the child is a group (in which case we need to recurse again into a ProcessGroup function) or is a device (in which case we add the device in our ProcessDevice function). Of course, if you have multiple levels of groups, you'll need some sort of additional checking that a grandchild group's parent group wasn't already added when the grandparent group was being processed.

When it comes to recreating sensors, in theory you can export all the raw properties with Get-ObjectProperty -Raw and reinsert them with Set-ObjectProperty -RawParameters in conjunction with some dynamic sensor parameters, however you're likely going to find this will be very challenging and there will be a litany of complex corner cases you need to deal with.

--

Bottom line, I recommend you forget trying to recreate sensors, and instead simply focus on a scheme to export and import the structure of your tree - i.e. your groups, devices and their hostnames/IP Addresses. If all goes well, you should be able to accomplish this in an hour or two.

[SamuelmAsh]

Hi! Thanks for that very fullsome reply! i have made some progress with the csv approach. A quickie though, do you have a tip for working out the nested groups issue?

My current theory is to use the parentID and TotalGroups field. Something along the lines of listing all the BaseType groups and their parent ID and TotalGroups and then referencing them against each other? or is there a way of querying groups to a certain depth? so i can do each layer at a time?

[lordmilko]

I'm not sure exactly what you're asking, however the following is a full code example of how you might convert your PRTG tree's structure to a CSV

My PRTG CI server contains a simple tree structured as follows

• Local Probe (Probe)
  • Probe Device (Device)
  • Servers (Group)
    • ci-prtg-1 (Device)
    • SensorTypes (Device)
    • Group (Group)
      • Nested Device (Device)
      • Nested Group (Group)

The following script demonstrates how you might convert this tree into a simple CSV

function ProcessContainer($obj, $parent)
{
    $groups = $obj | Get-Group -Recurse:$false -ParentId $obj.Id

    AddObject $obj $parent

    foreach($group in $groups)
    {
        ProcessContainer $group $obj
    }

    $devices = $obj | Get-Device -Recurse:$false -ParentId $obj.Id

    foreach($device in $devices)
    {
        ProcessDevice $device $obj
    }
}

function ProcessDevice($device, $parent)
{
    AddObject $device $parent
}

function AddObject($obj, $parent)
{
    $parentName = $null
    $parentId = $null

    if($parent -ne $null)
    {
        $parentName = $parent.Name
        $parentId = $parent.Id
    }

    return [PSCustomObject]@{
        Name = $obj.Name
        ParentName = $parentName
        ParentId = $parent.Id
        Type = $obj.Type
    }
}

$probe = Get-Probe

$result = ProcessContainer $probe

C:\> $result

Name         ParentName  ParentId Type  
----         ----------  -------- ----  
Local Probe                       Probe 
Servers      Local Probe 1        Group 
Group        Servers     2070     Group 
NestedGroup  Group       2073     Group 
NestedDevice Group       2073     Device
ci-prtg-1    Servers     2070     Device
SensorTypes  Servers     2070     Device
Probe Device Local Probe 1        Device

This should give you everything you need to then reconstruct the tree in your import script. Obviously you may need to tweak this based on your server's setup (e.g. if you have multiple probes). Note that despite the fact ProcessDevice doesn't do anything besides forward its parameters on to the AddObject function, we define the function anyway just in case we do in fact want to do some specialized processing on the device's sensors

[SamuelmAsh]

Hi LordMilko,

Thanks for your help with this. Thought you may be interested in seeing the final script i ended up using for rebuilding the groups on a new core. It works very well thanks to your guidance!

goprtg 1 

function ProcessContainer($obj, $parent)
{
    $groups = $obj | Get-Group -Recurse:$false #-ParentId $obj.Id

    $groups = $groups | Where-Object { $_.ParentId -eq $obj.Id }

    AddObject $obj $parent

    foreach($group in $groups)
    {
        ProcessContainer $group $obj
    }

    #$devices = $obj | Get-Device -Recurse:$false #-ParentId $obj.Id

    #foreach($device in $devices)
    #{
    #    ProcessDevice $device $obj
    #}
}

function ProcessDevice($device, $parent)
{
    AddObject $device $parent
}

function AddObject($obj, $parent)
{
    $parentName = $null
    $parentId = $null

    if($parent -ne $null)
    {
        $parentName = $parent.Name
        $parentId = $parent.Id
    } 

    Write-Host $obj.Name

    return [PSCustomObject]@{
        Name = $obj.Name
        Id = $obj.Id
        ParentName = $parentName
        ParentId = $parent.Id
        Type = $obj.Type
        NewId = -1
        NewParentId = -1
    }
}

$probe = Get-Probe -Id 90518

$result = ProcessContainer $probe

$result | ft

goprtg 2

foreach($thing in $result)
{
    if($thing.Type -eq "Probe")
    {
        $existingProbe = Get-Probe -Name $thing.Name
        if($existingProbe -ne $null)
        {
            $thing.NewId = $existingProbe.Id
            $thing.ParentId = $existingProbe.ParentId
        }
    }
    elseif($thing.Type -eq "Group")
    {
        $parent = ($result | Where-Object { $_.Id -eq $thing.ParentId } | select -First 1)

        if($parent -ne $null)
        {
            $newItem = $null
            if($parent.Type -eq "Probe")
            {
                Write-Host "Trying to find Probe: $($parent.NewId)"
                $newItem = Get-Probe -Id $parent.NewId
            }
            elseif($parent.Type -eq "Group")
            {
                Write-Host "Trying to find Group: $($parent.NewId)"
                $newItem = Get-Group -Id $parent.NewId
            }

            if($newItem -ne $null)
            {
                Write-Host "Found "
                $g = ($newItem | Get-Group $thing.Name);

                if($g -ne $null)
                {
                    $thing.NewId = $g.Id
                    $thing.NewParentId = $newItem.Id
                }
                else
                {
                    Write-Host " - Creating new group"
                    $newGroup = $newItem | Add-Group -Name $thing.Name
                    if($newGroup -ne $null)
                    {
                        $thing.NewId = $newGroup.Id

                        $thing.NewParentId = $newItem.Id    
                    }
                }
            }
        }
    }
}

[lordmilko]

Thanks Sam, glad to hear you got it all working

Regards, lordmilko

[wargal91]

Apologies for bringing this up again, but can something similar be used to export a group including all of its subgroups, devices and sensors in a format that can be used to import in another PRTG core instance using the same structure? Thanks in advance.

Hi LordMilko,

Thanks for your help with this. Thought you may be interested in seeing the final script i ended up using for rebuilding the groups on a new core. It works very well thanks to your guidance!

goprtg 1 

function ProcessContainer($obj, $parent)
{
    $groups = $obj | Get-Group -Recurse:$false #-ParentId $obj.Id

    $groups = $groups | Where-Object { $_.ParentId -eq $obj.Id }

    AddObject $obj $parent

    foreach($group in $groups)
    {
        ProcessContainer $group $obj
    }

    #$devices = $obj | Get-Device -Recurse:$false #-ParentId $obj.Id

    #foreach($device in $devices)
    #{
    #    ProcessDevice $device $obj
    #}
}

function ProcessDevice($device, $parent)
{
    AddObject $device $parent
}

function AddObject($obj, $parent)
{
    $parentName = $null
    $parentId = $null

    if($parent -ne $null)
    {
        $parentName = $parent.Name
        $parentId = $parent.Id
    } 

    Write-Host $obj.Name

    return [PSCustomObject]@{
        Name = $obj.Name
        Id = $obj.Id
        ParentName = $parentName
        ParentId = $parent.Id
        Type = $obj.Type
        NewId = -1
        NewParentId = -1
    }
}

$probe = Get-Probe -Id 90518

$result = ProcessContainer $probe

$result | ft

goprtg 2

foreach($thing in $result)
{
    if($thing.Type -eq "Probe")
    {
        $existingProbe = Get-Probe -Name $thing.Name
        if($existingProbe -ne $null)
        {
            $thing.NewId = $existingProbe.Id
            $thing.ParentId = $existingProbe.ParentId
        }
    }
    elseif($thing.Type -eq "Group")
    {
        $parent = ($result | Where-Object { $_.Id -eq $thing.ParentId } | select -First 1)

        if($parent -ne $null)
        {
            $newItem = $null
            if($parent.Type -eq "Probe")
            {
                Write-Host "Trying to find Probe: $($parent.NewId)"
                $newItem = Get-Probe -Id $parent.NewId
            }
            elseif($parent.Type -eq "Group")
            {
                Write-Host "Trying to find Group: $($parent.NewId)"
                $newItem = Get-Group -Id $parent.NewId
            }

            if($newItem -ne $null)
            {
                Write-Host "Found "
                $g = ($newItem | Get-Group $thing.Name);

                if($g -ne $null)
                {
                    $thing.NewId = $g.Id
                    $thing.NewParentId = $newItem.Id
                }
                else
                {
                    Write-Host " - Creating new group"
                    $newGroup = $newItem | Add-Group -Name $thing.Name
                    if($newGroup -ne $null)
                    {
                        $thing.NewId = $newGroup.Id

                        $thing.NewParentId = $newItem.Id    
                    }
                }
            }
        }
    }
}

[lordmilko]

Hi @wargal91,

You can theoretically achieve this with the tools PrtgAPI gives you; how complex this task will be will ultimately depend on your requirements

[wargal91]

Hi @wargal91,

You can theoretically achieve this with the tools PrtgAPI gives you; how complex this task will be will ultimately depend on your requirements

Yes I am trying to see what I can achieve. I asked to see if someone else has done it in the past or can assist to a certain extent.

[lordmilko]

Several years ago I started developing an infrastructure as code solution for PrtgAPI that would enable this (and is described on the wiki) however this has grown to such an extreme level of complexity it has been placed on hold for some time. This solution remains unreleased and nowhere near production ready.

I would advise that if you are not confident in your programming abilities this task is likely not going to be viable

[ArneBrucker]

Thank you for your outstanding project, @lordmilko. The solution to read and create the tree structure through the API seems quite intricate at this point. Wouldn't it have been simpler for the outcome to migrate the entire Core Probe (using the desktop app) and then delete the devices including sensors via script in the new instance? @wargal91

[wargal91]

I thought of this idea due to the fact that I have several core network devices that I intend to monitor via two different PRTG instances concurrently. Reason being that in case the first instance suffers an issue and I need to shift it to another instance (through a BCP plan which I have implemented using Windows Storage Replica), I need these important devices to be still monitored using the other seperate core server using its separate existing license.