support for several instances of sync-app

[akurach]

Hello, the main idea is to add support to create a lot of sync containers for each source to sync it simultaneously. For example for now i have 5 vmware instances, 5 openstack and 5 oVirt. I want to create 15 docker containers for sync.

Why not it is not possible:

Every container will watch Netbox-synced tag and set all devices/vms/etc to orphaned status because source is unavailable.

Solution maybe create tag netbox-synced-%datasouce_name% and watch after this one?

Thx

[bb-Ricardo]

Hi,

The mein reason behind this is to run one sync with all sources in one job.

Usually infrastructure shares resources like subnetworks and VLANs. If you run it independently simultaneously then one instance has no idea about resources discovered in the other instances. Then you would most likely run into race conditions and errors because one instance creates/updates a resources the other instance also wants to change.

I could also add an config option like instance_tag which should solve your issue.

[akurach]

it just looks like when we have a lot of sources smthing goes wrong

04-21 09:06:42,231 - ERROR: Unfortunately updated item assigned_object_id for object 192.168.8.89/24 could not be fully resolved: <NBVMInterface instance 'vNIC 2 (dvs.VCDVSbeta-internal-net-581bc89d-ba0c-44c2-b21c-81a5e (d00in (615780c9-d11f-47c4-bda5-0be9655014c8))' at 4339062384>

lots of such errors at the end of sync

[akurach]

and a lot of devices are not synced at first run

[bb-Ricardo]

So this happens when you have them all in one settings file and do just one run?

[akurach]

Looks like local issue with vmware.. sorry.

Back to main question: Are there any possibilities to add option in common section like "orphan_unavaliable_source" = True/False True - if datasource is not in cfg file - mark as Orphan
False - if datasource is not in cfg file - do not mark as Orphan just sync configured.

[bb-Ricardo]

Hi, I could add something like

ignore_unknown_source_object_puning = True

would this help in your case?

[akurach]

sounds great.

[bb-Ricardo]

I just pushed a new commit to the branch development. Can you try it out and see if it works?

https://github.com/bb-Ricardo/netbox-sync/blob/72f6ee2904b932e6e3b90189b3e255bc640be07d/settings-example.ini#L74-L77

[bb-Ricardo]

any updates on that?

[akurach]

Sorry for long response, I'll check it this week.

[akurach]

everything works as a charm! thanks! :)

[rizlas]

Hi, this is looking promising and for me is a must have, in particular if you are in a docker based solution.

Why this is a must have?

As @akurach stated you can have multiple instances that also means different sources. Let's say you need to sync 5 vmware sources, before this feature that would means a huge ini setting file. With ignore_unknown_source_object_puning now we can have different setting file for each source (that's amazing).

What is not working right now?

Tagging of orphan seems bugged. Since I switched from a "multiple source setting file, single container architecture" to "dedicated setting file, multiple container architecture" netbox-sync stopped to find and tag orphan item for each source.

I just tested by setting to False ignore_unknown_source_object_puning and orphan tagging came back (without solving the main problem arised in this issue)

Quick representation of the current status:

ignore_unknown_source_object_puning status

----------------------------------------

Current behavior

Source_A        Source_B        Source_C

True            True            True

No orphan tagging

----------------------------------------

----------------------------------------

Tested behavior

Source_A        Source_B        Source_C

False            True            True

Tagged correctly item that are orphan in Source_A
Tagged also all the item in Source_B and Source_C as orphan (that was expected, not desired)

----------------------------------------

----------------------------------------

Desired behavior

Source_A        Source_B        Source_C

True            True            True

Tag orphan Source_A and do not consider item in Source_B and Source_C
Tag orphan Source_B and do not consider item in Source_A and Source_C
Tag orphan Source_C and do not consider item in Source_A and Source_B

----------------------------------------

Thank you

[rizlas]

@bb-Ricardo just a bump message. I don't know if you receive any notification for closed issues.

[bb-Ricardo]

Hey, I will have a look at it. I still not recommend using different sync scripts to split up the syncing.

Also in one of the next versions the configuration management will be eased up. It should support environment vars and yaml input.

[rizlas]

Hi Ricardo, I remember that you said that to me, but as soon I saw this issue, I reworked my entire docker deployment 😅 The main problem arise with this condition https://github.com/bb-Ricardo/netbox-sync/blob/main/module/netbox/inventory.py#L314. A simple continue is not enough, the entire process of orphan tagging is skipped.

Maybe a condition like:

if this object has a tag that matches the current loaded source: orphan tagging is allowed.
Otherwise: object is not part of the loaded source so skip.

If I find some time, in the next few days, I will have a look at it and discuss with you a better solution. Thank you

[bb-Ricardo]

closing issue in favour of #273