Cacti 1.2.2: Multi Poller: unclear approach to shared configuration between Central and Remote Pollers

[stefanbrudny]

Describe the bug Configuration is completely unreadable and unreliable:

• Some parts are unified in the background between applications, while they should not be (spine config, paths, permissions etc.)
• Some are not (nobody knows which ones, but for sure one can select differnt poller or poller options - these are not, to my surprise, rsynced).
• poller and other files are rsynced between servers as well...

Additional context

This is either for fixing or documenting at least. I prefer proper design and skip documentation. My use case is advanced and I could have used containers to separate resources, which would greatly avoid this issue. Scenario is interesting when one server needs to pool into same ipv4 networks.

[netniV]

Yes, I have myself struggled between what is shared and what is not because of the way it was written many years ago. Generally speaking, I am sure that all php files and settings are replicated which includes paths which I suggested previously should have been suppressed but when we tried to suppress file paths to allow different arch's of remote collectors, it lead to unexpected issues so had to be reverted.

It was always designed that the remote collectors be on the same OS as the main server. That may be something we can review during the 1.3 development that is going to start soon but there's a big list of desired features.

I even mentioned that on an issue another user raised: https://github.com/Cacti/cacti/issues/2154

[cigamit]

I think this is a documentation issue, but if the inline description help is not sufficient, we would appreciate a pull request that has more consistent text.

[stefanbrudny]

What does it mean "all php files"? I suppose not really all, as config.php is not replicated. If really nearly all, then when? on poller sync? every spine session? etc.

I see it as an issue of normal severity, though it begs for documentation. User is not able to read about this (at least I couldn't, but I admit I was through only the current docs, not a forum this time) and he can really be surprised, when using some non standard approaches.

Eg. does it all have effect in kubernetes deployments, where services are very clearly exposed and some ngixes are build to glue everything up together.

[stefanbrudny]

There is also one nasty issue with sharing the config: I currently use 3 pollers on same host, no containers etc, just different network contexts.

And I have no real ability to separate cacti log. This results in sharing the 1 log in the interface of all Cactis, instead of having nice 3 logs from 3 pollers. Workarount is to look at some debug poller logs in console.

[netniV]

Cacti was never designed to run as separate instances on the same host. It may also be what is leading to some of your other issues.

[stefanbrudny]

Well, users see it differently.

Cacti always worked with multiple instances on one host. This design pattern has been dropped when remote pollers have been introduced. Users can accept this approach, as containers are here to stay.

As for my tests, it was trial by fire. Nobody explained this in any docs. Neverthless, no more items left in pipeline, all kinda works in this construction.

[netniV]

You can have multiple instances of Cacti (I have at least five on my development box), but not multiple instances on the same host talking to each other. In general, especailly when scaling to large systems, most users are actually setting up multiple pollers to be on different hosts in a distributed fashion to maximise the resources available to each cacti instance and be closer to the remote notes to be monitored.

Agreed, the documentation does not really do much to explain the poller architecture. We could really use a technical writer for that. Santa may one day deliver, and I have hope because we have had a few documentation updates from various users via pull requests. More is always needed though.

[stefanbrudny]

Historically, people always wanted to split to multiple pollers, I myself wanted, mainly for the reasons:

1. current spine inefficiency in handling handling downed hosts and poller timing out (my "favourite" deficency). Easiest way is to switch to eg 2 or 5 minutes polling time or split to several spines (and splitting I never succeeeded).

BTW: the best way to overcome this problem is to equip Cacti with separate poller detecting downed hosts and poll only the ones which are up. Spine is too busy and precious user, Cacti cannot waste its time for downed hosts. I wrote a bunch of shell scripts to make myself to not end in pain.

2. IOPS. Boost have greatly reduced the need, but a raid 10 composed of 4 SSDs combined with structured RRA with some glusterfs was also very effective approach. Sometimes, separate DB allows to decrease IOPS.

3. CPU load - mysql and spine shall be separated, whenever needed.

So - frankly, i see current design as bloated, as remote poller spine doesn't really need separate DB and nginx and php. I always thought remote poller is going to be an more aware spine, which has only a range of hosts to care about. Http and php is a bloatware here, I could accept DB (as I would still be using a central one and shared with Cacti).

Since Cacti permissions options and domains separation are acceptable, I would vote for dropping this remote poller redundancy GUI at all and focus on improvements within central cacti.

Spine seems somewhat ready with the range options and with current approach of copying of its data from main db to theirs db (I am trying to understand how it works).

This would be smart for small team. When the product is smart, Santa is going to come.

[cigamit]

We have been discussing the issue of multiple architectures and multiple paths. We have narrowed down to two elements that will need to be updated to support that.

1) The table poller_item currently includes a full path for scripts and rrdfiles. We would have to change that to a relative paths. 2) The settings sub-tab 'Paths' would have to write only to the local configuration. '%path%' variables are already not replicated, but we would have to change the settings page to only update the local instance when making these changes as well.

However, before we make these changes, we need feedback from people like @paulgevers, @mortenstevens , @DavidLiedke, and @BSOD2600 who maintain packages for us, and of course @netniV who continues to take on more and more architectural control with the tool per plan.

So, let's continue the discussion all.

[cigamit]

One additional comment related to the 'bloat' reference in another ticket. We chose this approach very thoughtfully. Earlier implementations of a multiple data collector architectures were only half solutions that did not cover all scenarios quite as well. We elected this approach as it was:

1) Simpler to install, 2) Remote collectors behave more like local Cacti servers, 3) Easier to support, if only marginally. 4) Easier to support things like proxies, nat's, firewalls etc. 5) Require only two open ports to operate (MySQL, HTTP) and are native protocols, we did not have to invent one.

When we first rolled this out, we were looking for fault tolerance only, and did not give much thought to scalability. Starting with Cacti 1.2.0, a side effect of this second generation was scalability as well as resiliency. So, we are pretty happy right now as to where it's at.

Our next critical move is to expose a SQL compatible interface to the RRDfiles that are Cacti schema aware. Again, we have given much thought to this. The RRDtool storage engine is a good one, but we need to have more modern access methods to the data that DBA's and maintainers of tools such as Grafana, Splunk, and Elastic can query it from.

Cacti seeks not to be the end all tool for everyone, just a convenient one for people to access data from, and of course as our founder @iankberry once said, "Cacti is first and foremost a graphing tool and should remain that". From that, and from his initial vision, we can not loose sight that it should be easy to install, configure and use.

Over the years, we have moved the "graphing" bar quite a bit from Ian's initial vision, and it has been a difficult migration not to capsize this thing. But we are doing the best we can. Honestly, we are still a few years behind the rest of the monitoring world for various reasons, but we keep pushing on with help of the remainder of the Cacti community.

Thanks for listening.

[stefanbrudny]

Hi there, holidays are coming and I thought why not to document sth of cacti?

I was thinking about:

1. a guide for running small scale cacti (as you know cacti doesn't scalle for me above small scale but lt's leave rants aside).
2. creating VM with Alpine and single docker for Cacti?
3. drawing arch diagram with multi poller and other stuff?
4. drawing basic diagram for Cacti with all communication and with some weight indicators (users do not know from where the load comes in, when spine dies, when ss is out etc)
5. writing couple of change requests streamlined for having cacti work at medium scale?
6. trying to discuss current ultimate failures in design, such as inability of detection of hosts down, how to deal with that?
7. What is needed - all except coding?

What I can offer is excellent orientation in management and configuration systems, detailed FCAPS+++ stuff knowledge, moderate Cacti skill (I would say fluency, but I do not read the code) and fluency in all fields required.

Heck, I could even startup with cacti services, if it gets in shape.

So do I, and where do I start?...

[TheWitness]

@stefanbrudny, reach out to me directly @ thewitness at cacti dot net.

[bmfmancini]

I'm in

I really want to help with the documentation and revamping something I have some ideas I'd like to share !

On Fri., Jan. 24, 2020, 7:57 a.m. South Park Neo, notifications@github.com wrote:

@stefanbrudny https://github.com/stefanbrudny, reach out to me directly @ thewitness at cacti dot net.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Cacti/cacti/issues/2559?email_source=notifications&email_token=ADGEXTEEPRJJ54F5S2XHFSLQ7LQVDA5CNFSM4HAKOG7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ2WSKY#issuecomment-578119979, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGEXTGFB6NCZ67NIE45BJLQ7LQVDANCNFSM4HAKOG7A .

[netniV]

I really wish people would stop using Cacti, so I can get some free time back ... just kidding ... it's nice to see even documentation getting looked at and improved, not just the code.

[bmfmancini]

Hahaha but I think for docs we should revive docs.cacti.net

I would like to also publish a vm with cacti setup like ez cacti was to help beginners or someone who wants to rapidly start things like that

Maybe we could do a google hangout session or something

On Fri., Jan. 24, 2020, 8:09 a.m. Mark Brugnoli-Vinten, < notifications@github.com> wrote:

I really wish people would stop using Cacti, so I can get some free time back ... just kidding ... it's nice to see even documentation getting looked at and improved, not just the code.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Cacti/cacti/issues/2559?email_source=notifications&email_token=ADGEXTE6XAG3EV6M4LMBZC3Q7LSCJA5CNFSM4HAKOG7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ2XOYI#issuecomment-578123617, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGEXTCZADAFUWIBGT2GKJDQ7LSCJANCNFSM4HAKOG7A .

[TheWitness]

So, I have a number of design docs, but they are all written in PPT. I could enhance then with what I know about the 1.2.x design, but my preferred authoring tool is PPT. Who can convert them to something that can be first retained, and then second exported for the documentation site?

[bmfmancini]

Maybe ppt to pdf?

As long as it's not Md lol

On Fri., Jan. 24, 2020, 8:29 a.m. South Park Neo, notifications@github.com wrote:

So, I have a number of design docs, but they are all written in PPT. I could enhance then with what I know about the 1.2.x design, but my preferred authoring tool is PPT. Who can convert them to something that can be first retained, and then second exported for the documentation site?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Cacti/cacti/issues/2559?email_source=notifications&email_token=ADGEXTHBQDO5ULHYGZ5ESB3Q7LUJ7A5CNFSM4HAKOG7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ2Y7KQ#issuecomment-578129834, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGEXTEQLTA6M5RJHSBLBGDQ7LUJ7ANCNFSM4HAKOG7A .

[TheWitness]

Can this issue be moved to doco repo?

[netniV]

@TheWitness if you send them to me, I should be able to resolve that.

[netniV]

Can this issue be moved to doco repo?

No, coz I already did it ## clicks fingers ##

[cigamit]

Ha, you beat me to it.

[netniV]

@bmfmancini it's @cigamit you want for that. When he had a lot more free time, he used to produce CactiEZ 👍

There are a few people who have been working on Docker images, though so far, I don't know of anyone whose stuck it out long term so far aside from our own @cigamit but he's a very busy man now.

[TheWitness]

Okay, almost a year has gone by and no activity on this. I think the separation of settings issue was resolved in Cacti 1.2 now at least the latest releases post 1.2.10 anyway. Some of the other documentation issues remain, of which @bmfmancini has done some nice video's, but we need more. I like the GlusterFS/Galera setup recommendation, in fact, I'm using it now. We should have a section on a Highly Available setup including not only Remote Data Collectors, but with the two aforementioned topics.

Who really has time to work on this?

[bmfmancini]

I got more videos coming soon on different ha setups I've been working with

I've offered in the past to provide the video files so cacti can have a official YouTube page I know netniv was conversd about submitting videos to the doc repo that were not owned by the cacti team

I think you guys should really consider opening up an account I would be willing to donate all of my past and upcoming videos

On Sat., Jan. 2, 2021, 10:11 TheWitness, notifications@github.com wrote:

Okay, almost a year has gone by and no activity on this. I think the separation of settings issue was resolved in Cacti 1.2 now at least the latest releases post 1.2.10 anyway. Some of the other documentation issues remain, of which @bmfmancini https://github.com/bmfmancini has done some nice video's, but we need more. I like the GlusterFS/Galera setup recommendation, in fact, I'm using it now. We should have a section on a Highly Available setup including not only Remote Data Collectors, but with the two aforementioned topics.

Who really has time to work on this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cacti/documentation/issues/66#issuecomment-753485086, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGEXTHTGMWRQIUJIJA4WHDSX4ZSJANCNFSM4KLGG4EA .

[TheWitness]

Yea, I'm not too happy with YouTube or Google at the current time though. It's a love hate relationship.

[bmfmancini]

How about vimeo?

On Sat., Jan. 2, 2021, 10:23 TheWitness, notifications@github.com wrote:

Yea, I'm not too happy with YouTube or Google at the current time though. It's a love hate relationship.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cacti/documentation/issues/66#issuecomment-753486273, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGEXTAIPTFK2NOQQCKMCPLSX426XANCNFSM4KLGG4EA .

[TheWitness]

I created a The Cacti Group channel on YouTube just now. I'll add you as a manager, email me the email address you want me to use.

[bmfmancini]

Very cool just emailed you now

[TheWitness]

Going to close this for the moment. It's there eternally of course.