DustinGrau-PSC
commented
1 year ago The server (monitor) should be able to accept data from multiple instances, even from the same server. The data should be sent from 2 distinct ports since that is how PAS instances are distinguished internally.
To debug this, we're going to need a COMPLETE rundown of every command you issued to set up the instances--this includes deploying the application code which supports the advanced monitoring, enabling the health metrics, starting the instances, and most importantly starting the monitoring on each instance. Create 2 documents, one for each of your PAS instances and collect the command run and any output--we need to see if there's any obvious key differences between the enablement processes for these by diff'ing the 2 files.
If the enablement processes are good, then we can move on to the collection side. In all cases there are logging.config options which can be altered--changing the logging-level attributed to the monitoring (collection or parsing) code to TRACE will generate much more detail including dumping of the JSON files being transferred. That's a last-resort as it's obviously going to be quite noisy unless we know exactly what to look for.
c3rberus
Looking at deploying the "monitoring" side of iceberg in production. The collector and monitoring application is deployed on the same machine (OE 12.2.12, AIX).
I can successfully deploy the Monitoring/Collector PASOE instance/database, and the Monitoring/Application code without issues, enabling metrics (pulse). Using default ports, I can get to the web UI and see monitoring data.
The moment I go to deploy Monitoring/Application to a 2nd PASOE instance (server has multiple PASOE instances), I get errors and there are no metrics being reported in the UI for the 2nd instance.
[2023-08-19T10:39:27.235-07:00] B.Intake | INFO - Health data received from production @ http://127.0.0.1:15605 [2023-08-19T10:39:27.236-07:00] B.Intake | ERROR - Error Parsing HealthCheck Dataset for production @ http://127.0.0.1:15605: Error parsing JSON: expected brace, but found bracket. (15358) [2023-08-19T10:39:27.238-07:00] B.Intake | ERROR - Error Processing Health Data: Unable to read HealthCheck JSON into dataset for further processing.The monitored PASOE instances are configured with identical settings other than the name/ports.
If I wipe everything and start over, this time deploying Monitoring/Application to the instance that was reporting errors processing health data FIRST - it works without any issues. Adding the 2nd instance (that previously worked when deployed to first), starts to experience the same errors.
Seems like the moment we go to add a 2nd instance to be monitored, the 2nd instance reports errors and no metrics are generated.
When enabling TRACE logging, and reviewing the monitor/temp folder, there is a bunch of .json files that get generated that have references to the FIRST instance that monitoring is deployed on, but no reference to the SECOND instance that has errors.
Any idea what we could do to debug this?
I suppose one could create multiple collector/monitor instances to potentially work around this, but it seems like the solution was built to support pulling in metrics from multiple PASOE instances.