Integration of script with monitoring dashboard

[chinmoy12c]

Currently, the monitoring script needs to be changed to integrate with the monitoring dashboard seamlessly.

End Goal:

An independent containerized image that provides access to

• [x] Logs of services
• [x] Real Time grepping of specific logs statements
• [ ] Cassandra data
• [ ] Telemetry Data

All Logs Data

A complete dump of all the logs from all the services must be stored. The logs would be segregated into separate daily files. The following queries can be done while fetching logs -

• [x] Log dump for specific days.
• [x] Log dump for a specific number of lines.

The overall flow of the system would be as follows.

Real-Time analysis

This would enable specific log statements to be grepped and counted in real time (just like axiom). Before triggering a bot, the real-time analysis needs to be enabled or switched on, which would start monitoring incoming logs and start creating a dump of analyzed data in a file. This data would include the following information:

• [x] Overview data needed for PT
• [x] Grepped statement count of all services
• [ ] Kafka topic counts

The configuration of greppable statements would be in the following format:

services = [
    {
        'service_name': 'UCI API',
        'service_container': 'uci-api-service',
        'match_statements': {
            'BotServiceError': 'ERROR [BotService]',
            'DeviceAdded': 'Added device to registry',
            'ResolveUser': 'ServiceService::resolve: Resolving users'
        }
    },
    {
        'service_name': 'Broadcast Transformer',
        'service_container': 'docker-deploy-prod_broadcast-transformer_1',
        'match_statements': {
            'Kafka Push to outbound': 'Kafka::Send: Pushed to topic process-outbound',
            'Broadcast Processed': 'Notification processed by broadcast-transformer:',
        }
    },
]

Log Format Spec

This is the format that would be followed while creation of logs. This spec allows UCI-API to read data without communicating directly with the LogBase container.

Endpoints

• Fetch Service log GET /admin/monitoring/logs/[SERVICE_NAME]?lines=[NUMBER OF LINES]&date=dd_mm_yyyy
• Trigger Real-Time data analysis POST /admin/monitoring/realTime/start
• Fetch Real-Time data GET /admin/monitoring/realtime?date=ss_mi_hh_dd_mm_yyyy
• Stop Real-Time data analysis POST /admin/monitoring/realtime/stop
• Get available realtime data files GET /admin/monitoring/realtime/available
• Fetch Overview data GET /admin/monitoring/overview?date=ss_mi_hh_dd_mm_yyyy
• Get download link for log GET /admin/monitoring/logs/[SERVICE_NAME]/download/log?date=dd_mm_yyyy
• Get download link for error logs GET /admin/monitoring/logs/[SERVICE_NAME]/download/error?date=dd_mm_yyyy

[chinmoy12c]

@karntrehan please review this

[karntrehan]

The logs would be segregated into separate daily files. This should be config driven. At a high load system I might need this hourly.

Log dump for a specific number of lines. What is the usecase for this?

The diagrams are helpful. Is log base going to be different layer altogether? Is it optional?

Overall this looks good!

[chinmoy12c]

The logs would be segregated into separate daily files. This should be config driven. At a high load system I might need this hourly.

Log dump for a specific number of lines. What is the usecase for this?

The diagrams are helpful. Is log base going to be different layer altogether? Is it optional?

Overall this looks good!

Yes, LogBase would be a separate service altogether which would be a simplistic and independent service. This is so that this component could also be used in other projects as an independent logger service.

[aakashyadav-kgp]

Dev done. Need to deploy and test on staging.

[karntrehan]

Awaiting some optimizations (https://github.com/samagra-comms/utils/issues/59) to be deployed before this goes out. Unit tests being written.