ELK-based Analytics Support

[IsankaSR]

The product IS has the ELK integration capability developed via an intern incubator project [1]. Need to fine-tune the existing feature-set further and ship ELK Analytics support for product IS with the 5.12.0 release. Following are the sub tasks related to this effort;

• [x] Setting up the environment

  • Required tools for ELK stack.
  • Deployment guidelines including ELK and IS configurations.

• [x] Try out and analyze all the default templates implemented in [1] for the analytics dashboard.

  • Identify the issues when importing default templates and fix them.

• [x] Evaluate the feasibility of porting all of the existing features of IS analytics to ELK analytics.

  • In contrast to IS Analytics dashboard, implement any missing features, panels/widgets to the dashboards.

• [x] Action plan for subscription features

  • If the required features are beyond ELK free tier's listed features, we should provide a different strategy to customers to cater for those features.

• [x] Documentation

  • Environment Setup
  • Integration

Timeline for this effort : https://docs.google.com/spreadsheets/d/1hGJnH2yctkX4iC9olEgNIEbnzNHN4x3FgKDSHCI6vGY/edit?usp=sharing

[IsankaSR]

1. Setting up the environment

ELK integration incubator project was completed back in December of 2020, and since then, there have been significant changes for the ELK stack. A major version(8.0.0) has also been released in feb mid, it has major installation changes.

ELK version 8.0 requires Java 17 or higher support. Setting up both IS and ELK in the same environment makes some complications. We switched our samples to Docker and Docker-Compose to simulate a comprehensive deployment example. With this approach, we can isolate different components and easily update our dashboards for different upcoming ELK versions.

ELK environment and IS configuration guide: https://docs.google.com/document/d/1gkz--rE97Cw_PAOGYm8JGmuKPWQevuTascgAcqLSYbI/edit?usp=sharing

[IsankaSR]

2. Try out and analyze ELK analytics dashboards

We have the index templates and save objects implemented in [1] for analytics dashbords. When trying out them, we've identified few issues due to version incompatabilities.

• [x] Version incompatible issues in index templates: PRs https://github.com/wso2-incubator/identity-elk-integration/pull/3

• [x] HTTP input plugin install issues:

  This plugin had some issues related to it's Gem dependecies when working with the latest version. The issue is temporarly fixed but need to further investigate how it affect templates. Issue: https://github.com/elastic/logstash/issues/13777

  Tempory fix: https://github.com/elastic/logstash/issues/13777#issuecomment-1043141969

• [x] Version incompatible issues in Logstash pipeline

Fixed all the issues related to dashboards. You can try out dashboard following this doc: https://docs.google.com/document/d/1gkz--rE97Cw_PAOGYm8JGmuKPWQevuTascgAcqLSYbI/edit?usp=sharing

[IsankaSR]

3. Evaluate the feasibility of porting all of the existing features of IS analytics to ELK analytics.

Analysis : https://docs.google.com/document/d/1IRMpIIil1xTaEvucqyWqf-DYVanHn-IlGjGq5Feqdgw/edit?usp=sharing This analysis includes how we port IS analytics features to ELK analytics. Please read the document.

When trying out the features, identified some issues related to the dashboard panels.

• activeSessionCount does not work in session dashoboard: Had to add extra configuration to the toml and and update the overallsession stream version to 1.0.1
• Document tables are not working in dashboards. Enhanced data table plugin is used for this and its not supported for the latest version of ELK. Find an alternative way : Used default document table provided by Kibana, It doesn't include some advance features(ex: search, pagination etc). Compatible Enhanced data table plugin will be released soon, then we can apply the plugin and enhance the document table"
• Maps panel does not work properly : Had to add some changes to index templates

Fixed above issues and ensure the stability of dashboards.

Missing features/panels:

• Alerting feature: suspicious login alerts and long session alert
• Auth dashboard panels: Loging attempts by username, logging attempts by userstore
• Session dashboard panels: active session count

Added above missing features and dashboards panels.

[IsankaSR]

[Progress Update] ELK Analytics Integration

Meeting Notes - 21/03/2022 Participants - @IsuraD , @omindu , @avarjana and @IsankaSR

This meeting was held to dicuss the analysis of porting all the existing features of IS Analytics to ELK analytics. Here we presented how we port IS analytics features to ELK analytics dashboards. In IS analytics, following are the 7 dashboard pages:

• Overall login attempts summary
• Local login attempts summary
• Federated login attempts summary
• Login session
• Alert Type
• Suspicious login alerts
• Long session alert

We concluded above dashboard pages to 3 dashboards in ELK analytics.

• Auth dashboard
  • Includes the Overall login attempts summary.
  • Local login attempts summary can be seen by filtering identity provider by LOCAL(resident IDP name).
  • Federated login attempts summary can be seen by filtering identity provider type by FEDERATED.
• Session dashboard
  • Includes the Login session
• Alert dashboard
  • Includes the both Suspicious login alerts and Long session alert

Here are the main talking points we discussed:

• Add a different name mapping for resident IDP instead of LOCAL.
• ELK provides different alerting methods.In addition to the alert dashboard, decided to use Alert Rules with Connectors approach also. This can be used to receive notifications for the same events which can be viewed from the alert dashboard. This is a push-based approach where the alert is sent to a given set of connectors such as email, Jira, slack etc.
• Add single sign-on with WSO2 IS using SAML/OIDC.
• Currently, all the files are in an incubator repository-https://github.com/wso2-incubator/identity-elk-integration. Suggested two repositories to transfer.

  
    Organization - wso2-extensions```
    or
    ```Transfer files into sample-is```
  Send an email to take the opinions from the team

[piraveena]

@IsankaSR / @Avarjana close this issue if all tasks are done

[IsankaSR]

All the tasks are completed. Hence closing this issue.