[FEATURE] OpenSearch Synthetics

[ps48]

Synthetics Design Document

1. Overview
2. Motivation
3. How is it different from other plugins?
4. Requirements
5. UI Mockups
6. Architecture
7. Miscellaneous
8. Data Model
9. OpenSearch/Plugins REST endpoints
10. Appendix
11. References

Code: https://github.com/opensearch-project/observability/tree/uptime

1. Overview

Synthetics is a new module in observability that enables users to monitor availability and response times of applications and services in real time. This tool provides the ability to understand availability and response time components of services and applications. Users can detect problems before they affect their end customers.

2. Motivation

Synthetics systems are useful for measuring stability, reliability and analysis of health on live systems. Continuous monitoring of micro-service based software systems is an essential component of observability. Synthetics opens up the door for infrastructure visibility by proactively pinging API endpoints. This can be considered as an auto-pilot extension to human observation, especially when merged with other OpenSearch capabilities namely: reporting, anomaly detection and alerting.

Synthetics can be utilized for the following use-cases:

1. Measure availability of apps or a services from a public endpoint, as well as specific components
2. Showcasing different response times from different locations around the world
3. Get historical view of logs over a period of time
4. Get live status view of apps and services

3. How is it different from other plugins?

• App Analytics module in observability is mainly responsible for aggregating logs, traces and metrics under one umbrella. The availability part in App Analytics home page comes from pre-indexed data queried over with a PPL Query. Whereas, Synthetics component fetches it from an API request run at scheduled intervals. It drills down further into analysis of individual test-suites metrics including duration, requests over time and historical data.
• Alerting plugin notifies a user when data from one or more OpenSearch indices meets certain conditions. Unlike Synthetics, Alerting doesn’t monitor public endpoints and retain live/historic values of the responses.

4. Requirements

4.1 Functional Requirement

1. Users must be able to make endpoint API requests as scheduled jobs.
2. Users must be able to make endpoint API requests on Demand
3. Users should be able to create synthetics test-suites for public endpoints.
4. Users should be able to create test-suites for OpenSearch, OpenSearch Dashboards and its plugins using REST endpoints.

4.1 Dashboards Observability

4.1.1 Synthetics Home

1. Users should be able to view summary graphs of test-suites availability and pings over time.
2. Users should see a history table of all recent requests made from all test-suites.
3. Users should be able to filter the graphs and table on home page with a PPL search bar.
4. Users should be able to filter home page with a date filter.
5. Users should be able to investigate deeper into a test-suite with links to related events, visualizations, panels and App Analytics page.

4.1.2 Test-Suite View

1. Users should see test-suite summary table with locations, URL, protocol and tags.
2. Users should be able to view 3 graphs: test-suite duration, pings over time and location(table/map) based test-suites.
3. Users should be able to see a history table with recent pings (1000), each row should expand with the detailed request view of response headers, body and timing chart.
4. Users should be able filter the history table by location and sort table by response duration, error message or response code.

4.1.3 Add & Configure Test-Suite

1. Users should be able to add new test-suites to the synthetics plugin.
2. Users should be able to process response with PPL queries, to validate the app and service being up.
3. Users should be able to configure a test-suite with the options below:
   1. Id
   2. Name
   3. Protocol/Type of endpoint: [http, tcp, imcp]
   4. App Id
   5. Notebook Id
   6. Saved Query Id
   7. Saved Visualization Id
   8. Panel Id
   9. Enabled bool
   10. Scheduler Job Id
   11. ipv4 bool
   12. ipv6 bool
   13. ip resolver mode
   14. timeout
   15. tags
   16. keep_null
   17. Type specific configuration:
       1. ICMP:
          1. Hosts Array
          2. Wait duration
       2. TCP:
          1. hosts
          2. ports
          3. check
          4. proxy url
          5. proxy resolver
          6. ssl
             1. certificate authorities
             2. certificate [for client SSL auth]
             3. key [for client SSL auth]
             4. supported protocols e.g ["TLSv1.0", "TLSv1.1", "TLSv1.2"]
             5. key-passphrase
             6. verification-mode: [full, strict, verification, none]
       3. HTTP
          1. hosts
          2. max redirects
          3. proxy url
          4. proxy headers
          5. username
          6. password
          7. ssl [similar to tcp]
          8. index headers
          9. index response: [on_error, never, always] controls when to index the response
          10. check: [method/status, headers, body] applies for both request and response

Note: Username and passwords are stored for accessing an endpoint with HTTP basic authentication.

More detailed HTTP request:

# Synthetics Client in Opensearch-Observability

## Test-Suite Configuration:

Taking a look at `sample_testsuite.yml`, we can see:
---
  name: "Sample Test Suite"
  type: "http"
  appId: ""
  notebookId: ""
  savedQueryId: ""
  savedVisualizationId: ""
  operationalPanelId: ""
  ipv4: true
  ipv6: true
  resolverMode: "all"
  timeoutSeconds: 16
  tags:
  - "news"
  - "apis"
  keepNull: true
  hosts:
  - "https://opensearch.org"
  - "https://opensearch.org/synthetics"
  - "https://github.com/opensearch-project"
  maxRedirects: 1
  proxyURL: ""
  proxyHeaders: {}
  username: ""
  password: ""
  ssl:
    enabled: false
    default: true
    certificateAuthorities: CAINFO
    certificate: ""
    key: ""
    supportedProtocols: "TLSv1.1" # https://curl.se/libcurl/c/CURLOPT_SSLVERSION.html
    keyPassphrase: ""
    ecAlgorithmCurves: ""
    falseStart: false # https://curl.se/libcurl/c/CURLOPT_SSL_FALSESTART.html
    cipherList: "" # https://curl.se/docs/ssl-ciphers.html
    verifyHost: true # https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html
    verifyPeer: true # https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html
  indexHeaders: true
  indexResponse: "always"
  request:
    method: "GET"
    headers: {'Accept-Encoding': None, 'Content-Encoding':'gzip'}
    body: ""
    json: {}
  response:
    status:
    - 200
    - 301
    headers: {}
    body: {}
  scheduler:
    scheduleType: "interval"
    schedule:
      period: 20
      unit: "seconds"

* The `name` field is the name you specify for this particular Test-Suite, has to be alpha-numberic.
* `type` is the protocol for this Test-Suite, you can choose from either http, tcp, or icmp.
* `appId`, `notebookId`, `savedQueryId`, `savedVisualizationId`, and `operationalPanelId` are all neccessary ids for if you want to connect this Test-Suite with other OpenSearch components.
* `ipv4` is a boolean value specifying if the request should be handled through IPv4
* `ipv6` is a boolean value specifying if the request should be handled through IPv6
* `resolverMode` takes a string that specifies the resolver mode
* `timeoutSeconds` is the amount of time (in seconds) before any connection should time out, takes a number
* `keepNull` is about whether fields with no values should be kept null, takes a boolean
* `hosts` can be a list of endpoints that are in the Test-Suite and will be pinged based on the schedule
* `maxRedirects` is the max number of redirects the connection should go through before being cut off.
* `proxyURL` takes a url that can be the proxy url
* `proxyHeaders` are the headers that would go with the proxy url
* `username` is a string that can be attached onto the request that will act as a username
* `password` is a string that can be attached onto the request that will act as a password
* `ssl` is where various ssl certificates' paths can be specified and used:
  - `enabled` by default (also when ssl is not specified) will not attempt to use SSL. However, if true is specified, then SSL is required for all communication or there will be an ERROR thrown 
  - `default` is an option, where if true, will attempt to find and use CA bundles contained in the `certifi` library. If false, a CA path is needed in `certificateAuthorities`.
  - `certificateAuthorities` is a needed file path leading to a valid CA cert bundle for SSL unless `verifyPeer` is false, in which case an empty file can be given.
  - `certificate` should be a file path leading to a certificate with the type "PEM".
  - `key` should be a file path to a private key with type "PEM". 
  - `keyPassphrase` would be the password required to use `key`
  - `supportedProtocols` is the protocol version range for the SSL/TLS handshake to use. The input protocol verison will be the minimum protocol used, where later versions can be used as well. For exmaple, if the value here is "TLSv1.1", then TLSv1.1, TLSv1.2, and TLSv1.3 would be available for use. The default value is "TLSv1.0". This method of input is in use to keep behavior of TLS libraries consistent.
  - `ecAlgorithmCurves` needs input as a colon delimited list of EC algorithms. 
  - `falseStart` is a boolean with a default value of false, where when true false start would be in use. False start is a mode where a TLS client saves a round trip on a full handshake by sending data before verifying the server message.
  - `cipherList` needs to be a colon delimited string of one or more ciphers. Default follows an internal configuration.
  - `verifyHost`, when true, verifys that the connecting server has the same name in the certificate and host name, and will fail if otherwise. Default is true.
  - `verifyPeer` checks if the server certificate is authentic. This authenticity is based in the CA certificates supplied. Default is true. Disabling makes communication insecure and allows for man-in-the-middle attacks.
* `indexHeaders` is a boolean that will index the headers if true
* `indexResponse` specifys whether an index response should return or not
* `request.method` is the protocol's method that should be used
* `request.headers` are the request headers that will be sent along with the request. Must be in JSON format.
* `request.body` is the body that will be sent along with the request
* `request.json` is a json that can be sent along with the request
* `response.status` are possible statuses that can result in an 'UP' status for a host.
* `response.headers` are headers that the response will check to have
* `response.body` is a body that the response will check to have
* `scheduler.schedule-type` is the type of schedule that should be used. Possible types are 'interval' and 'cron'
* `scheduler.schedule` can either have two settings based on what the schedule-type was:
  - example interval:

    period: 20
    unit: 

The period specifies the quantity of time between each interval (has to be a number) and the unit specifies the unit of that time (has to be one of "weeks, "days", "hours", "minutes", and "seconds"). The job will trigger once and then after each interval occurs. Documentation

• example cron:

  year: "*"
  month: "*"
  day: "*"
  week: "*"
  day_of_week: "*"
  hour: "*"
  minute: "*"
  second: "0,15,30,45"

  This is like 'cron' based scheduling, where the time in each component is the time in which the job triggers. There can either be a valid string put in or '*' can be used as a wildcard. So in the above example, the job will trigger at every minute on the 0th, 15th, 30th, and 45th second. Note that each component's default value is the wildcard so most components can be left out. Just looking up 'cron' will give more neccessary information on how to use this scheduler. Documentation

• The below configurations can be added to either interval or cron based scheduling.

  start_date: '2010-10-10 09:30:00'
  end_date: '2014-06-15 11:00:00'
  timezone: ''
  jitter: 120

4.1.4 Certificates

1. Users should be able to store certificates for TLS/SSL requests.
2. Users should be able to view a table of all certificates with their name, age, expiration date, issuer and SHA fingerprints.

4.1.5 Settings

1. Users should be able select certificate age threshold, to display certificate warning.
2. Users should be able to configure the auto-delete interval for deleting old endpoint responses stored in the observability index.

4.2 OpenSearch Observability

4.2.1 Observability Scheduler

1. Register scheduled jobs for running synthetics test-suites in the jobs-definition.
   1. Based on intervals
   2. Based on cron jobs
2. Use Endpoint Client APIs to run scheduled jobs at given interval/cron time.
3. Callback Endpoint Client after a scheduled job is run, to store the response and timings.

4.2.2 Endpoint Client

1. Allow users to make calls to ICMP, TCP and HTTP endpoints.
2. Endpoint client should be to make auth based requests with certificates and other SSL configurations.
3. Endpoint client should be to send a payload in request and validate received payload in response.
4. Endpoint client should be able to support SOCKS5 proxy endpoints.

4.2.3 Indexing Client

1. Query and store monitor configuration including endpoint details in .opensearch-observability index.
2. Query and store endpoint certificates in .opensearch-observability index.
3. Query and store all request information (headers, responses and timings) in a new observability-synthetics-logs index
4. Use the scheduler to auto-delete old response at regular intervals.

4.3 Optional requirements:

4.3.1 Plugin Integrations

1. Reporting needs a direct integration to the synthetics component, there is no “custom” way to use it like alerting and anomaly detection.
2. For Alerting & Anomaly Detection, users can configure alerts and detectors on the observability-synthetics-logs index. This index will store the logs generated by synthetics test-suites.

4.3.2 Reporting

1. Users should be able to generate reports on summary of synthetics stats for a given time period.
2. Users should be able to generate report for a particular test-suite.

4.3.3 Anomaly Detection

1. Users should be able add a detector to find anomaly in response time over a given period for a test-suite.
2. Users should be able to view/delete/edit added anomaly detectors.

4.3.4 Alerting

1. Users should be able to add and configure alerting monitors for:
   1. A down status of an endpoint
   2. A down status over a period of time
   3. Increase in response time over a threshold
2. Users should be able to view all the alerting monitors configured for synthetics
3. Users should be able to delete an alerting monitor.

6. Architecture

6.1 Architecture 1

6.1.1 OpenSearch-Observability

• Index client: As of OSS 1.2.0 observability backend plugin is already part of OpenSearch plugins. This backend has an indexing client to add/get/delete/update the observability index. We can extend this backend plugin to perform similar operations with new types of data model: syntheticsTestSuite, synthetics``Logs and syntheticsSettings

• Endpoint client: This will be a new component added to the observability backend. This component will be responsible to perform all the API requests and store the result to the index using index client APIs. The component will also validate the response using PPL processors.

  • PPL Processors: Once the endpoint response is fetched, the logs are indexed in observability-synthetics-logs . The user has the capability to run PPL queries on the request or response parameters, to check for values and validate if the pinged app or service is up. These PPL queries are called PPL processors. They’ll specifically run on each response of the endpoint. The other values added by PPL processor is as follows:
    • Reduce the number of exported fields
    • Add more metadata or metrics to the logs
    • Perform additional operations over response data
  • Client Requests: The Endpoint client should be able to make requests to:
    • External endpoints provided by users in synthetics test-suites.
    • REST endpoints exposed by OpenSearch Plugins.

• Job Scheduler: The observability plugin will have its own abstract component of the OpenSearch job scheduler SPI. This will enable other observability components to use its extensible interfaces to register a job and run those with appropriate callbacks on completion of the job. Later, synthetics test-suites will use these interfaces to run the pings to respective endpoints. Apart from the scheduled synthetics requests, an auto-deletion job should be scheduled as a daily cron job to delete logs below a given threshold. The log age threshold may be changed in the setting page of synthetics.

6.1.2 Dashboards-Observability

• Synthetics Test-Suite UI:
  • The UI component of synthetics test-suites will be integrated to the existing dashboards-observability plugin.
  • The graphs used in UI will use plotly library, similar to other module in observability.
  • The cumulative stats in history/summary will be generated with a PPL query over observability-synthetics-logs index.
• Plugin Usage & Integrations:
  • Alerting: Users can navigate to the Alerting plugin to create synthetics specific alerts. They can create a “custom alerting monitor” on observability index. Further then can create an “extraction query” as a trigger that meets conditions on a DSL query response.
  • Reporting: This integration will be similar to Notebooks, users can see reporting menu options if it is available. They should be also able to create reports through the reporting plugin.
  • Anomaly Detection: Similar to alerting, users can create custom anomaly detectors with observability index as a data source. Further, they can use a DSL query to parse syntheticsRequests and get a field e.g. response duration or even a collection of fields. Finally, users can configure their ML model over extracted fields.

Pros

• The plugin stays within the OpenSearch cluster realm.
• OpenSearch Job-scheduler is extended and re-used.

Cons

• The endpoint client making requests to public endpoints, may need lot of security changes to be used in VPC domains.

6.2 Architecture 2

6.2.1 OpenSearch-Observability

• Endpoint Client: In Architecture 2, the Endpoint Client is moved out of the OpenSearch cluster. It interacts with the OpenSearch using a client library (Python, Java, etc). The Endpoint client is polled by Job-Schedular based on set time intervals.
• Rest of the components remain same as Architecture 1.

6.2.2 Dashboards-Observability

• No Changes UI remains same as Architecture 1.
• Plugin integrations & usages remain same as Architecture 1.

Pros

• The endpoint client stays outside the OpenSearch realm and only pushes log data to the OpenSearch. The would be easier to work with, in VPC domains/domains with secure gateways.
• OpenSearch Job-scheduler is extended and re-used.

Cons

• The scheduler still polls the endpoint client with the test-suite meta-data. Therefore, need to make exception on gateway for the endpoint client.

6.3 Architecture 3 [Preferred]

6.3.1 OpenSearch-Observability

• Endpoint Client:
  • In Architecture 3, the endpoint client handles the work of job scheduler on its own.
  • The test-suite configurations would be stored at .yml files, instead of docs in the .observability-index.
  • There would be many instances of endpoint client working to push the log data to OpenSearch. The endpoint client would make the scheduled requests and store the logs in observability-synthetics-logs index.
• OpenSearch SPI Job-Scheduler is not used. Rest of the components remain same as Architecture 1.

6.3.2 Dashboards-Observability

• UI doesn’t have major changes here, it behaves similar to trace-analytics UI. UI reads the observability-synthetics-logs index and populates the pages.
• Plugin integrations & usages remain same as Architecture 1.

Pros

• The endpoint client works on a push mechanism. No need for two communication between OpenSearch and endpoint client
• No need for special security gateway changes.

Cons

• OpenSearch job-scheduler is not re-used.

7. Miscellaneous

7.1 FGAC for OpenSearch

• observability-synthetics-logs must adhere to all access control permissions
• PPL queries/calls made throughout the plugin must adhere to all access control permissions

7.2 Options for having location in Synthetics:

Some potential solutions:

1. Require users to provide their own Location requesting service with their own api key so that they can be responsible with how many requests they might make
2. Find some way to keep a database for location requesting in our own service as to allow users to make unlimited calls
3. Pay for it ourselves (not recommended)
4. Have a location table index, where IP addresses are prepopulated and the customer can just grab the index and do the IP-location conversion themselves. This can then be incorperated into any information/maps in Synthetics UI [preferred]

8. Data Model

8.1 syntheticsTestSuite (Architecture-1 [6.1])

• Stores synthetics test-suite configuration in .opensearch-observability index.

{eiifccugvrllerjcdgggkhfhkikcngueekfhrccbvvhn

    "syntheticsTestSuite": {
        "name": "Sample Test-Suite",
        "type": "http",
        "appId": "",
        "notebookId": "",
        "savedQueryId": "",
        "savedVisualizationId": "",
        "operationalPanelId": "",
        "enabled": true,
        "ipv4": true,
        "ipv6": true,
        "resolverMode": "all",
        "timeoutSeconds": 16,
        "tags": ["news", "apis"],
        "keep_null": true,
        "hosts": ["http://samplehost:8000"],
        "maxRedirects": 0,
        "proxyURL": "http://proxy.mydomain.com:3128",
        "proxyHeaders": {},
        "username": "",
        "password": "",
        "ssl": {},
        "indexHeaders": true,
        "indexResponse": "always",
        "check": {
            "request": {
                "method": "GET",
                "headers": {},
                "body": ""
            },
            "response": {
                "status": 200,
                "headers": {},
                "body": {}
            }
        }
    }
}

// Information passed to observability job scheduler, 
// while configuring a test-suite.
// Returned Job Scheduler Id is stored in the test-suite index. 
{ 
    "scheduler": {
        "schedule_type": "recurring",
        "schedule": {
            "interval": {
                "period": 12,
                "unit": "HOURS",
                "start_time": 1635463349333
            },
            "enabled": true,
            "enabled_time": 1635463349332
        }
    }
}

8.2 syntheticsTestSuite (Architecture-2 6.3)

• Stores synthetics test-suite configuration in synthetics-client

---
  name: "Sample Test Suite"
  type: "http"
  appId: ""
  notebookId: ""
  savedQueryId: ""
  savedVisualizationId: ""
  operationalPanelId: ""
  enabled: true
  ipv4: true
  ipv6: true
  resolverMode: "all"
  timeoutSeconds: 16
  tags:
  - "news"
  - "apis"
  keep_null: true
  hosts:
  - "http://opensearch.org"
  - "http://opensearch.org/synthetics"
  - "http://github.com/opensearch-project"
  maxRedirects: 0
  proxyURL: ""
  proxyHeaders: {}
  username: ""
  password: ""
  ssl: {}
  indexHeaders: true
  indexResponse: "always"
  request:
    method: "GET"
    headers: {'Accept-Encoding': None, 'Content-Encoding':'gzip'}
    body: ""
    json: {}
  response:
    status:
    - 200
    headers: {}
    body: {}
  scheduler:
    schedule-type: "interval"
    schedule:
      period: 20
      unit: "seconds"

8.3 syntheticsLogs

• Stores synthetics logs in observability-synthetics-logs index.

{
    "syntheticsLog": {
        "syntheticsSuiteId": "",
        "status": "UP",
        "type": "http",
        "URL": "http://samplehost:8000",
        "request": {
            "method": "GET",
            "headers": {},
            "body": ""
        },
        "response": {
            "status": 200,
            "headers": {},
            "body": {}
        },
        "startTime": 1635463349345,
        "endTime": 1635463349398,
        "dnsTimeMs": 100,
        "ConnectionTimeMs": 500,
        "sslTimeMs": 0, 
        "ttfbMs": 121,
        "downloadTimeMs": 2, 
        "contentSizeKB": 3.5
    }
}

[Optional] 8.4 syntheticsSettings

• Stores synthetics plugin configuration in .opensearch-observability index.

{
    "syntheticsSettings": {
        "certificateThresholdDays": 30, //UI will show a warning if a certificate is under the mentioned threshold  
        "logAutoDeletePeriod": 30, // Logs above this age will be auto deleted
        "logAutoDeleteUnit": "DAYS"
    }
}

9. OpenSearch/Plugins REST endpoints

9.1 Using pre-existing REST endpoints

9.1.1 Stats APIs

• OpenSearch and many of its plugins have pre-exposed REST APIs, on which the users can create synthetics test-suites. These APIs are exposed with FGAC controls. Below are some links to stats API docs:
  • Search Stats
  • Backpressure Stats
  • Node Stats
  • Alerting Stats
  • KNN Stats
  • Anomaly Detector Stats

9.1.2 Other miscellaneous APIs

• OpenSearch and OpenSearch Dashboards also expose other REST APIs to users like CAT, Index, Cluster, Search, Status, etc. Users can use these endpoints for creating synthetics test-suites. An easy way to digest the output of these APIs is to use PPL processors over the endpoint response.

9.2 Health Check endpoints

• Health Check endpoints will be one stop shop, to check health status of all OpenSearch & OpenSearch-Dashboard plugins. The API endpoints will provide users the capability to check each plugin and get their detailed health status. These health check endpoints can be used to create synthetics test-suites.
• Health Checks conceptualize two types of APIs
  • Startup Check APIs - This check makes sure that the plugin has initialized(after a start/restart), doesn’t have any internal issues and is ready to take incoming requests
  • Liveness check APIs - This check makes sure that the plugin is still running post startup, doesn’t have any internal issues and is ready to take incoming requests
• Synthetics test-suites can utilize the liveness check APIs. The response from liveness APIs can be checked with PPL processors.
• More details on health checks can be seen here: https://github.com/opensearch-project/opensearch-plugins/issues/74

10. Appendix

10.1 Alerting & Anomaly Detection

• Synthetics test-suites users with surely use the alerting and anomaly detection plugins, as their combined value is the real MVP. Therefore, to make it easy for initial users we can provide sample alerts and detectors with dummy synthetics test-suites.

10.2 Reporting

• Reporting needs a deeper integration with all the observability components. Hence, we can come up with an integration plan that works with all the components.

10.3 Synthetic monitoring

• Going further, the Endpoint client can be extended for monitoring APIs with synthetic testing. Chain HTTP requests and execute API calls sequentially with Multistep API tests

11. References

11.1 https://geekflare.com/monitor-website-uptime/

11.2 https://www.datadoghq.com/uptime-monitoring-tools/

11.3 https://cabotapp.com/

11.4 https://github.com/arachnys/cabot#single-service-overview

11.5 https://alyvix.com/learn/introduction.html

[ps48]

Synthetics Demo Video from @paulstn

https://user-images.githubusercontent.com/4348487/166390233-11dbe004-3408-4174-905b-e0fef43fb035.mov

[ps48]

Issues associated:

Synthetics client

1. opensearch-project/dashboards-observability#74
2. opensearch-project/dashboards-observability#73
3. opensearch-project/dashboards-observability#72
4. opensearch-project/dashboards-observability#71
5. opensearch-project/dashboards-observability#70
6. opensearch-project/dashboards-observability#69
7. opensearch-project/dashboards-observability#68
8. opensearch-project/dashboards-observability#67
9. opensearch-project/dashboards-observability#66
10. opensearch-project/dashboards-observability#65
11. opensearch-project/dashboards-observability#64

Synthetics UI

TODO: add work breakdown/ pending issues

[elfisher]

hey is this still tracking for 2.3?

[rafael-gumiero]

Any plan in which version this functionality will be included?

[ps48]

@rafael-gumiero we don't have this feature in our priority right now. We are open to guiding community contributions. cc: @anirudha @paulstn