Axway-API-Management-Plus / apigateway-openlogging-elk

Monitor API-Gateway realtime based on Elasticsearch (Elastic-Stack)
https://www.elastic.co/elastic-stack
Apache License 2.0
28 stars 16 forks source link

Adding support for JMS-Traffic #118

Closed cwiechmann closed 3 years ago

cwiechmann commented 3 years ago

User story It should be possible to use the API-Gateway Traffic-Monitor also for JMS-Traffic

Additional context For the JMS-Traffic-Search data must be returned as given in the example below:

{
  "processId": "",
  "data": [
    {
      "jmsMessageID": "ID:interface1-37293-1625392418519-7135:209:1:1:1",
      "jmsCorrelationID": "XXXX_UOV_PKCONF_2021_07_05_09_49_07_297.JSON",
      "jmsDestination": "queue://com.customer.APPLIC.msg.WWSWO.PROD.Cisco.OrderStatus_IQ",
      "jmsProviderURL": "local",
      "jmsDeliveryMode": 2,
      "jmsPriority": 4,
      "jmsReplyTo": "",
      "jmsRedelivered": 0,
      "jmsTimestamp": 1625493159614,
      "jmsExpiration": 0,
      "jmsType": "TextMessage",
      "jmsStatus": "Success",
      "jmsStatusText": null,
      "leg": 0,
      "timestamp": 1625493159684,
      "duration": 2131,
      "correlationId": "a70ee360d37ee38935e0ab54",
      "serviceName": "OrderStatusService",
      "subject": null,
      "operation": "OrderStatus",
      "type": "jms",
      "finalStatus": "Pass"
    },
    {
      "jmsMessageID": "ID:interface1-37293-1625392418519-77:119:1:1:1",
      "jmsCorrelationID": "",
      "jmsDestination": "queue://com.customer.APPLIC.msg.WWSWO.PROD.Huawei.LoadTenderService_IQ",
      "jmsProviderURL": "local",
      "jmsDeliveryMode": 2,
      "jmsPriority": 4,
      "jmsReplyTo": "",
      "jmsRedelivered": 0,
      "jmsTimestamp": 1625492980071,
      "jmsExpiration": 0,
      "jmsType": "TextMessage",
      "jmsStatus": "Success",
      "jmsStatusText": null,
      "leg": 0,
      "timestamp": 1625492980096,
      "duration": 1955,
      "correlationId": "f40de3605e77c4403466a197",
      "serviceName": "LoadTenderService",
      "subject": null,
      "operation": "RequestForLoadTender",
      "type": "jms",
      "finalStatus": "Pass"
    }
  ]
}

Example OpenTraffic-Events, which are structured in the same way as for HTTPS:

Transaction-Element - Leg 0:

{
  "timestamp": 1625040681215,
  "correlationId": "2927dc60e52ae8c48fb505da",
  "processInfo": {
    "hostname": "api-back-11",
    "domainId": "8591ad8b-c13d-492a-83cd-c3f937bb91bd",
    "groupId": "group-2",
    "groupName": "Back",
    "serviceId": "instance-1",
    "serviceName": "api.customer.com",
    "version": "7.7.20201130"
  },
  "transactionElement": {
    "leg": 0,
    "duration": 2296,
    "serviceName": "ShipmentService",
    "operation": "ShipmentService",
    "finalStatus": "Pass",
    "protocolInfo": {
      "jms": {
        "jmsMessageID": "ID:interface1-36199-1623824816079-189:755:1:1:1",
        "jmsCorrelationID": "",
        "jmsDestination": "queue://com.customer.APPLIC.msg.WWSWO.PROD.P44.ShipmentService_IQ",
        "jmsProviderURL": "local",
        "jmsDeliveryMode": 2,
        "jmsPriority": 4,
        "jmsReplyTo": "",
        "jmsRedelivered": 0,
        "jmsTimestamp": 1625040681191,
        "jmsExpiration": 0,
        "jmsType": "TextMessage",
        "jmsStatus": "Success",
        "jmsStatusText": null,
        "authSubjectId": null
      },
      "recvHeader": "{ \"process_id\": \"CHRISTXM2JS\", \"BDID\": \"E1CHRISTXM2JS131L8181037302510000001\", \"partner_info\": \"P44\", \"receiver\": \"P44\", \"sender\": \"HARD\", \"web_service_name\": \"ShipmentService\", \"transfer_id\": \"E1CHRISTXM2JS131L8181037302510000001\", \"direction\": \"HARD2API\", \"timestamp\": \"1625040653\" }",
      "sentHeader": null,
      "recvPayload": "file:\/\/\/Cloud\/shared\/Openlogs\/2021-06-30\/10.11\/2927dc60e52ae8c48fb505da-0-received",
      "sentPayload": null
    }
  }
}

Transaction-Summary:

{
  "timestamp": 1625040683511,
  "correlationId": "2927dc60e52ae8c48fb505da",
  "processInfo": {
    "hostname": "api-back-11",
    "domainId": "8591ad8b-c13d-492a-83cd-c3f937bb91bd",
    "groupId": "group-2",
    "groupName": "Back",
    "serviceId": "instance-1",
    "serviceName": "api.customer.com",
    "version": "7.7.20201130"
  },
  "transactionSummary": {
    "path": null,
    "protocol": "jms",
    "protocolSrc": "local/com.customer.APPLIC.msg.WWSWO.PROD.P44.ShipmentService_IQ",
    "status": "success",
    "serviceContexts": [

    ]
  }
}
cwiechmann commented 3 years ago

For the records: The JMS-Property filter in the API-Gateway Traffic-Monitor based on Elasticsearch will work a bit differently due to the way how Elasticsearch is indexing the data. The recvHeader or sentHeader of a JMS-Request are stored as "text" in Elasticsearch using the standard text tokenizer. This tokenizer splits up the given text based on whitespaces, tabs, ... and will ignore all special characters. So, each word of this text is saved into the index for the field recvHeader: image

Based on that, you can query for documents containing a specific word in the recvHeader. BUT, the relation between the headerName (e.g. process_id) and headerValue (e.g. JOHNTXM2JS) get lost.

For example the following filter: image

Will turn into the following Elasticsearch query:

"match": {
  "recvHeader": {
    "query": "process_id JOHNTXM2JS",
    "operator": "and"
  }
}

And returns the expected result as the document recvHeader contained both words.

But, also this filter will return the same result: image as the recvHeader contains the words process_id AND BDID.

cwiechmann commented 3 years ago

Support for JMS has been added.