api-events are not appearing

[naidu1205]

[org.logstash.plugins.pipeline.PipelineBus] Attempted to send event to 'api-events' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.

[rfeiten]

[2021-01-11T19:58:14,758][WARN ][org.logstash.plugins.pipeline.PipelineBus] Attempted to send event to 'api-events' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.

Also facing it. Please keep us posted if this has been fixed. Thanks!

[michaelhyatt]

Need a bit more info on that. What version are you using? Is it happening at startup or a shutdown?

[rfeiten]

logstash 7.10.1 - it's happening during the startup. I also had to comment out the "UseParNewGC" instruction in java options file. It was giving me an error, even considering I'm using JDK 8.

[michaelhyatt]

Are there any other errors indicating that api-events pipeline is unable to start? Does this error repeat every X seconds (i.e. login http_poller frequency), or is it just one off?

[rfeiten]

Yes, I just realized that there were some outputs being thrown before, as here -->

[ERROR][logstash.filters.http ] error during HTTP request {:url=>"https://anypoint.mulesoft.com/accounts/api/me", :code=>401, :response=>"Unauthorized"}

It seems something with the login pipeline. I even tried to hardcode my password and user in the file, but no luck. It only worked when I hardcoded the generated token, by requesting it through a cURL request, as following:

curl -XPOST -H 'Content-Type: application/x-www-form-urlencoded' -d 'username=xxx&password=xxx' 'https://anypoint.mulesoft.com/accounts/login'

For me, it seems something during the login phase which is not actually being able to extract the generated token, or even, not being able to properly login

` http_poller { urls => { login => {

Supports all options supported by ruby's Manticore HTTP client

    method => post
    url => "https://anypoint.mulesoft.com/accounts/login"
    headers => {
      "Content-Type" => "application/x-www-form-urlencoded"
    }
    body => "username=${CH_USER}&password=${CH_PASSWORD}"
 }
}

`

[michaelhyatt]

So, I understand the curl login statement works, while http_poller doesn't?

What payload does curl return?

[rfeiten]

This is correct. I tried to debug but it's a bit hard to in Logstash, not an expert on it... By the way, the payload cURL returns the following structure...

{ "access_token": "197bxxe1-ae02-466b-...", "token_type": "bearer", "redirectUrl": "/home/" }

I see that your script is trying to whitelist the access_token and key name is fine. Not sure why it does not authenticate in the ruby request.

[michaelhyatt]

Is there cookies: false set on http_poller, as per #2 ?

[rfeiten]

Yes, I have added in the http_poller , as shown here -->

# 1. Kick off with the login poll http_poller { cookies => "false" urls => { login => {

The error changed, meaning that it could pass from the "api/me" and other calls contained in the login pipeline. Now, the issue is happening during the next steps of the pipeline. In my case, I have just configured the logs pipeline to run, to simplify the process and troubleshooting.

[ERROR][logstash.filters.http ] error during HTTP request {:url=>"https://anypoint.mulesoft.com/cloudhub/api/v2/applications", :code=>401, :response=>"{\"error\":\"Unauthorized\",\"message\":\"Failed to create session using the supplied Authorization header\"}"}

Seems that it cannot recover the access_token from the previous pipeline poller. I also tried to search from the http object reference as shown here https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html but it doesn't have a configuration to disable cookies... Any idea?

[michaelhyatt]

There is http filter plugin being used there, not input and it has cookies flag. https://www.elastic.co/guide/en/logstash/current/plugins-filters-http.html

This is the curl equivalent for the GET applications call that uses the token that works for me:

curl --location --request GET 'https://anypoint.mulesoft.com/cloudhub/api/applications?retrieveStatistics=true&period=3600000&intervalCount=50&authentication_token=c3xxx9d4-52ed-46a1-9fc1-xxx' \
--header 'X-ANYPNT-ENV-ID: yyy-8fe9-4331-b552-xxx' \
--header 'Authorization: Bearer c3xxx9d4-52ed-46a1-9fc1-xxx'

[rfeiten]

You are right, spent sometime today trying to make it work, finally worked once I have also included the cookies => "false" to all http operations during the pipeline.

Therefore, it worked for a couple minutes fine, but after that (probably due to token expiration), it started again to have issues, as same as before

[ERROR][logstash.filters.http ] error during HTTP request {:url=>"https://anypoint.mulesoft.com/cloudhub/api/v2/applications", :code=>401, :response=>"{\"error\":\"Unauthorized\",\"message\":\"Failed to create session using the supplied Authorization header\"}"}

What do you mean mean above, I presume, is getting the code to work with the old version of the API (non v2)?

Thanks.

[michaelhyatt]

I used v1 and v2 APIs when I built it.

I think the problem with your setup is getting the initial token. Does initial login http_poller output give the token_type: bearer?

Also, this is the equivalent curl statement I run to get the bearer token:

curl --location --request POST 'https://anypoint.mulesoft.com/accounts/login' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'username=yy' \
--data-urlencode 'password=xx'

The response I am getting is:

{
  "access_token": "xxx-52ed-46a1-9fc1-yyy",
  "token_type": "bearer",
  "redirectUrl": "/home/"
}

Is it the same as you are seeing and using?

[rfeiten]

This is correct. The token is generated of type bearer.

{ "access_token": "xxx-133e-4f8d-xxx-16bae2cxxx", "token_type": "bearer", "redirectUrl": "/home/" }

The workflow is now working, but the problem is that it only works for a couple minutes, until the TTL associated with the token expires, I start to see the mentioned issues again, and need to restart the process in order to work it properly. It seems the token is not re-generated. Is there a way to output the token information being used, as well as http requests by outputting it to logs? That could be helpful to understand which token is being used, and if in fact it's regenerating the token or using always the same one.

[michaelhyatt]

Can you please test the following?

login.conf:

input {

  http_poller {
    urls => {
      login => {
        # Supports all options supported by ruby's Manticore HTTP client
        method => post
        url => "https://anypoint.mulesoft.com/accounts/login"
        headers => {
          "Content-Type" => "application/x-www-form-urlencoded"
        }
        body => "username=${CH_USER}&password=${CH_PASSWORD}"
     }
    }
    request_timeout => 120
    schedule => { "every" => "30s"}
    codec => "json"
  }
}

filter {
  ruby {
    code => "event.set('now_ms', Time.now.to_i * 1000)"
  }

  # After login, leave just the access_token
  prune {
    whitelist_names => [ "now_ms", "access_token" ]
  }

}

output {
  stdout { codec => rubydebug }
}

Run logstash with the above config and your username and password following:

CH_USER=yourusername CH_PASSWORD=yourpwd bin/logstash -f login.conf

Validate that you are seeing the token being displayed after about 30 seconds or so:

[2021-01-20T21:59:50,751][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
    "access_token" => "xxx-65ab-4465-92a1-yyy",
          "now_ms" => 1611140453000
}

[rfeiten]

Yes, I can see the access token being generated.

[Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} { "access_token" => "yyy-7082-4476-9098-xxx", "now_ms" => 1611153275000 }

The token seems to be generated OK, but the problem seems to be when it tries to renew the token in each iteration, like if it would be attached to the cookies or session, something like that... and that throws the 401 then.

[michaelhyatt]

I am seeing the problem with the 2nd call and onwards.

[2021-01-21T09:28:25,048][ERROR][logstash.codecs.json     ][main][ceaf93f695de8bcf2a668df85bdcf794245f718da55f4966e7d8555433bab440] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized token 'invalid': was expecting ('true', 'false' or 'null')
 at [Source: (String)"invalid csrf token"; line: 1, column: 8]>, :data=>"invalid csrf token"}
{
    "now_ms" => 1611181705000
}

[michaelhyatt]

Adding cookies => "false" option fixes it:

input {

  http_poller {
    urls => {
      login => {
        # Supports all options supported by ruby's Manticore HTTP client
        method => post
        url => "https://anypoint.mulesoft.com/accounts/login"
        headers => {
          "Content-Type" => "application/x-www-form-urlencoded"
        }
        body => "username=${CH_USER}&password=${CH_PASSWORD}"
     }
    }
    request_timeout => 120
    schedule => { "every" => "10s"}
    codec => "json"
    cookies => "false"
  }
}

filter {
  ruby {
    code => "event.set('now_ms', Time.now.to_i * 1000)"
  }

  # After login, leave just the access_token
  prune {
    whitelist_names => [ "now_ms", "access_token" ]
  }

}

output {
  stdout { codec => rubydebug }
}

Let me know if it works for you.

[sm0w0ms]

I tried this and now it dose not store any CSRF token and the flow works fine but, I still didn't get the data of api-events `input {

1. Kick off with the login poll

http_poller { urls => { login => {

Supports all options supported by ruby's Manticore HTTP client

    method => post
    url => "https://anypoint.mulesoft.com/accounts/login"
    cookies => "false"
    headers => {
      "Content-Type" => "application/x-www-form-urlencoded"
    }
    body => "username=${CH_USER}&password=${CH_PASSWORD}"
 }
}
request_timeout => 120
# Supports "cron", "every", "at" and "in" schedules by rufus scheduler
# schedule => { cron => "* * * * * UTC"}
schedule => { "every" => "60s"}
codec => "json"
cookies => "false"
# A hash of request metadata info (timing, response headers, etc.) will be sent here
# metadata_target => "http_poller_metadata"

} }`

[michaelhyatt]

I tried this and now it dose not store any CSRF token and the flow works fine but, I still didn't get the data of api-events `input {

1. Kick off with the login poll

http_poller { urls => { login => { # Supports all options supported by ruby's Manticore HTTP client method => post url => "https://anypoint.mulesoft.com/accounts/login" cookies => "false" headers => { "Content-Type" => "application/x-www-form-urlencoded" } body => "username={CH_USER}&password={CH_USER}&password={CH_PASSWORD}" } } request_timeout => 120 # Supports "cron", "every", "at" and "in" schedules by rufus scheduler # schedule => { cron => " * UTC"} schedule => { "every" => "60s"} codec => "json" cookies => "false" # A hash of request metadata info (timing, response headers, etc.) will be sent here # metadata_target => "http_poller_metadata" } }`

Can you replicate this API call in Postman and see if there are any API events in your app at all?

[sm0w0ms]

As you suggested, I tried calling event api it return empty array curl --request GET \ --url 'https://anypoint.mulesoft.com/analytics/1.0/********-*****-*****/events?format=json&startDate=2022-05-01T00%3A40%3A19.000Z&endDate=2022-05-30T06%3A19%3A05.674Z&fields=Application%20Name%2C%20City' \ --header 'Authorization: Bearer e*****-******-*********'

[sm0w0ms]

Do we have enable some api-events in anypoint ? If yes then please tell me how I can do that.

[michaelhyatt]

Do we have enable some api-events in anypoint ? If yes then please tell me how I can do that.

API events are generated by APIs, you need to define APIs on your HTTP endpoints.

[sm0w0ms]

So i am getting api-events after changing url https://anypoint.mulesoft.com:443/analytics/1.0/%{organization_id}/environments/%{environment_id}/events?format=json&startDate=%{latest_timestamp}&endDate=%{now_formatted}&fields=Application.Application%20Name.Browser.City.Client%20IP.Continent.Country.Hardware%20Platform.Message%20ID.OS%20Family.OS%20Major%20Version.OS%20Minor%20Version.OS%20Version.Postal%20Code.Request%20Outcome.Request%20Size.Resource%20Path.Response%20Size.Response%20Time.Runtime%20Host.Status%20Code.Timezone.User%20Agent%20Name.User%20Agent%20Version.Verb.Violated%20Policy%20Name We have to specify environment

[sm0w0ms]

You can also use this url https://anypoint.mulesoft.com:443/analytics/1.0/%{organization_id}/environments/%{environment_id}/events?format=json&duration=5h&fields=Application.Application%20Name.Browser.City.Client%20IP.Continent.Country.Hardware%20Platform.Message%20ID.OS%20Family.OS%20Major%20Version.OS%20Minor%20Version.OS%20Version.Postal%20Code.Request%20Outcome.Request%20Size.Resource%20Path.Response%20Size.Response%20Time.Runtime%20Host.Status%20Code.Timezone.User%20Agent%20Name.User%20Agent%20Version.Verb.Violated%20Policy%20Name

Instead of start and end date, mention duration.