Traffic Monitor not working as expected

lucasbuschlinger commented 2 years ago

ELK-Solution version

4.1.0

Elastic-Stack version

7.16.3

API-Management version

7.7 August Update

API-Management deployment option

No response

Question

Hi,

we are deploying the ELK solution to our Axway cluster.

We have setup the the ANM + policies as described but we currently have the following issues with the Traffic Manager:

We cannot retrieve logs older than about 2 days. In Elastic/Kibana we can see the logs for up to 14 days (as per Elastic default)
The restricted view does not work. Even users that are only an API GW Operator are able to see all logs instead of their organization in API Manager only

Your help would be greatly appreciated

cwiechmann commented 2 years ago

Hi @lucasbuschlinger,

Have you followed the steps as described here? https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk#configure-axway-api-management

If yes, very likely the communication from the Admin-Node-Manager (ANM) to the API-Builder-HTTPS-Endpoint is not working. Quite often, caused by the certificate not trusted by the ANM.

You can verify this by inspecting the trace-file from the ANM to see, if there are any trace messages logged in regards of an untrusted certificate. If yes, please import the CA into the ANM-Key-Store. This is how it looks on my side. Obviously the name of the CA might be a different depending on your setup:

lucasbuschlinger commented 2 years ago

Hi @cwiechmann, thank you for your answer. Indeed there was a misconfiguration of the ANM URL in the .evn settings file. Now the Traffic Monitor uses the Elastic data. But this only works for Admin Users.

The restricted view does not work, it only returns the message {"message":"Cannot return API-Manager config without groupId and region as no default API-Manager is configured."}. I have narrowed it down in the apibuilder4elastic code (I hope I didn't miss anything though).

This happens in the actions.js code during the lookupCurrentUsers function. After this function gets the current ANM user (which works correctly) it calls _getManagerUser(user);. This in turn calls getManagerConfig(pluginConfig.apimanager, groupId);.

However, the groupId is not passed in the _getManagerUser(user) call. This causes the getManagerConfig in utils.js to evaluate the first condition to true and trying to load the default config. But as we have configured multiple API Managers, we do not a have a default one (cf. parseAPIManagerConfig).

So, is this a bug or do we have some misconfiguration here?

Thanks!

cwiechmann commented 2 years ago

Hi @lucasbuschlinger

This is because there is no default API-Manager configured, which is unfortunately required in that situation, because during the login process to the ANM no group-id and region is given.

I hope it's possible in your setup to select one API-Manager that contains your users otherwise we have to find another solution.

I'll make this more clear in the documentation.

Cheers, Chris

lucasbuschlinger commented 2 years ago

Hi again,

that's unfortunate but so be it.

So do I understand it correctly, that we would have to simply add a default API Manager like this for example: API_MANAGER=url_EMEA:8075,group-2|EMEA|url_EMEA:8075,group-2|US|url_US:8075,group-2|CN|url_CN:8075? Which would still give uns the groups and regions for the API lookup but also the default one for the user lookup?

And secondly, would this mean that we would need to have the same users on all configured groups/regions in order to guarantee that the traffic manager works as intended?

Best, Lucas

cwiechmann commented 2 years ago

Your configuration looks good now. And yes, you are right, the Group-Specific API-Manager configurations are used during data ingestion to lookup the correct API- and Application-Details at the correct API-Manager.

And secondly, would this mean that we would need to have the same users on all configured groups/regions in order to guarantee that the traffic manager works as intended?

Yes, that is unfortunately correct. I'm pretty sure, there are other options to make this working, but as it is today, the solution can only use the default API-Manager.

cwiechmann commented 2 years ago

I checked the option to use an external HTTP-REST-API to perform the user authorization. When using the External-REST-API the user is not looked up anymore at the API-Manager.

With that, you can put any REST-API in place (e.g. on the API-Gateway itself), that gets the username and some other options as input. With that, this REST-API decides what is the user allowed to see.

It may talk to an LDAP-System
perhaps the username includes region, etc.

Whatever this REST-API returns is then used to construct the restricted Elasticsearch-Query. See here an example with some documentation: https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/blob/develop/config/authorization-config-sample.js

lucasbuschlinger commented 2 years ago

Thank you so much again for your help. We will see if this is an option for us, else we will have to stick with the internal OPSDB logs.

cwiechmann commented 2 years ago

have to stick with the internal OPSDB logs

I hope not @lucasbuschlinger. That would be very sad just for that reason. Let me know if there is anything else I can do.

lucasbuschlinger commented 2 years ago

Absolutely, but as we do have differing users on the three hubs, it does not make sense to just use one for looking them up.

Maybe alternatively we could 'disable' the restricted view and still give all users (i.e. Admins and Operators) the full logs, albeit them being retrieved from Elastic instead of the OPSDB?

cwiechmann commented 2 years ago

Absolutely, but as we do have differing users on the three hubs, it does not make sense to just use one for looking them up.

Sure, that's why I was thinking to have this REST-API which does the internal lookup against the three hubs based on some custom logic.

Maybe alternatively we could 'disable' the restricted view and still give all users (i.e. Admins and Operators) the full logs, albeit them being retrieved from Elastic instead of the OPSDB?

That might be a start and this is just a config option:

    // If you would like to disable user authorization completely, set this flag to false
    enableUserAuthorization: false,

Have you any news concerning the issue #180 ?

lucasbuschlinger commented 2 years ago

Makes sense, we will take a look at it.

Regarding the other issue, Niko will reach out to you for a session soon.

Axway-API-Management-Plus / apigateway-openlogging-elk