search

IBM / apiconnect-trawler

API Connect metrics exporter
MIT License
5 stars 10 forks source link

Resolving objectstatus of datapower takes way more than 1s #59

Closed djarcan closed 1 year ago

djarcan commented 1 year ago

Hello,

in our deployment, the trawler times out while fetching the objectstatus over the RMI of the datapower gateways. Currently, the timout is hardcoded to 1s.

In our test environment, we observe times of 5s (environment A) and 15s (environment B) with a rough download size of 25MiB.

Please provide a way to configure such timeouts from the outside by e.g. using environment variables which we can configure in a configmap.

Affected line: https://github.com/IBM/apiconnect-trawler/blob/main/datapower_net.py#L264

djarcan commented 1 year ago

Note: the affected line is only an example, we already observed other datapower metrics to take longer than 1s to fetch for. At best, all datapower relevant timeouts can be configured. (Otherwise, there will be more issues to open)

rickymoorhouse commented 1 year ago

Yes in bigger environments it's quite optimistic to get this all back in 1s - configurable DP timeouts would definitely be the way to go

rickymoorhouse commented 1 year ago

In the latest trawler release the datapower timeout is now configurable with the addition of timeout under the datapower net configuration. It will also use a different DataPower API to just return the object counts rather than pulling back more detail than needed and not using it - so this should perform better as well.