Usage with flaskRESTFUL

[captify-dieter]

How can metrics be logged and customized when API is constructed using flaskRESTful library, where instead of routes, you have:

class Coordinates(Resource):
    def __init__(self):
        self.parser = reqparse.RequestParser()
        self.parser.add_argument('radius', type=float, required=False, help="Radius", location='args', default=1)

    def post(self):
        args = self.parser.parse_args()

[rycus86]

Hey @captify-dieter , that's a good question! Let me investigate and come back to you on it. I'd expect the defaults should work, and perhaps the decorators too, but haven't tested explicitly.

[captify-dieter]

Would the decorator need to be added for the class? (since the end point is defined on starting the flask server)

api.add_resource(Coordinates, '/coordinates')

[ziedbf]

Hi @captify-dieter and @rycus86, I am hitting the same issues but with Flask-restplus, i managed to get the endpoint metrics exposed however i end up with an issue related to duplicate registration of metrics.

class AuthFlaskRestplusPrometheusMetricsDto:
    api = Namespace('metrics', description='metrics collection for prometheus')

api = AuthFlaskRestplusPrometheusMetricsDto.api

@api.route('/')
class FlaskRestplusPrometheusMetrics(PrometheusMetrics, Resource):

  def __init(app, **kwargs):
    self.app = app

  def register_endpoint(self,path, app=None):
    pass

  @api.doc('report metrics using flask prometheus metrics collector ')
  def get(self):
    """
    Register the metrics endpoint on the Flask application.
    :param path: the path of the endpoint
    :param app: the Flask application to register the endpoint on
        (by default it is the application registered with this class)
    """
    headers = {'Content-Type': CONTENT_TYPE_LATEST}
    return generate_latest(registry), 200, headers  

and basically just register it as resources to restplus:

from .main import api as metrics_ns

blueprint = Blueprint('api', __name__)
....
api.add_namespace(metrics_ns, path='/metrics')

The issues i am facing upon that as i mentioned above is the registery duplication:

27.0.0.1 - - [25/Sep/2019 15:29:41] "GET /api/metrics/ HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask_reverse_proxy_fix/middleware/__init__.py", line 55, in __call__
    return self.app(environ, start_response)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
    return self.app(environ, start_response)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask_restplus/api.py", line 584, in error_router
    return original_handler(e)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/_compat.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask_restplus/api.py", line 584, in error_router
    return original_handler(e)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/_compat.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask_restplus/api.py", line 325, in wrapper
    resp = resource(*args, **kwargs)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/flask/views.py", line 88, in view
    self = view.view_class(*class_args, **class_kwargs)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/prometheus_flask_exporter/__init__.py", line 133, in __init__
    self.init_app(app)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/prometheus_flask_exporter/__init__.py", line 155, in init_app
    self._defaults_prefix, app
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/prometheus_flask_exporter/__init__.py", line 279, in export_defaults
    **buckets_as_kwargs
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/prometheus_client/metrics.py", line 499, in __init__
    labelvalues=labelvalues,
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/prometheus_client/metrics.py", line 107, in __init__
    registry.register(self)
  File "/Users/zied/Desktop/codebase/infrastructure/banzaicloud/spotguides/flask-postgres-redis/env-3/lib/python3.6/site-packages/prometheus_client/registry.py", line 29, in register
    duplicates))
ValueError: Duplicated timeseries in CollectorRegistry: {'flask_http_request_duration_seconds_count', 'flask_http_request_duration_seconds_bucket', 'flask_http_request_duration_seconds_sum', 'flask_http_request_duration_seconds_created'}

I am trying to figure out how to unregister and register metrics again, any advise @rycus86

[ziedbf]

This is approach seems to be conflicting due to inheritance of classes within python, will try to figure another approach.

[rycus86]

Hi @captify-dieter and @ziedbf ! Thanks for the details so far! It would help if you could set up a really small example that demonstrates the problem, then I could use that as a reference to work out a solution.

Cheers!

[ziedbf]

@rycus86 please find the public repos https://github.com/ziedbouf/flask-prometheus-demo inlcudes the flask-restplus, i think the method should work for both restfull and restplus.

I am still thinking of how this could be achieved

[rycus86]

Thanks a lot @ziedbf !

[ziedbouf]

Welcome @rycus86 i figure out easier approach to apply and expose, however i am only stuck with flask restplus as it will try to parse the bytes objects returned including prometheus metrics.

I will update the last changes on my end and i need to figure out JSON serialisation within flask restplus.

[rycus86]

Would the decorator need to be added for the class? (since the end point is defined on starting the flask server)

api.add_resource(Coordinates, '/coordinates')

I just realized I already have an example for Flask-RESTful here :man_facepalming: https://github.com/rycus86/prometheus_flask_exporter/blob/master/examples/restful-with-blueprints/server.py

Does this help answer your question @captify-dieter ?

[rycus86]

@ziedbf that^ should also work for RESTplus I think. Are you trying to add individual metrics to a single endpoint, or you want to add a metric for all the endpoints (like https://github.com/rycus86/prometheus_flask_exporter/issues/34) ?

[ziedbouf]

I think the issues on exposing the metrics endpoints which is not straight forward on flask restplus. I update the codes and i end up with the metrics endpoints exposed however i am hitting the issue related to json seralizer trying to parse bytes object returned by the flask restplus (#40)

[rycus86]

Got it. Will have a look now.

[ziedbouf]

decoding the bytes object works fine however no idea if it has an impact from prometheus side, return generate_latest(metrics.registry).decode('utf-8'), 200, headers 'o.o'.

output:

"# HELP python_gc_objects_collected_total Objects collected during gc\n# TYPE python_gc_objects_collected_total counter\npython_gc_objects_collected_total{generation=\"0\"} 13509.0\npython_gc_objects_collected_total{generation=\"1\"} 2109.0\npython_gc_objects_collected_total{generation=\"2\"} 156.0\n# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC\n# TYPE python_gc_objects_uncollectable_total counter\npython_gc_objects_uncollectable_total{generation=\"0\"} 0.0\npython_gc_objects_uncollectable_total{generation=\"1\"} 0.0\npython_gc_objects_uncollectable_total{generation=\"2\"} 0.0\n# HELP python_gc_collections_total Number of times this generation was collected\n# TYPE python_gc_collections_total counter\npython_gc_collections_total{generation=\"0\"} 186.0\npython_gc_collections_total{generation=\"1\"} 16.0\npython_gc_collections_total{generation=\"2\"} 1.0\n# HELP python_info Python platform information\n# TYPE python_info gauge\npython_info{implementation=\"CPython\",major=\"3\",minor=\"6\",patchlevel=\"5\",version=\"3.6.5\"} 1.0\n# HELP flask_http_request_duration_seconds Flask HTTP request duration in seconds\n# TYPE flask_http_request_duration_seconds histogram\n# HELP flask_http_request_total Total number of HTTP requests\n# TYPE flask_http_request_total counter\n# HELP flask_exporter_info Information about the Prometheus Flask exporter\n# TYPE flask_exporter_info gauge\nflask_exporter_info{version=\"0.9.1\"} 1.0\n"

[rycus86]

I think the metrics endpoint should be exposed in a different way, but let me test it.

[rycus86]

Are you actually just trying to add Swagger on the metrics endpoint? If not, then simply using the wrapper (rather than extending it) might work like in this example: https://github.com/rycus86/prometheus_flask_exporter/blob/master/examples/restful-with-blueprints/server.py

[ziedbouf]

yes it works fine the only issues in encoding

@api.produces('text/plain ')
class PrometheusMetricsEndpoint(Resource):
  status = 200

  @staticmethod
  def get():
    from prometheus_client import multiprocess, CollectorRegistry

    if 'prometheus_multiproc_dir' in os.environ:
        registry = CollectorRegistry()
    else:
        registry = metrics.registry

    if 'name[]' in request.args:
        registry = registry.restricted_registry(request.args.getlist('name[]'))

    if 'prometheus_multiproc_dir' in os.environ:
        multiprocess.MultiProcessCollector(registry)

    headers = {'Content-Type': CONTENT_TYPE_LATEST}
    return generate_latest(metrics.registry).encode('utf-8'), 200, headers

api.add_resource(PrometheusMetricsEndpoint, '/metrics', endpoint='metrics')

[rycus86]

Right, so this is not using prometheus_flask_exporter anymore. :) This seems to work for me, not sure what am I missing?

app = Flask(__name__)
blueprint = Blueprint('api_v1', __name__, url_prefix='/api/v1')
api = Api(blueprint)
metrics = PrometheusMetrics(blueprint)

class Test(Resource):
    status = 200

    @staticmethod
    @metrics.summary('test_by_status', 'Test Request latencies by status', labels={
        'code': lambda r: r.status_code
    })
    def get():
        if 'fail' in request.args:
            return 'Not OK', 400
        else:
            return 'OK'

api.add_resource(Test, '/test', endpoint='test')
app.register_blueprint(blueprint)

Why do you want to register the metrics endpoint as a resource? If you just give it the underlying Flask Blueprint or app, it should work OK.

[ziedbf]

The issues if i don't expose the metrics endpoints, you cannot collect the metrics as the flask server will reply with 404,

[rycus86]

How are you setting up the PrometheusMetrics object? If you pass it either the app = Flask(..) object or the Blueprint instance, it should work.

[captify-dieter]

The issues if i don't expose the metrics endpoints, you cannot collect the metrics as the flask server will reply with 404,

Note @ziedbf you have a typo in the url: "/me[r]trics"

[ziedbouf]

@captify-dieter just mistake from my end but the issue still persisting.

I solved the issue by just avoiding dealing with flask restplus and just use regular flask blueprints.

br_metrics = Blueprint('metrics', __name__, url_prefix='/metrics')
metrics = PrometheusMetrics(br_metrics)

@br_metrics.route('/')
def meter():
  from prometheus_client import multiprocess, CollectorRegistry

  if 'prometheus_multiproc_dir' in os.environ:
      registry = CollectorRegistry()
  else:
      registry = metrics.registry

  if 'name[]' in request.args:
      registry = registry.restricted_registry(request.args.getlist('name[]'))

  if 'prometheus_multiproc_dir' in os.environ:
      multiprocess.MultiProcessCollector(registry)

  headers = {'Content-Type': CONTENT_TYPE_LATEST}
  return generate_latest(metrics.registry), 200, headers

and just register the blueprint with flask app. the root cause is flask restplus force the any response to be serialize to json object which is not the case. I am still checking the documentation to find any options of avoiding the serialization to json object.

[rycus86]

You only need the first bit, this library will expose /metrics for you, so you don't have to implement this part.

br_metrics = Blueprint('metrics', __name__, url_prefix='/metrics')
metrics = PrometheusMetrics(br_metrics)

In this case, you're endpoint will be at /metrics/metrics, because this library registers /metrics on the root of the app/blueprint, and you set that to /metrics as well. :)

[ziedbouf]

That's what i was suspecting, the issues is that what's happening with me that the metrics endpoints is not registered without using meter() func. o.O and no idea what's the reason behind such behavior.

[captify-dieter]

Thanks @rycus86 the example worked perfectly! Happy to close this (@ziedbf @ziedbouf)

[rycus86]

Thanks @captify-dieter ! Let me close this then, but feel free to comment or reopen if this is still an issue.