Karton Producers: method `serve` as a wrapper for errors handling

[rakovskij-stanislav]

It would be nice if we will have a possibility to store exceptions and worker count for pure producers.

E.g.:

from karton.core import Producer

class EventsHandler(Producer):
    identity = "grab_from_ext_queue"

    def serve(self):
        pass
    def serve_forever(self, delta):
        while True:
            try:
                t = time.time()
                self.serve()
                self.logger.info("Go back to sleep...")
                dt = time.time() - t
                if dt < delta:
                    time.sleep(delta- dt)
            except KeyboardInterrupt:
                break
            except Exception:
                traceback.print_exc() # plus send the error to redis
                time.sleep(delta)

It will help to know is service alive and does it have any exceptions during serve method execution.

[psrok1]

Yeah, I think it's good idea! I see actually three feature requests there:

1. One that I have already implemented is worker count (and version information) for non-consumers: https://github.com/CERT-Polska/karton/pull/206

2. Dedicated base class for feeders (aka looping producers) built on top of KartonServiceBase (https://github.com/CERT-Polska/karton/blob/master/karton/core/base.py#L175) which would be a nice addition, it's very common use-case and it would be nice to have it standardized. KartonServiceBase gives access to Karton logging and CLI features and can be easily joined with Producer base class.

3. Third thing is keeping information about exceptions. We need to be aware that redis is in-memory database so we need to limit the amount of stored information. In the same time: exceptions are not very heavy, so maybe we can store that information in length-limited queue to be shown in dashboard :thinking:

[rakovskij-stanislav]

I came up with a solution to the problem through method number 2: I have a superproducer that checks are there an active task for "producers" (consumers on steroids that waits superconsumer task), if not - generate it :) But it looks not so good at least because this code is a crutch and it needs to check the full karton queue in redis.

I will look forward for extended service information. This final feature will look good!

For exception handling I can suggest:

• crashed tasks metadata lifetime, as for already implemented consumer tasks.
• storing producer exceptions not in redis but in s3 - also with decay time.