Can't see logs due to 'flush' issue

[FelipeLahti]

Issue

Logs aren't propagating to axiom.

How to reproduce:

• Log ~900 in less than 1 sec
• Don't send anything else after this
• Logs won't show in axiom as it's not getting flushed

Possible solution:

• Flush periodically after x seconds even if the buffer isn't full or the last log was less than 1 s ago.

[FelipeLahti]

Workaround: Spin up a thread to flush it every 30s. It could be add as built in into the lib if that makes sense.

        client = axiom_py.Client()
        handler = AxiomHandler(client, "my-dataset")

        def flush_handler_periodically():
            while True:
                handler.flush()
                time.sleep(30)

        thread = threading.Thread(target=flush_handler_periodically)
        thread.daemon = True
        thread.start()

        logging.getLogger().addHandler(handler)

[bahlo]

Hi @FelipeLahti, thanks! You’re right, we should fix this. Another option would be using a threading.Timer to flush logs at least every second—similarly to how we handle this situation in axiom-js. Will keep his open until that‘s implemented 😌

[bahlo]

Fixed in https://github.com/axiomhq/axiom-py/pull/140, will be in the next release

[subham1099]

@bahlo Could we implement the same for structlog? Should I raise a PR with a similar change to the structlog module?