questdb / py-questdb-client

Python client for QuestDB InfluxDB Line Protocol
https://py-questdb-client.readthedocs.io
Apache License 2.0
50 stars 8 forks source link

Possible stickiness in auto_flush_interval behaviour #73

Closed nwoolmer closed 6 months ago

nwoolmer commented 6 months ago

Related to: https://questdb.slack.com/archives/C1NFJEER0/p1711587900594789

Code:

connection = Sender(Protocol.Http, self.config['IP'], self.config['HTTP_Port'],
                                    auto_flush=True,
                                    auto_flush_interval=timedelta(seconds=self.config['Flush_Time']))
                with connection as sender:
                    logger.info(f"Connect to QuestDB at {self.config['IP']}")
                    while True:
                        table_name, symbols, columns, timestamp = self.write_queue.get()
                        start = perf_counter()
                        sender.row(
                            table_name,
                            symbols=symbols,
                            columns=columns,
                            at=timestamp)

                        logger.info(f"Sent row | {perf_counter() - start:.2f} | {len(sender)}")

Output (truncated):

2024-03-27 21:01:21.108 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 76560
2024-03-27 21:01:21.109 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 77949
2024-03-27 21:01:22.087 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 78916
2024-03-27 21:01:22.107 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 79736
2024-03-27 21:01:22.136 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 80557
2024-03-27 21:01:22.153 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 81378
2024-03-27 21:01:22.161 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 82199
2024-03-27 21:01:22.163 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 83020
2024-03-27 21:01:22.183 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 83841
2024-03-27 21:01:22.185 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 84662
2024-03-27 21:01:22.186 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 85483
2024-03-27 21:01:22.187 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 86304
2024-03-27 21:01:22.188 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.00 | 87693
2024-03-27 21:01:23.816 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.74 | 0
2024-03-27 21:01:23.943 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.13 | 0
2024-03-27 21:01:24.102 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.16 | 0
2024-03-27 21:01:24.194 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.08 | 0
2024-03-27 21:01:24.395 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.20 | 0
2024-03-27 21:01:24.481 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.09 | 0
2024-03-27 21:01:24.579 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.09 | 0
2024-03-27 21:01:24.704 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.12 | 0
2024-03-27 21:01:24.820 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.10 | 0
2024-03-27 21:01:24.919 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.09 | 0
2024-03-27 21:01:25.027 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.11 | 0
2024-03-27 21:01:25.116 EDT [INFO] | QuestDB_Logger:write_loop:91 | Sent row | 0.09 | 0

After the first flush after the 10 second interval, the flush happens on each subsequent row.

amunra commented 6 months ago

Bug confirmed. Fix coming.