mymarilyn / clickhouse-driver

ClickHouse Python Driver with native interface support
https://clickhouse-driver.readthedocs.io
Other
1.2k stars 214 forks source link

Client seems to be stuck forever - sometimes - possible deadlock? #439

Closed panthony closed 2 months ago

panthony commented 3 months ago

Describe the bug

Hi ๐Ÿ‘‹๐Ÿป

I have occasionally a python worker using clickhouse-driver client that is stuck forever.

I have yet to really pinpoint exactly where but I suspect an issue with this client because there is nothing else that could hang forever other than a call made with the client (I have logs before / after).

I cannot provide how to reproduce the problem (I wish I could) as it occurs sometimes once every several thousands of queries but digging into the client looking for blocking calls I noticed this method:

    def check_query_execution(self):
        self._lock.acquire(blocking=False)

        if self.is_query_executing:
            raise errors.PartiallyConsumedQueryError()

        self.is_query_executing = True
        self._lock.release()

Should not be the culprit since I've yet to see a PartiallyConsumedQueryError (and we try to acquire with blocking=False) but shouldn't the lock be released in case of PartiallyConsumedQueryError?

Not exactly sure what is the point of that lock anyway since we do not even look if we did acquire the lock or not.

I'm not seeing any other usage for that log than this method.

Otherwise if someone has any idea where there could be blocking calls without the client I'd be interested.

panthony commented 2 months ago

The deadlock seems to come from ClickHouse:

https://github.com/ClickHouse/ClickHouse/issues/66351

So I'll close this issue, but this is still a lock used in clickhouse-driver that may not be released.