Clustering fails if we wait too long

[ADBond]

Triggered in linker.clustering.cluster_pairwise_predictions_at_threshold(): get SQL execution error DB::Exception: Expected ALL or DISTINCT in SelectWithUnion query, because setting (union_default_mode) is empty.

Not sure exactly what's going on here, but have managed to narrow down problem slightly.

Seems that if we wait too long (I guess after instantiation of the ClickhouseAPI??) then the SET union_default_mode = 'DISTINCT' gets 'forgotten'.

To see this chuck a time.sleep(65) at the very start of this test and it will fail. This is somewhere around the limit of the time - time.sleep(60) runs fine 🤷 .

Don't know the root cause of this, but at a guess the clickhouse_connect.driver.client.Client has some kind of reset after a time that means the setting gets wiped, and as that setting only gets set in ClickhouseAPI.__init__() it isn't refreshed, so we hit the same issue we originally had. Might also be some timer in the Clickhouse Server itself.

[ADBond]

See this docs page - specifically:

session_timeout - Number of seconds of inactivity before the identified by the session id will timeout and no longer be considered valid. Defaults to 60 seconds.

[ADBond]

Couple of things then:

• in testing we probably don't want to instantiate a ClickhouseAPI until we actually need it
• I guess this could be problematic to users working interactively? So maybe we should consider setting union_default_mode more frequently