[YSQL] Extension creation failing with statement timeout in a geo-partitioned universe

[rjalan-yb]

Jira Link: DB-7136

Description

While automating Geo-Partitioning for language layer, I came across a behaviour where installation of extension btree_gist fails majority of times with this error: canceling statement due to statement timeout In this case, we have 2 us zones and 1 asia-pacific zone configured in universe. It is very frequently occurring and has showed up in almost every run for last couple of days.

Jenkins runs: https://jenkins.dev.yugabyte.com/job/itest-system-developer/7125/

Steps:

testgeopartitioningwithysqllanguagelayer-aws-rf3-geo-partition-multiregion: Start
    (     0.544s) User Login : Success
    (     0.206s) Refresh YB Version : Success
    (   120.575s) Setup Provider : Success
    (     0.051s) Updating Health Check Interval to 60000 sec : Success
    (  1604.364s) Create universe rjalan-isd7091-e5f16750d8-20230704-021427 : Success
    (   124.805s) Create Database gp_language_layer : Success
    (    66.659s) Create Colocated Database gp_language_layer : Success
    (  1354.918s) Create tablespace, parent table and geo partitioned tables : >>> Integration Test Failed <<< 
canceling statement due to statement timeout

    (    54.365s) Saved server log files and keys at /share/jenkins/workspace/itest-system-developer/logs/2.19.1.0_testgeopartitioningwithysqllanguagelayer-aws-rf3-geo-partition-multiregion_20230704_025955 : Success
    (   116.752s) Destroy universe : Success
    (     0.234s) Check and stop workloads : Success
testgeopartitioningwithysqllanguagelayer-aws-rf3-geo-partition-multiregion: End

Test logs:

INFO:root:CREATE EXTENSION IF NOT EXISTS btree_gist;
2023-07-04 03:41:06,838 database_operations.py:871 INFO testgeopartitioningwithysqllanguagelayer-aws-rf3-geo-partition-multiregion CREATE EXTENSION IF NOT EXISTS btree_gist;
2023-07-04 03:54:27,563 test_base.py:179 ERROR testgeopartitioningwithysqllanguagelayer-aws-rf3-geo-partition-multiregion ITEST FAILED testgeopartitioningwithysqllanguagelayer-aws-rf3-geo-partition-multiregion : QueryCanceled('canceling statement due to statement timeout\n')

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.

[sushantrmishra]

@rjalan-yb Is there statement_timeout set during this test ? If yes then please share value.

Also btree_gist is not supported extension. Do you see the issue with any supported extension ?

[rjalan-yb]

No @sushantrmishra, not in any supported extension. Here I have 80000 ms (1.3 mins) set. But this goes not for 13 minutes and than fails. But a lot of times it succeeds too. So if it is not supported, then this installation should result in error everytime, should it not?