Concurrency issues - Githubissues

dargi98 commented 6 months ago

I'm using toxiproxy to introduce 5 seconds of latency to the DB to do this tests. https://github.com/Shopify/toxiproxy

BeginTx() hangs when there's another transaction going on from a different go routine. The first BeginTx is immediate, but the second one takes 46s.

Same with statements, the second concurrent statement hangs, until the other go routine has finished.

sijms commented 6 months ago

what is sql statement

dargi98 commented 6 months ago

A simple select:

SELECT a FROM b FETCH NEXT 1 ROWS ONLY

sijms commented 6 months ago

would you please send example code to reproduce the error on my environment

dargi98 commented 6 months ago

Run toxiproxy so it connects to Oracle DB in 172.17.0.1:1521. The proxy redirects 0.0.0.0:5432 to 172.17.0.1:1521.

docker run -d -p 5432:5432 -p 8474:8474 ghcr.io/shopify/toxiproxy
curl -XPOST "localhost:8474/proxies" -d "{ \"name\": \"proxy_es\", \"listen\": \"0.0.0.0:5432\", \"upstream\": \"172.17.0.1:1521\", \"enabled\": true}"

Add 5 seconds of latency to the proxy:

curl -X POST "localhost:8474/proxies/proxy_es/toxics" -d '{               
  "name": "toxic_latency",
  "type": "latency",
  "stream": "upstream",
  "attributes": {
    "latency": 5000
  }
}'

The code I used. I'm setting CONNECTION TIMEOUT=30 in the connection URL and I'm connecting to 0.0.0.0:5432:

package main

import (
    "context"
    "database/sql"
    "fmt"
    "log"
    "os"
    "time"

    "github.com/go-co-op/gocron"
    _ "github.com/sijms/go-ora/v2"
)

func main() {
    pool, err := sql.Open("oracle", os.Getenv("DSN"))
    if err != nil {
        log.Fatalf("Err conn: %v", err)
    }
    pool.SetMaxOpenConns(500)
    defer pool.Close()
    scheduler := gocron.NewScheduler(time.UTC)
    ctx, cancel := context.WithTimeout(context.Background(), time.Second*100)
    defer cancel()
    now := time.Now()
    stmt, err := pool.PrepareContext(ctx, "Select * from dba_objects a, dba_objects b, dba_objects c, dba_objects d, dba_objects e")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println("stmt prepared in: ", time.Since(now))

    if _, err := scheduler.Every(250).Millisecond().Do(execQuery, stmt); err != nil {
        log.Fatalf("Err job: %v", err)
    }

    scheduler.StartBlocking()
}

func execQuery(stmt *sql.Stmt) {
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    now := time.Now()
    row := stmt.QueryRowContext(ctx)
    if err := row.Err(); err != nil {
        log.Printf("Rows err: %v", err)
        fmt.Println("statement failed in: ", time.Since(now))
        return
    }
    fmt.Println("statement done in: ", time.Since(now))
}

This is the output:

stmt prepared in:  40.077690565s
statement done in:  6.142609202s                                                                                                                                                                                                                                                                                                                                                            
statement done in:  6.090926273s                                                                                                                                                                                                                                                                                                                                                            
statement done in:  6.187848496s                                                                                                                                                                                                                                                                                                                                                            
statement done in:  6.059206376s                                                                                                                                                                                                                                                                                                                                                            
statement done in:  6.047403345s                                                                                                                                                                                                                                                                                                                                                            
statement done in:  6.543754875s                                                                                                                                                                                                                                                                                                                                                            
2024/01/29 20:22:36 Rows err: write tcp 172.17.0.1:50644->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.920602666s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:22:36 Rows err: write tcp 172.17.0.1:50650->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.973024767s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:22:36 Rows err: write tcp 172.17.0.1:50654->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.903278367s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:22:37 Rows err: write tcp 172.17.0.1:50658->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.941343741s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:22:37 Rows err: write tcp 172.17.0.1:50672->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.906323782s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:22:37 Rows err: write tcp 172.17.0.1:50678->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.926543026s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:22:37 Rows err: write tcp 172.17.0.1:50680->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.927807901s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:22:38 Rows err: write tcp 172.17.0.1:50688->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.921435211s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:22:38 Rows err: write tcp 172.17.0.1:50698->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.957215029s

At first its working, but then it starts failing. Notice that it timeouts in 40s, even when the timeout is set to 30s. Also, do you know why the first call (preparing the statement) takes 40s?

Also, waiting a couple of seconds I see this:

2024/01/29 20:23:04 Rows err: write tcp 172.17.0.1:36514->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.956718953s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:23:04 Rows err: ORA-03120: two-task conversion routine: integer overflow                                                                                                                                                                                                                                                                                                      
statement failed in:  10.211139534s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:23:04 Rows err: write tcp 172.17.0.1:36520->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.92381371s                                                                                                                                                                                                                                                                                                                                                          
2024/01/29 20:23:04 Rows err: ORA-03120: two-task conversion routine: integer overflow                                                                                                                                                                                                                                                                                                      
statement failed in:  10.20250386s                                                                                                                                                                                                                                                                                                                                                          
2024/01/29 20:23:04 Rows err: ORA-01001: invalid cursor                                                                                                                                                                                                                                                                                                                                     
statement failed in:  10.167295507s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:23:04 Rows err: write tcp 172.17.0.1:36524->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.949943106s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:23:05 Rows err: write tcp 172.17.0.1:36530->172.17.0.1:5432: i/o timeout                                                                                                                                                                                                                                                                                                      
statement failed in:  40.913941042s                                                                                                                                                                                                                                                                                                                                                         
2024/01/29 20:23:05 Rows err: ORA-01001: invalid cursor                               
statement failed in:  10.170467176s
2024/01/29 20:23:05 Rows err: write tcp 172.17.0.1:36534->172.17.0.1:5432: i/o timeout
statement failed in:  40.893319349s

And later, all logs are like this:

2024/01/29 20:24:38 Rows err: ORA-01001: invalid cursor
statement failed in:  14.614671403s
2024/01/29 20:24:38 Rows err: ORA-01001: invalid cursor
statement failed in:  14.365169549s
2024/01/29 20:24:38 Rows err: ORA-01001: invalid cursor
statement failed in:  14.115401431s
2024/01/29 20:24:39 Rows err: ORA-01001: invalid cursor
statement failed in:  13.865371991s
2024/01/29 20:24:39 Rows err: ORA-01001: invalid cursor
statement failed in:  13.614005377s
2024/01/29 20:24:39 Rows err: ORA-01001: invalid cursor
statement failed in:  13.365707303s
2024/01/29 20:24:39 Rows err: ORA-01001: invalid cursor
statement failed in:  13.115971611s
2024/01/29 20:24:40 Rows err: ORA-01001: invalid cursor
statement failed in:  12.865368855s
2024/01/29 20:24:40 Rows err: ORA-01001: invalid cursor
statement failed in:  12.615023489s
2024/01/29 20:24:40 Rows err: ORA-01001: invalid cursor
statement failed in:  12.365609053s
2024/01/29 20:24:40 Rows err: ORA-01001: invalid cursor
statement failed in:  12.115340549s
2024/01/29 20:24:41 Rows err: ORA-01001: invalid cursor
statement failed in:  11.865191884s

sijms commented 6 months ago

this is the code I use

package main

import (
    "database/sql"
    _ "github.com/sijms/go-ora/v2"
    "log"
    "sync"
    "time"

    //toxiproxy "github.com/Shopify/toxiproxy/client"
    "os"
)

func query(db *sql.DB, index int, wg *sync.WaitGroup) {
    defer wg.Done()
    log.Printf("start query #%d\n", index)
    tx, err := db.Begin()
    if err != nil {
        log.Printf("error in begin trans for query #%d: %v\n", index, err)
        return
    }
    var result uint64
    err = tx.QueryRow("SELECT 18446744073709551615 FROM DUAL").Scan(&result)
    if err != nil {
        log.Printf("error in query #%d: %v\n", index, err)
        return
    }
    err = tx.Commit()
    if err != nil {
        log.Printf("error in commit for query #%d: %v\n", index, err)
        return
    }
    log.Printf("result for query #%d: %d\n", index, result)
}
func main() {
    db, err := sql.Open("oracle", os.Getenv("DSN"))
    if err != nil {
        log.Println("can't open db: ", err)
        return
    }
    defer func() {
        err = db.Close()
        if err != nil {
            log.Println("can't close db: ", err)
        }
    }()
    wg := &sync.WaitGroup{}
    for x := 0; x < 5; x++ {
        wg.Add(1)
        go query(db, x+1, wg)
                // tested also without sleep same result
        time.Sleep(1 * time.Second)
    }
    wg.Wait()
}

and this is the result

2024/01/29 23:04:41 start query #1
2024/01/29 23:04:42 start query #2
2024/01/29 23:04:43 start query #3
2024/01/29 23:04:44 start query #4
2024/01/29 23:04:45 start query #5
2024/01/29 23:06:03 result for query #1: 18446744073709551615
2024/01/29 23:06:04 result for query #2: 18446744073709551615
2024/01/29 23:06:05 result for query #3: 18446744073709551615
2024/01/29 23:06:06 result for query #4: 18446744073709551615
2024/01/29 23:06:07 result for query #5: 18446744073709551615

as you can see the difference between start and end for each query is about 1 minute and 20 seconds and this is normal because connection is multi-step process and toxiproxy make delay 5 seconds for each step

no waiting occur between go-routines

dargi98 commented 6 months ago

But in my code, the first requests took ~5s, how could they work then?

And also, why I get the invalid cursor errors?

sijms commented 6 months ago

I test your code and I get the following result:

stmt prepared in:  1m8.245632401s
statement done in:  6.328794948s
statement done in:  6.140061228s
statement done in:  6.523442681s
statement done in:  6.102066788s
2024/01/30 01:02:03 Rows err: read tcp [::1]:50176->[::1]:1522: i/o timeout
statement failed in:  30.000839852s
2024/01/30 01:02:03 Rows err: read tcp [::1]:50178->[::1]:1522: i/o timeout
statement failed in:  30.000138372s
2024/01/30 01:02:03 Rows err: read tcp [::1]:50180->[::1]:1522: i/o timeout
statement failed in:  30.000459547s
2024/01/30 01:02:04 Rows err: read tcp [::1]:50182->[::1]:1522: i/o timeout
statement failed in:  30.000186707s
2024/01/30 01:02:04 Rows err: read tcp [::1]:50184->[::1]:1522: i/o timeout
statement failed in:  30.000196741s
2024/01/30 01:02:04 Rows err: read tcp [::1]:50186->[::1]:1522: i/o timeout
statement failed in:  30.000112941s
2024/01/30 01:02:04 Rows err: read tcp [::1]:50188->[::1]:1522: i/o timeout
statement failed in:  30.000073124s
statement done in:  6.131562949s
2024/01/30 01:02:05 Rows err: read tcp [::1]:50190->[::1]:1522: i/o timeout
statement failed in:  30.000222789s
2024/01/30 01:02:05 Rows err: read tcp [::1]:50192->[::1]:1522: i/o timeout
statement failed in:  30.000159735s
2024/01/30 01:02:05 Rows err: read tcp [::1]:50194->[::1]:1522: i/o timeout
statement failed in:  30.000131664s
2024/01/30 01:02:05 Rows err: read tcp [::1]:50196->[::1]:1522: i/o timeout
statement failed in:  30.000079306s
2024/01/30 01:02:06 Rows err: read tcp [::1]:50198->[::1]:1522: i/o timeout
statement failed in:  30.000163905s
2024/01/30 01:02:06 Rows err: read tcp [::1]:50200->[::1]:1522: i/o timeout
statement failed in:  30.000138342s
2024/01/30 01:02:06 Rows err: read tcp [::1]:50202->[::1]:1522: i/o timeout
statement failed in:  30.000129405s
2024/01/30 01:02:06 Rows err: read tcp [::1]:50204->[::1]:1522: i/o timeout
statement failed in:  30.000138547s
2024/01/30 01:02:07 Rows err: read tcp [::1]:50206->[::1]:1522: i/o timeout
statement failed in:  30.000063101s
2024/01/30 01:02:07 Rows err: read tcp [::1]:50208->[::1]:1522: i/o timeout
statement failed in:  30.000123928s
2024/01/30 01:02:07 Rows err: read tcp [::1]:50210->[::1]:1522: i/o timeout
statement failed in:  30.000067623s
2024/01/30 01:02:07 Rows err: read tcp [::1]:50212->[::1]:1522: i/o timeout
statement failed in:  30.000112196s
2024/01/30 01:02:08 Rows err: read tcp [::1]:50214->[::1]:1522: i/o timeout
statement failed in:  30.000296898s
2024/01/30 01:02:08 Rows err: read tcp [::1]:50216->[::1]:1522: i/o timeout
statement failed in:  30.000087121s
2024/01/30 01:02:08 Rows err: read tcp [::1]:50218->[::1]:1522: i/o timeout
statement failed in:  30.000091175s
2024/01/30 01:02:08 Rows err: read tcp [::1]:50220->[::1]:1522: i/o timeout
statement failed in:  30.000092089s
2024/01/30 01:02:09 Rows err: read tcp [::1]:50222->[::1]:1522: i/o timeout
statement failed in:  30.0001276s
2024/01/30 01:02:09 Rows err: read tcp [::1]:50224->[::1]:1522: i/o timeout
statement failed in:  30.00013249s
2024/01/30 01:02:09 Rows err: read tcp [::1]:50226->[::1]:1522: i/o timeout
statement failed in:  30.000180774s
2024/01/30 01:02:10 Rows err: read tcp [::1]:50228->[::1]:1522: i/o timeout
statement failed in:  30.000096298s
2024/01/30 01:02:10 Rows err: read tcp [::1]:50230->[::1]:1522: i/o timeout
statement failed in:  30.000102481s
2024/01/30 01:02:10 Rows err: read tcp [::1]:50232->[::1]:1522: i/o timeout
statement failed in:  30.000133483s
2024/01/30 01:02:10 Rows err: read tcp [::1]:50234->[::1]:1522: i/o timeout
statement failed in:  30.00009444s
2024/01/30 01:02:11 Rows err: read tcp [::1]:50236->[::1]:1522: i/o timeout
statement failed in:  30.001950952s
statement done in:  6.218362938s
2024/01/30 01:02:11 Rows err: read tcp [::1]:50238->[::1]:1522: i/o timeout
statement failed in:  30.000137388s
2024/01/30 01:02:11 Rows err: read tcp [::1]:50240->[::1]:1522: i/o timeout
statement failed in:  30.000128665s
2024/01/30 01:02:11 Rows err: read tcp [::1]:50242->[::1]:1522: i/o timeout
statement failed in:  30.000326975s
2024/01/30 01:02:12 Rows err: read tcp [::1]:50244->[::1]:1522: i/o timeout
statement failed in:  30.000143777s
2024/01/30 01:02:12 Rows err: read tcp [::1]:50246->[::1]:1522: i/o timeout
statement failed in:  30.000055547s
2024/01/30 01:02:12 Rows err: read tcp [::1]:50248->[::1]:1522: i/o timeout
statement failed in:  30.000136224s
2024/01/30 01:02:12 Rows err: read tcp [::1]:50250->[::1]:1522: i/o timeout
statement failed in:  30.000208727s
2024/01/30 01:02:13 Rows err: read tcp [::1]:50252->[::1]:1522: i/o timeout
statement failed in:  30.000378295s
2024/01/30 01:02:13 Rows err: read tcp [::1]:50254->[::1]:1522: i/o timeout
statement failed in:  30.000147241s
2024/01/30 01:02:13 Rows err: read tcp [::1]:50256->[::1]:1522: i/o timeout
statement failed in:  30.000256313s
2024/01/30 01:02:13 Rows err: read tcp [::1]:50258->[::1]:1522: i/o timeout
statement failed in:  30.000756057s
2024/01/30 01:02:14 Rows err: read tcp [::1]:50260->[::1]:1522: i/o timeout
statement failed in:  30.000589526s
2024/01/30 01:02:14 Rows err: read tcp [::1]:50262->[::1]:1522: i/o timeout
statement failed in:  30.000093569s
2024/01/30 01:02:14 Rows err: read tcp [::1]:50264->[::1]:1522: i/o timeout
statement failed in:  30.000104042s
2024/01/30 01:02:14 Rows err: read tcp [::1]:50266->[::1]:1522: i/o timeout
statement failed in:  30.00010362s
2024/01/30 01:02:15 Rows err: read tcp [::1]:50268->[::1]:1522: i/o timeout
statement failed in:  30.000139374s
2024/01/30 01:02:15 Rows err: read tcp [::1]:50272->[::1]:1522: i/o timeout
statement failed in:  30.000109143s
2024/01/30 01:02:16 Rows err: read tcp [::1]:50274->[::1]:1522: i/o timeout
statement failed in:  30.000112234s
2024/01/30 01:02:16 Rows err: read tcp [::1]:50276->[::1]:1522: i/o timeout
statement failed in:  30.000181449s
2024/01/30 01:02:16 Rows err: read tcp [::1]:50278->[::1]:1522: i/o timeout
statement failed in:  30.000866921s
2024/01/30 01:02:16 Rows err: read tcp [::1]:50280->[::1]:1522: i/o timeout
statement failed in:  30.00005071s
2024/01/30 01:02:17 Rows err: read tcp [::1]:50282->[::1]:1522: i/o timeout
statement failed in:  30.000107245s
2024/01/30 01:02:17 Rows err: read tcp [::1]:50284->[::1]:1522: i/o timeout
statement failed in:  30.000373203s
statement done in:  6.193549543s
2024/01/30 01:02:17 Rows err: read tcp [::1]:50286->[::1]:1522: i/o timeout
statement failed in:  30.000119553s
2024/01/30 01:02:17 Rows err: read tcp [::1]:50288->[::1]:1522: i/o timeout
statement failed in:  30.000683133s
2024/01/30 01:02:18 Rows err: read tcp [::1]:50290->[::1]:1522: i/o timeout
statement failed in:  30.000089257s
2024/01/30 01:02:18 Rows err: read tcp [::1]:50292->[::1]:1522: i/o timeout
statement failed in:  30.000175009s
2024/01/30 01:02:18 Rows err: read tcp [::1]:50294->[::1]:1522: i/o timeout
statement failed in:  30.000117828s
2024/01/30 01:02:18 Rows err: read tcp [::1]:50296->[::1]:1522: i/o timeout
statement failed in:  30.000109641s
2024/01/30 01:02:19 Rows err: read tcp [::1]:50298->[::1]:1522: i/o timeout
statement failed in:  30.00056103s
2024/01/30 01:02:19 Rows err: read tcp [::1]:50300->[::1]:1522: i/o timeout
statement failed in:  30.000031719s
2024/01/30 01:02:19 Rows err: read tcp [::1]:50302->[::1]:1522: i/o timeout
statement failed in:  30.000114788s
2024/01/30 01:02:19 Rows err: read tcp [::1]:50304->[::1]:1522: i/o timeout
statement failed in:  30.00009155s
2024/01/30 01:02:20 Rows err: read tcp [::1]:50306->[::1]:1522: i/o timeout
statement failed in:  30.000172108s
2024/01/30 01:02:20 Rows err: read tcp [::1]:50308->[::1]:1522: i/o timeout
statement failed in:  30.000101538s
2024/01/30 01:02:20 Rows err: read tcp [::1]:50310->[::1]:1522: i/o timeout
statement failed in:  30.000081558s
2024/01/30 01:02:20 Rows err: read tcp [::1]:50312->[::1]:1522: i/o timeout
statement failed in:  30.00008285s
2024/01/30 01:02:21 Rows err: read tcp [::1]:50314->[::1]:1522: i/o timeout
statement failed in:  30.000078755s
2024/01/30 01:02:21 Rows err: read tcp [::1]:50316->[::1]:1522: i/o timeout
statement failed in:  30.00010114s
2024/01/30 01:02:21 Rows err: read tcp [::1]:50318->[::1]:1522: i/o timeout
statement failed in:  30.000080834s
2024/01/30 01:02:21 Rows err: read tcp [::1]:50320->[::1]:1522: i/o timeout
statement failed in:  30.000252645s
2024/01/30 01:02:22 Rows err: read tcp [::1]:50322->[::1]:1522: i/o timeout
statement failed in:  30.000070395s
2024/01/30 01:02:22 Rows err: read tcp [::1]:50324->[::1]:1522: i/o timeout
statement failed in:  30.000512289s
2024/01/30 01:02:22 Rows err: read tcp [::1]:50326->[::1]:1522: i/o timeout
statement failed in:  30.000059706s
2024/01/30 01:02:23 Rows err: read tcp [::1]:50328->[::1]:1522: i/o timeout
statement failed in:  30.000133009s
2024/01/30 01:02:23 Rows err: read tcp [::1]:50330->[::1]:1522: i/o timeout
statement failed in:  30.00008677s
2024/01/30 01:02:23 Rows err: read tcp [::1]:50332->[::1]:1522: i/o timeout
statement failed in:  30.00014404s
statement done in:  6.241708065s
2024/01/30 01:02:23 Rows err: read tcp [::1]:50334->[::1]:1522: i/o timeout
statement failed in:  30.000192021s
2024/01/30 01:02:24 Rows err: read tcp [::1]:50336->[::1]:1522: i/o timeout
statement failed in:  30.000488672s
2024/01/30 01:02:24 Rows err: read tcp [::1]:50338->[::1]:1522: i/o timeout
statement failed in:  30.000075213s
2024/01/30 01:02:24 Rows err: read tcp [::1]:50340->[::1]:1522: i/o timeout
statement failed in:  30.000164674s
2024/01/30 01:02:24 Rows err: read tcp [::1]:50342->[::1]:1522: i/o timeout
statement failed in:  30.000117301s
2024/01/30 01:02:25 Rows err: read tcp [::1]:50344->[::1]:1522: i/o timeout
statement failed in:  30.000165058s
2024/01/30 01:02:25 Rows err: read tcp [::1]:50346->[::1]:1522: i/o timeout
statement failed in:  30.000707794s
2024/01/30 01:02:25 Rows err: read tcp [::1]:50348->[::1]:1522: i/o timeout
statement failed in:  30.000498064s
2024/01/30 01:02:25 Rows err: read tcp [::1]:50350->[::1]:1522: i/o timeout
statement failed in:  30.000274865s
2024/01/30 01:02:26 Rows err: EOF
statement failed in:  40.734203835s
2024/01/30 01:02:26 Rows err: read tcp [::1]:50352->[::1]:1522: i/o timeout
statement failed in:  30.000109877s
2024/01/30 01:02:26 Rows err: read tcp [::1]:50354->[::1]:1522: i/o timeout
statement failed in:  30.000737918s
2024/01/30 01:02:26 Rows err: read tcp [::1]:50356->[::1]:1522: i/o timeout
statement failed in:  30.000423395s
2024/01/30 01:02:26 Rows err: read tcp [::1]:50358->[::1]:1522: i/o timeout
statement failed in:  30.000759114s
2024/01/30 01:02:27 Rows err: read tcp [::1]:50360->[::1]:1522: i/o timeout
statement failed in:  30.000229556s
2024/01/30 01:02:27 Rows err: read tcp [::1]:50362->[::1]:1522: i/o timeout
statement failed in:  30.000077573s
2024/01/30 01:02:27 Rows err: read tcp [::1]:50364->[::1]:1522: i/o timeout
statement failed in:  30.000126091s
2024/01/30 01:02:27 Rows err: read tcp [::1]:50366->[::1]:1522: i/o timeout
statement failed in:  30.000188403s
2024/01/30 01:02:28 Rows err: read tcp [::1]:50368->[::1]:1522: i/o timeout
statement failed in:  30.000099689s
2024/01/30 01:02:28 Rows err: read tcp [::1]:50370->[::1]:1522: i/o timeout
statement failed in:  30.00037488s
2024/01/30 01:02:28 Rows err: read tcp [::1]:50372->[::1]:1522: i/o timeout
statement failed in:  30.000050458s
2024/01/30 01:02:29 Rows err: read tcp [::1]:50374->[::1]:1522: i/o timeout
statement failed in:  30.000108308s
2024/01/30 01:02:29 Rows err: read tcp [::1]:50376->[::1]:1522: i/o timeout
statement failed in:  30.000122031s
2024/01/30 01:02:29 Rows err: read tcp [::1]:50378->[::1]:1522: i/o timeout
statement failed in:  30.000255766s
2024/01/30 01:02:29 Rows err: read tcp [::1]:50380->[::1]:1522: i/o timeout
statement failed in:  30.000427189s
2024/01/30 01:02:30 Rows err: read tcp [::1]:50382->[::1]:1522: i/o timeout
statement failed in:  30.000220028s
statement done in:  6.308267814s
2024/01/30 01:02:30 Rows err: read tcp [::1]:50384->[::1]:1522: i/o timeout
statement failed in:  30.000079131s
2024/01/30 01:02:30 Rows err: read tcp [::1]:50386->[::1]:1522: i/o timeout
statement failed in:  30.000738571s
2024/01/30 01:02:30 Rows err: read tcp [::1]:50388->[::1]:1522: i/o timeout
statement failed in:  30.000066478s
2024/01/30 01:02:31 Rows err: read tcp [::1]:50390->[::1]:1522: i/o timeout
statement failed in:  30.000181308s
2024/01/30 01:02:31 Rows err: read tcp [::1]:50392->[::1]:1522: i/o timeout
statement failed in:  30.000121243s
2024/01/30 01:02:31 Rows err: read tcp [::1]:50394->[::1]:1522: i/o timeout
statement failed in:  30.000076445s
2024/01/30 01:02:31 Rows err: read tcp [::1]:50396->[::1]:1522: i/o timeout
statement failed in:  30.000177627s
2024/01/30 01:02:32 Rows err: read tcp [::1]:50398->[::1]:1522: i/o timeout
statement failed in:  30.000149979s
2024/01/30 01:02:32 Rows err: read tcp [::1]:50400->[::1]:1522: i/o timeout
statement failed in:  30.00017744s
2024/01/30 01:02:32 Rows err: read tcp [::1]:50402->[::1]:1522: i/o timeout
statement failed in:  30.000045853s
2024/01/30 01:02:32 Rows err: read tcp [::1]:50404->[::1]:1522: i/o timeout
statement failed in:  30.000147502s
2024/01/30 01:02:33 Rows err: read tcp [::1]:50406->[::1]:1522: i/o timeout
statement failed in:  30.000067556s
2024/01/30 01:02:33 Rows err: read tcp [::1]:50408->[::1]:1522: i/o timeout
statement failed in:  30.000204682s
2024/01/30 01:02:33 Rows err: read tcp [::1]:50410->[::1]:1522: i/o timeout
statement failed in:  30.000120342s
2024/01/30 01:02:33 Rows err: read tcp [::1]:50412->[::1]:1522: i/o timeout
statement failed in:  30.000209197s
2024/01/30 01:02:34 Rows err: read tcp [::1]:50414->[::1]:1522: i/o timeout
statement failed in:  30.000139686s
2024/01/30 01:02:34 Rows err: read tcp [::1]:50416->[::1]:1522: i/o timeout
statement failed in:  30.000162475s
2024/01/30 01:02:34 Rows err: read tcp [::1]:50418->[::1]:1522: i/o timeout
statement failed in:  30.000689387s
2024/01/30 01:02:34 Rows err: read tcp [::1]:50420->[::1]:1522: i/o timeout
statement failed in:  30.000235784s
2024/01/30 01:02:35 Rows err: read tcp [::1]:50422->[::1]:1522: i/o timeout
statement failed in:  30.000043086s
2024/01/30 01:02:35 Rows err: read tcp [::1]:50424->[::1]:1522: i/o timeout
statement failed in:  30.000522617s
2024/01/30 01:02:35 Rows err: read tcp [::1]:50426->[::1]:1522: i/o timeout
statement failed in:  30.000121032s
2024/01/30 01:02:36 Rows err: read tcp [::1]:50428->[::1]:1522: i/o timeout
statement failed in:  30.000164297s
2024/01/30 01:02:36 Rows err: read tcp [::1]:50430->[::1]:1522: i/o timeout
statement failed in:  30.040404495s
2024/01/30 01:02:36 Rows err: read tcp [::1]:50432->[::1]:1522: i/o timeout
statement failed in:  30.000558569s
statement done in:  6.27489249s
2024/01/30 01:02:36 Rows err: read tcp [::1]:50434->[::1]:1522: i/o timeout
statement failed in:  30.000324135s
2024/01/30 01:02:37 Rows err: read tcp [::1]:50436->[::1]:1522: i/o timeout
statement failed in:  30.000111698s
2024/01/30 01:02:37 Rows err: read tcp [::1]:50438->[::1]:1522: i/o timeout
statement failed in:  30.00028236s
2024/01/30 01:02:37 Rows err: read tcp [::1]:50440->[::1]:1522: i/o timeout
statement failed in:  30.000107959s
2024/01/30 01:02:37 Rows err: read tcp [::1]:50442->[::1]:1522: i/o timeout
statement failed in:  30.000194641s
2024/01/30 01:02:38 Rows err: read tcp [::1]:50444->[::1]:1522: i/o timeout
statement failed in:  30.000072546s
2024/01/30 01:02:38 Rows err: read tcp [::1]:50446->[::1]:1522: i/o timeout
statement failed in:  30.000188978s
2024/01/30 01:02:38 Rows err: read tcp [::1]:50448->[::1]:1522: i/o timeout
statement failed in:  30.000097273s
2024/01/30 01:02:38 Rows err: read tcp [::1]:50450->[::1]:1522: i/o timeout
statement failed in:  30.000124822s
2024/01/30 01:02:39 Rows err: read tcp [::1]:50452->[::1]:1522: i/o timeout
statement failed in:  30.00008066s
2024/01/30 01:02:39 Rows err: read tcp [::1]:50454->[::1]:1522: i/o timeout
statement failed in:  30.000073763s
2024/01/30 01:02:39 Rows err: read tcp [::1]:50456->[::1]:1522: i/o timeout
statement failed in:  30.000132557s
2024/01/30 01:02:39 Rows err: read tcp [::1]:50458->[::1]:1522: i/o timeout
statement failed in:  30.000166524s
2024/01/30 01:02:40 Rows err: read tcp [::1]:50460->[::1]:1522: i/o timeout
statement failed in:  30.000084551s
2024/01/30 01:02:40 Rows err: read tcp [::1]:50462->[::1]:1522: i/o timeout
statement failed in:  30.000131726s
2024/01/30 01:02:40 Rows err: read tcp [::1]:50464->[::1]:1522: i/o timeout
statement failed in:  30.000833842s
2024/01/30 01:02:40 Rows err: read tcp [::1]:50466->[::1]:1522: i/o timeout
statement failed in:  30.000049202s
2024/01/30 01:02:41 Rows err: read tcp [::1]:50468->[::1]:1522: i/o timeout
statement failed in:  30.000656882s
2024/01/30 01:02:41 Rows err: read tcp [::1]:50470->[::1]:1522: i/o timeout
statement failed in:  30.001429971s
2024/01/30 01:02:41 Rows err: read tcp [::1]:50472->[::1]:1522: i/o timeout
statement failed in:  30.000732016s
2024/01/30 01:02:42 Rows err: read tcp [::1]:50474->[::1]:1522: i/o timeout
statement failed in:  30.001075304s
2024/01/30 01:02:42 Rows err: read tcp [::1]:50476->[::1]:1522: i/o timeout
statement failed in:  30.000122354s
2024/01/30 01:02:42 Rows err: read tcp [::1]:50478->[::1]:1522: i/o timeout
statement failed in:  30.000175864s
2024/01/30 01:02:42 Rows err: read tcp [::1]:50480->[::1]:1522: i/o timeout
statement failed in:  30.000322649s
statement done in:  6.160623086s
2024/01/30 01:02:43 Rows err: read tcp [::1]:50482->[::1]:1522: i/o timeout
statement failed in:  30.000149795s
2024/01/30 01:02:43 Rows err: read tcp [::1]:50484->[::1]:1522: i/o timeout
statement failed in:  30.000212531s
2024/01/30 01:02:43 Rows err: read tcp [::1]:50486->[::1]:1522: i/o timeout
statement failed in:  30.00013722s
2024/01/30 01:02:43 Rows err: read tcp [::1]:50488->[::1]:1522: i/o timeout
statement failed in:  30.000067462s
2024/01/30 01:02:44 Rows err: read tcp [::1]:50490->[::1]:1522: i/o timeout
statement failed in:  30.000197276s
2024/01/30 01:02:44 Rows err: read tcp [::1]:50492->[::1]:1522: i/o timeout
statement failed in:  30.000193657s
2024/01/30 01:02:44 Rows err: read tcp [::1]:50494->[::1]:1522: i/o timeout
statement failed in:  30.000124044s
2024/01/30 01:02:44 Rows err: read tcp [::1]:50496->[::1]:1522: i/o timeout
statement failed in:  30.000158253s
2024/01/30 01:02:45 Rows err: read tcp [::1]:50498->[::1]:1522: i/o timeout
statement failed in:  30.001685687s
2024/01/30 01:02:45 Rows err: read tcp [::1]:50500->[::1]:1522: i/o timeout
statement failed in:  30.00059829s
2024/01/30 01:02:45 Rows err: read tcp [::1]:50502->[::1]:1522: i/o timeout
statement failed in:  30.000066264s
2024/01/30 01:02:45 Rows err: read tcp [::1]:50504->[::1]:1522: i/o timeout
statement failed in:  30.000121997s
2024/01/30 01:02:46 Rows err: read tcp [::1]:50506->[::1]:1522: i/o timeout
statement failed in:  30.000040116s
2024/01/30 01:02:46 Rows err: read tcp [::1]:50508->[::1]:1522: i/o timeout
statement failed in:  30.000148619s
2024/01/30 01:02:46 Rows err: read tcp [::1]:50510->[::1]:1522: i/o timeout
statement failed in:  30.00012586s
2024/01/30 01:02:46 Rows err: read tcp [::1]:50512->[::1]:1522: i/o timeout
statement failed in:  30.00008316s
2024/01/30 01:02:47 Rows err: read tcp [::1]:50514->[::1]:1522: i/o timeout
statement failed in:  30.000159623s
2024/01/30 01:02:47 Rows err: read tcp [::1]:50516->[::1]:1522: i/o timeout
statement failed in:  30.000136116s
2024/01/30 01:02:47 Rows err: read tcp [::1]:50518->[::1]:1522: i/o timeout
statement failed in:  30.000305195s
2024/01/30 01:02:48 Rows err: read tcp [::1]:50520->[::1]:1522: i/o timeout
statement failed in:  30.000449895s
2024/01/30 01:02:48 Rows err: read tcp [::1]:50522->[::1]:1522: i/o timeout
statement failed in:  30.000216108s
2024/01/30 01:02:48 Rows err: read tcp [::1]:50524->[::1]:1522: i/o timeout
statement failed in:  30.000287179s
2024/01/30 01:02:48 Rows err: read tcp [::1]:50526->[::1]:1522: i/o timeout
statement failed in:  30.000088367s
2024/01/30 01:02:49 Rows err: read tcp [::1]:50528->[::1]:1522: i/o timeout
statement failed in:  30.000079728s
statement done in:  6.154966375s
2024/01/30 01:02:49 Rows err: read tcp [::1]:50530->[::1]:1522: i/o timeout
statement failed in:  30.000253489s
2024/01/30 01:02:49 Rows err: read tcp [::1]:50532->[::1]:1522: i/o timeout
statement failed in:  30.000208341s
2024/01/30 01:02:49 Rows err: read tcp [::1]:50534->[::1]:1522: i/o timeout
statement failed in:  30.0000943s
2024/01/30 01:02:50 Rows err: read tcp [::1]:50536->[::1]:1522: i/o timeout
statement failed in:  30.002863282s
2024/01/30 01:02:50 Rows err: read tcp [::1]:50538->[::1]:1522: i/o timeout
statement failed in:  30.000099659s
2024/01/30 01:02:50 Rows err: read tcp [::1]:50540->[::1]:1522: i/o timeout
statement failed in:  30.000082837s
2024/01/30 01:02:50 Rows err: read tcp [::1]:50542->[::1]:1522: i/o timeout
statement failed in:  30.000610064s
2024/01/30 01:02:51 Rows err: read tcp [::1]:50544->[::1]:1522: i/o timeout
statement failed in:  30.000828593s
2024/01/30 01:02:51 Rows err: read tcp [::1]:50546->[::1]:1522: i/o timeout
statement failed in:  30.002355321s
2024/01/30 01:02:51 Rows err: read tcp [::1]:50548->[::1]:1522: i/o timeout
statement failed in:  30.00036317s
2024/01/30 01:02:51 Rows err: read tcp [::1]:50550->[::1]:1522: i/o timeout
statement failed in:  30.000728469s
2024/01/30 01:02:52 Rows err: read tcp [::1]:50551->[::1]:1522: i/o timeout
statement failed in:  30.001923196s
2024/01/30 01:02:52 Rows err: read tcp [::1]:50554->[::1]:1522: i/o timeout
statement failed in:  30.001429663s
2024/01/30 01:02:52 Rows err: read tcp [::1]:50556->[::1]:1522: i/o timeout
statement failed in:  30.000252886s
2024/01/30 01:02:52 Rows err: read tcp [::1]:50558->[::1]:1522: i/o timeout
statement failed in:  30.000243172s
2024/01/30 01:02:53 Rows err: read tcp [::1]:50560->[::1]:1522: i/o timeout
statement failed in:  30.000091646s
2024/01/30 01:02:53 Rows err: read tcp [::1]:50562->[::1]:1522: i/o timeout
statement failed in:  30.000307746s
2024/01/30 01:02:53 Rows err: read tcp [::1]:50564->[::1]:1522: i/o timeout
statement failed in:  30.000123804s
2024/01/30 01:02:54 Rows err: read tcp [::1]:50566->[::1]:1522: i/o timeout
statement failed in:  30.000254794s
2024/01/30 01:02:54 Rows err: read tcp [::1]:50568->[::1]:1522: i/o timeout
statement failed in:  30.000406504s
2024/01/30 01:02:54 Rows err: read tcp [::1]:50570->[::1]:1522: i/o timeout
statement failed in:  30.000105993s
2024/01/30 01:02:54 Rows err: read tcp [::1]:50572->[::1]:1522: i/o timeout
statement failed in:  30.000092737s
2024/01/30 01:02:55 Rows err: read tcp [::1]:50574->[::1]:1522: i/o timeout
statement failed in:  30.00010635s
2024/01/30 01:02:55 Rows err: read tcp [::1]:50576->[::1]:1522: i/o timeout
statement failed in:  30.000651656s
statement done in:  6.164090994s
2024/01/30 01:02:55 Rows err: read tcp [::1]:50578->[::1]:1522: i/o timeout
statement failed in:  30.000840124s
2024/01/30 01:02:55 Rows err: read tcp [::1]:50580->[::1]:1522: i/o timeout
statement failed in:  30.000304721s
2024/01/30 01:02:56 Rows err: read tcp [::1]:50582->[::1]:1522: i/o timeout
statement failed in:  30.000188652s
2024/01/30 01:02:56 Rows err: read tcp [::1]:50584->[::1]:1522: i/o timeout
statement failed in:  30.000103516s
2024/01/30 01:02:56 Rows err: read tcp [::1]:50586->[::1]:1522: i/o timeout
statement failed in:  30.004668015s
2024/01/30 01:02:56 Rows err: read tcp [::1]:50588->[::1]:1522: i/o timeout
statement failed in:  30.021670143s
2024/01/30 01:02:57 Rows err: read tcp [::1]:50590->[::1]:1522: i/o timeout
statement failed in:  30.000381069s
2024/01/30 01:02:57 Rows err: read tcp [::1]:50592->[::1]:1522: i/o timeout
statement failed in:  30.000735881s
2024/01/30 01:02:57 Rows err: read tcp [::1]:50594->[::1]:1522: i/o timeout
statement failed in:  30.000244152s
2024/01/30 01:02:57 Rows err: read tcp [::1]:50596->[::1]:1522: i/o timeout
statement failed in:  30.000398801s
2024/01/30 01:02:58 Rows err: read tcp [::1]:50598->[::1]:1522: i/o timeout
statement failed in:  30.00007298s
2024/01/30 01:02:58 Rows err: read tcp [::1]:50599->[::1]:1522: i/o timeout
statement failed in:  30.000099013s
2024/01/30 01:02:58 Rows err: read tcp [::1]:50602->[::1]:1522: i/o timeout
statement failed in:  30.000078424s
2024/01/30 01:02:58 Rows err: read tcp [::1]:50603->[::1]:1522: i/o timeout
statement failed in:  30.000633471s
2024/01/30 01:02:59 Rows err: read tcp [::1]:50606->[::1]:1522: i/o timeout
statement failed in:  30.000242691s
2024/01/30 01:02:59 Rows err: read tcp [::1]:50607->[::1]:1522: i/o timeout
statement failed in:  30.000108162s
2024/01/30 01:02:59 Rows err: read tcp [::1]:50610->[::1]:1522: i/o timeout
statement failed in:  30.000657071s
2024/01/30 01:02:59 Rows err: read tcp [::1]:50612->[::1]:1522: i/o timeout
statement failed in:  30.000129236s
2024/01/30 01:03:00 Rows err: read tcp [::1]:50614->[::1]:1522: i/o timeout
statement failed in:  30.000113844s
2024/01/30 01:03:00 Rows err: read tcp [::1]:50616->[::1]:1522: i/o timeout
statement failed in:  30.000187655s
2024/01/30 01:03:01 Rows err: read tcp [::1]:50618->[::1]:1522: i/o timeout
statement failed in:  30.050455198s
2024/01/30 01:03:01 Rows err: read tcp [::1]:50620->[::1]:1522: i/o timeout
statement failed in:  30.003096489s
2024/01/30 01:03:01 Rows err: read tcp [::1]:50622->[::1]:1522: i/o timeout
statement failed in:  30.000463984s
2024/01/30 01:03:01 Rows err: read tcp [::1]:50624->[::1]:1522: i/o timeout
statement failed in:  30.007299417s
statement done in:  6.154530249s
2024/01/30 01:03:01 Rows err: read tcp [::1]:50626->[::1]:1522: i/o timeout
statement failed in:  30.001167069s
2024/01/30 01:03:02 Rows err: read tcp [::1]:50628->[::1]:1522: i/o timeout
statement failed in:  30.000095254s
2024/01/30 01:03:02 Rows err: read tcp [::1]:50630->[::1]:1522: i/o timeout
statement failed in:  30.000137877s
2024/01/30 01:03:02 Rows err: read tcp [::1]:50632->[::1]:1522: i/o timeout
statement failed in:  30.000278715s
2024/01/30 01:03:02 Rows err: read tcp [::1]:50634->[::1]:1522: i/o timeout
statement failed in:  30.000187522s
2024/01/30 01:03:03 Rows err: read tcp [::1]:50636->[::1]:1522: i/o timeout
statement failed in:  30.00457702s
2024/01/30 01:03:03 Rows err: read tcp [::1]:50638->[::1]:1522: i/o timeout
statement failed in:  30.012742953s
2024/01/30 01:03:03 Rows err: read tcp [::1]:50640->[::1]:1522: i/o timeout
statement failed in:  30.003842848s
2024/01/30 01:03:03 Rows err: read tcp [::1]:50642->[::1]:1522: i/o timeout
statement failed in:  30.000140977s
2024/01/30 01:03:04 Rows err: read tcp [::1]:50644->[::1]:1522: i/o timeout
statement failed in:  30.000141342s
2024/01/30 01:03:04 Rows err: read tcp [::1]:50646->[::1]:1522: i/o timeout
statement failed in:  30.000177672s
2024/01/30 01:03:04 Rows err: read tcp [::1]:50648->[::1]:1522: i/o timeout
statement failed in:  30.000182929s
2024/01/30 01:03:04 Rows err: read tcp [::1]:50650->[::1]:1522: i/o timeout
statement failed in:  30.000212935s
2024/01/30 01:03:05 Rows err: read tcp [::1]:50652->[::1]:1522: i/o timeout
statement failed in:  30.000977167s
2024/01/30 01:03:05 Rows err: read tcp [::1]:50654->[::1]:1522: i/o timeout
statement failed in:  30.000483651s
2024/01/30 01:03:05 Rows err: read tcp [::1]:50656->[::1]:1522: i/o timeout
statement failed in:  30.000072253s
2024/01/30 01:03:05 Rows err: read tcp [::1]:50658->[::1]:1522: i/o timeout
statement failed in:  30.000153795s
2024/01/30 01:03:06 Rows err: read tcp [::1]:50660->[::1]:1522: i/o timeout
statement failed in:  30.000093448s
2024/01/30 01:03:06 Rows err: read tcp [::1]:50662->[::1]:1522: i/o timeout
statement failed in:  30.000478188s
2024/01/30 01:03:06 Rows err: read tcp [::1]:50664->[::1]:1522: i/o timeout
statement failed in:  30.000169901s
2024/01/30 01:03:07 Rows err: read tcp [::1]:50666->[::1]:1522: i/o timeout
statement failed in:  30.000120504s
2024/01/30 01:03:07 Rows err: read tcp [::1]:50668->[::1]:1522: i/o timeout
statement failed in:  30.00059215s
2024/01/30 01:03:07 Rows err: read tcp [::1]:50670->[::1]:1522: i/o timeout
statement failed in:  30.000140373s
2024/01/30 01:03:07 Rows err: read tcp [::1]:50672->[::1]:1522: i/o timeout
statement failed in:  30.000394375s
statement done in:  6.126106248s
2024/01/30 01:03:08 Rows err: read tcp [::1]:50674->[::1]:1522: i/o timeout
statement failed in:  30.000157873s
2024/01/30 01:03:08 Rows err: read tcp [::1]:50676->[::1]:1522: i/o timeout
statement failed in:  30.000153908s
2024/01/30 01:03:08 Rows err: read tcp [::1]:50678->[::1]:1522: i/o timeout
statement failed in:  30.000126798s
2024/01/30 01:03:08 Rows err: read tcp [::1]:50680->[::1]:1522: i/o timeout
statement failed in:  30.00013628s
2024/01/30 01:03:09 Rows err: read tcp [::1]:50682->[::1]:1522: i/o timeout
statement failed in:  30.000146828s
2024/01/30 01:03:09 Rows err: read tcp [::1]:50684->[::1]:1522: i/o timeout
statement failed in:  30.000121711s
2024/01/30 01:03:09 Rows err: read tcp [::1]:50686->[::1]:1522: i/o timeout
statement failed in:  30.000108288s
2024/01/30 01:03:09 Rows err: read tcp [::1]:50688->[::1]:1522: i/o timeout
statement failed in:  30.000109934s
2024/01/30 01:03:10 Rows err: read tcp [::1]:50690->[::1]:1522: i/o timeout
statement failed in:  30.000182727s
2024/01/30 01:03:10 Rows err: read tcp [::1]:50692->[::1]:1522: i/o timeout
statement failed in:  30.000059279s
2024/01/30 01:03:10 Rows err: read tcp [::1]:50694->[::1]:1522: i/o timeout
statement failed in:  30.00011949s
2024/01/30 01:03:10 Rows err: read tcp [::1]:50696->[::1]:1522: i/o timeout
statement failed in:  30.000122076s
2024/01/30 01:03:11 Rows err: read tcp [::1]:50698->[::1]:1522: i/o timeout
statement failed in:  30.000168302s
2024/01/30 01:03:11 Rows err: read tcp [::1]:50700->[::1]:1522: i/o timeout
statement failed in:  30.000136333s
2024/01/30 01:03:11 Rows err: read tcp [::1]:50702->[::1]:1522: i/o timeout
statement failed in:  30.000200957s
2024/01/30 01:03:11 Rows err: read tcp [::1]:50704->[::1]:1522: i/o timeout
statement failed in:  30.000207704s
2024/01/30 01:03:12 Rows err: read tcp [::1]:50706->[::1]:1522: i/o timeout
statement failed in:  30.000208738s
2024/01/30 01:03:12 Rows err: read tcp [::1]:50708->[::1]:1522: i/o timeout
statement failed in:  30.000404344s
2024/01/30 01:03:13 Rows err: read tcp [::1]:50712->[::1]:1522: i/o timeout
statement failed in:  30.000149326s
2024/01/30 01:03:13 Rows err: read tcp [::1]:50716->[::1]:1522: i/o timeout
statement failed in:  30.000137435s
2024/01/30 01:03:14 Rows err: read tcp [::1]:50718->[::1]:1522: i/o timeout
statement failed in:  30.000287922s
2024/01/30 01:03:14 Rows err: read tcp [::1]:50720->[::1]:1522: i/o timeout
statement failed in:  30.000768478s
2024/01/30 01:03:14 Rows err: read tcp [::1]:50722->[::1]:1522: i/o timeout
statement failed in:  30.000643868s
2024/01/30 01:03:15 Rows err: read tcp [::1]:50724->[::1]:1522: i/o timeout
statement failed in:  30.000710202s
statement done in:  6.033844214s
2024/01/30 01:03:15 Rows err: read tcp [::1]:50726->[::1]:1522: i/o timeout
statement failed in:  30.000492024s
2024/01/30 01:03:15 Rows err: read tcp [::1]:50728->[::1]:1522: i/o timeout
statement failed in:  30.000471931s
2024/01/30 01:03:15 Rows err: read tcp [::1]:50730->[::1]:1522: i/o timeout
statement failed in:  30.000962984s
2024/01/30 01:03:16 Rows err: read tcp [::1]:50732->[::1]:1522: i/o timeout
statement failed in:  30.000322299s
2024/01/30 01:03:16 Rows err: read tcp [::1]:50734->[::1]:1522: i/o timeout
statement failed in:  30.000378505s
2024/01/30 01:03:16 Rows err: read tcp [::1]:50736->[::1]:1522: i/o timeout
statement failed in:  30.00033803s
2024/01/30 01:03:16 Rows err: read tcp [::1]:50738->[::1]:1522: i/o timeout
statement failed in:  30.001228437s
2024/01/30 01:03:17 Rows err: read tcp [::1]:50740->[::1]:1522: i/o timeout
statement failed in:  30.000315329s
2024/01/30 01:03:17 Rows err: read tcp [::1]:50742->[::1]:1522: i/o timeout
statement failed in:  30.001008162s
2024/01/30 01:03:17 Rows err: read tcp [::1]:50744->[::1]:1522: i/o timeout
statement failed in:  30.000578001s
2024/01/30 01:03:17 Rows err: read tcp [::1]:50746->[::1]:1522: i/o timeout
statement failed in:  30.000147272s
2024/01/30 01:03:18 Rows err: read tcp [::1]:50748->[::1]:1522: i/o timeout
statement failed in:  30.000115433s
2024/01/30 01:03:18 Rows err: read tcp [::1]:50750->[::1]:1522: i/o timeout
statement failed in:  30.000774223s
2024/01/30 01:03:18 Rows err: read tcp [::1]:50752->[::1]:1522: i/o timeout
statement failed in:  30.00102947s
2024/01/30 01:03:18 Rows err: read tcp [::1]:50754->[::1]:1522: i/o timeout
statement failed in:  30.002505799s
2024/01/30 01:03:19 Rows err: read tcp [::1]:50756->[::1]:1522: i/o timeout
statement failed in:  30.000647325s
2024/01/30 01:03:19 Rows err: read tcp [::1]:50758->[::1]:1522: i/o timeout
statement failed in:  30.00128544s
2024/01/30 01:03:19 Rows err: read tcp [::1]:50760->[::1]:1522: i/o timeout
statement failed in:  30.000907568s
2024/01/30 01:03:19 Rows err: read tcp [::1]:50762->[::1]:1522: i/o timeout
statement failed in:  30.000654869s
2024/01/30 01:03:20 Rows err: read tcp [::1]:50764->[::1]:1522: i/o timeout
statement failed in:  30.00041145s
2024/01/30 01:03:20 Rows err: read tcp [::1]:50766->[::1]:1522: i/o timeout
statement failed in:  30.00020088s
2024/01/30 01:03:20 Rows err: read tcp [::1]:50768->[::1]:1522: i/o timeout
statement failed in:  30.000669323s
2024/01/30 01:03:21 Rows err: read tcp [::1]:50770->[::1]:1522: i/o timeout
statement failed in:  30.000367125s
2024/01/30 01:03:21 Rows err: read tcp [::1]:50772->[::1]:1522: i/o timeout
statement failed in:  30.000749902s
2024/01/30 01:03:21 Rows err: read tcp [::1]:50774->[::1]:1522: i/o timeout
statement failed in:  30.000684775s
2024/01/30 01:03:21 Rows err: read tcp [::1]:50776->[::1]:1522: i/o timeout
statement failed in:  30.000973098s
statement done in:  6.049930259s
2024/01/30 01:03:22 Rows err: read tcp [::1]:50778->[::1]:1522: i/o timeout
statement failed in:  30.000073911s
2024/01/30 01:03:22 Rows err: read tcp [::1]:50780->[::1]:1522: i/o timeout
statement failed in:  30.000716407s
2024/01/30 01:03:22 Rows err: read tcp [::1]:50782->[::1]:1522: i/o timeout
statement failed in:  30.000473918s
2024/01/30 01:03:22 Rows err: read tcp [::1]:50784->[::1]:1522: i/o timeout
statement failed in:  30.001272315s
2024/01/30 01:03:23 Rows err: read tcp [::1]:50786->[::1]:1522: i/o timeout
statement failed in:  30.000159989s
2024/01/30 01:03:23 Rows err: read tcp [::1]:50788->[::1]:1522: i/o timeout
statement failed in:  30.000173555s
2024/01/30 01:03:23 Rows err: read tcp [::1]:50790->[::1]:1522: i/o timeout
statement failed in:  30.001148626s
2024/01/30 01:03:23 Rows err: read tcp [::1]:50792->[::1]:1522: i/o timeout
statement failed in:  30.000124096s
2024/01/30 01:03:24 Rows err: read tcp [::1]:50794->[::1]:1522: i/o timeout
statement failed in:  30.000786192s
2024/01/30 01:03:24 Rows err: read tcp [::1]:50796->[::1]:1522: i/o timeout
statement failed in:  30.000189467s
2024/01/30 01:03:24 Rows err: read tcp [::1]:50798->[::1]:1522: i/o timeout
statement failed in:  30.001385449s
2024/01/30 01:03:24 Rows err: read tcp [::1]:50800->[::1]:1522: i/o timeout
statement failed in:  30.001136763s
2024/01/30 01:03:25 Rows err: read tcp [::1]:50802->[::1]:1522: i/o timeout
statement failed in:  30.000678195s
2024/01/30 01:03:25 Rows err: read tcp [::1]:50804->[::1]:1522: i/o timeout
statement failed in:  30.000912017s
2024/01/30 01:03:25 Rows err: read tcp [::1]:50806->[::1]:1522: i/o timeout
statement failed in:  30.000091732s
2024/01/30 01:03:25 Rows err: read tcp [::1]:50808->[::1]:1522: i/o timeout
statement failed in:  30.000837641s
2024/01/30 01:03:26 Rows err: read tcp [::1]:50810->[::1]:1522: i/o timeout
statement failed in:  30.00135689s
2024/01/30 01:03:26 Rows err: read tcp [::1]:50812->[::1]:1522: i/o timeout
statement failed in:  30.000271256s
2024/01/30 01:03:26 Rows err: read tcp [::1]:50814->[::1]:1522: i/o timeout
statement failed in:  30.000420435s
2024/01/30 01:03:27 Rows err: read tcp [::1]:50816->[::1]:1522: i/o timeout
statement failed in:  30.000823847s
2024/01/30 01:03:27 Rows err: read tcp [::1]:50818->[::1]:1522: i/o timeout
statement failed in:  30.00113488s
2024/01/30 01:03:27 Rows err: read tcp [::1]:50820->[::1]:1522: i/o timeout
statement failed in:  30.000185701s
2024/01/30 01:03:27 Rows err: read tcp [::1]:50822->[::1]:1522: i/o timeout
statement failed in:  30.000470777s
2024/01/30 01:03:28 Rows err: read tcp [::1]:50825->[::1]:1522: i/o timeout
statement failed in:  30.000352213s
2024/01/30 01:03:28 Rows err: read tcp [::1]:50827->[::1]:1522: i/o timeout
statement failed in:  30.002718824s
2024/01/30 01:03:28 Rows err: read tcp [::1]:50829->[::1]:1522: i/o timeout
statement failed in:  30.00137752s
2024/01/30 01:03:28 Rows err: read tcp [::1]:50831->[::1]:1522: i/o timeout
statement failed in:  30.000183472s
2024/01/30 01:03:29 Rows err: read tcp [::1]:50833->[::1]:1522: i/o timeout
statement failed in:  30.000279398s
2024/01/30 01:03:29 Rows err: read tcp [::1]:50835->[::1]:1522: i/o timeout
statement failed in:  30.000187725s
2024/01/30 01:03:29 Rows err: read tcp [::1]:50837->[::1]:1522: i/o timeout
statement failed in:  30.000639443s
2024/01/30 01:03:29 Rows err: read tcp [::1]:50839->[::1]:1522: i/o timeout
statement failed in:  30.000369611s
2024/01/30 01:03:30 Rows err: read tcp [::1]:50841->[::1]:1522: i/o timeout
statement failed in:  30.000722169s
2024/01/30 01:03:30 Rows err: read tcp [::1]:50843->[::1]:1522: i/o timeout
statement failed in:  30.000479362s
statement done in:  6.16170024s
2024/01/30 01:03:30 Rows err: read tcp [::1]:50845->[::1]:1522: i/o timeout
statement failed in:  30.000334448s
2024/01/30 01:03:30 Rows err: read tcp [::1]:50847->[::1]:1522: i/o timeout
statement failed in:  30.000648308s
2024/01/30 01:03:31 Rows err: read tcp [::1]:50849->[::1]:1522: i/o timeout
statement failed in:  30.000326245s
2024/01/30 01:03:31 Rows err: read tcp [::1]:50851->[::1]:1522: i/o timeout
statement failed in:  30.001204341s
2024/01/30 01:03:31 Rows err: read tcp [::1]:50853->[::1]:1522: i/o timeout
statement failed in:  30.000238577s
2024/01/30 01:03:31 Rows err: read tcp [::1]:50855->[::1]:1522: i/o timeout
statement failed in:  30.000279288s
2024/01/30 01:03:32 Rows err: read tcp [::1]:50857->[::1]:1522: i/o timeout
statement failed in:  30.000984336s
2024/01/30 01:03:32 Rows err: read tcp [::1]:50859->[::1]:1522: i/o timeout
statement failed in:  30.001296198s
2024/01/30 01:03:32 Rows err: read tcp [::1]:50861->[::1]:1522: i/o timeout
statement failed in:  30.000427518s
2024/01/30 01:03:33 Rows err: read tcp [::1]:50863->[::1]:1522: i/o timeout
statement failed in:  30.000235591s
2024/01/30 01:03:33 Rows err: read tcp [::1]:50865->[::1]:1522: i/o timeout
statement failed in:  30.001008789s
2024/01/30 01:03:33 Rows err: read tcp [::1]:50867->[::1]:1522: i/o timeout
statement failed in:  30.000836436s
2024/01/30 01:03:33 Rows err: read tcp [::1]:50869->[::1]:1522: i/o timeout
statement failed in:  30.000147923s
2024/01/30 01:03:34 Rows err: read tcp [::1]:50871->[::1]:1522: i/o timeout
statement failed in:  30.000264403s
2024/01/30 01:03:34 Rows err: read tcp [::1]:50873->[::1]:1522: i/o timeout
statement failed in:  30.001112778s
2024/01/30 01:03:34 Rows err: read tcp [::1]:50875->[::1]:1522: i/o timeout
statement failed in:  30.000840127s
2024/01/30 01:03:34 Rows err: read tcp [::1]:50877->[::1]:1522: i/o timeout
statement failed in:  30.001128846s
2024/01/30 01:03:35 Rows err: read tcp [::1]:50879->[::1]:1522: i/o timeout
statement failed in:  30.001024539s
2024/01/30 01:03:35 Rows err: read tcp [::1]:50881->[::1]:1522: i/o timeout
statement failed in:  30.000657715s
2024/01/30 01:03:35 Rows err: read tcp [::1]:50883->[::1]:1522: i/o timeout
statement failed in:  30.000946727s
2024/01/30 01:03:35 Rows err: read tcp [::1]:50885->[::1]:1522: i/o timeout
statement failed in:  30.000281008s
2024/01/30 01:03:36 Rows err: read tcp [::1]:50887->[::1]:1522: i/o timeout
statement failed in:  30.000561932s
2024/01/30 01:03:36 Rows err: read tcp [::1]:50889->[::1]:1522: i/o timeout
statement failed in:  30.000838489s
2024/01/30 01:03:36 Rows err: read tcp [::1]:50891->[::1]:1522: i/o timeout
statement failed in:  30.00107201s
2024/01/30 01:03:36 Rows err: read tcp [::1]:50893->[::1]:1522: i/o timeout
statement failed in:  30.000123878s
2024/01/30 01:03:37 Rows err: read tcp [::1]:50895->[::1]:1522: i/o timeout
statement failed in:  30.001259649s
2024/01/30 01:03:37 Rows err: read tcp [::1]:50897->[::1]:1522: i/o timeout
statement failed in:  30.001013656s
2024/01/30 01:03:37 Rows err: read tcp [::1]:50899->[::1]:1522: i/o timeout
statement failed in:  30.000243127s
2024/01/30 01:03:37 Rows err: read tcp [::1]:50901->[::1]:1522: i/o timeout
statement failed in:  30.019519263s
2024/01/30 01:03:38 Rows err: read tcp [::1]:50903->[::1]:1522: i/o timeout
statement failed in:  30.001140687s
2024/01/30 01:03:38 Rows err: read tcp [::1]:50905->[::1]:1522: i/o timeout
statement failed in:  30.00079973s
2024/01/30 01:03:38 Rows err: read tcp [::1]:50907->[::1]:1522: i/o timeout
statement failed in:  30.001374034s
2024/01/30 01:03:38 Rows err: read tcp [::1]:50909->[::1]:1522: i/o timeout
statement failed in:  30.000340393s
2024/01/30 01:03:39 Rows err: read tcp [::1]:50911->[::1]:1522: i/o timeout
statement failed in:  30.000469159s
2024/01/30 01:03:39 Rows err: read tcp [::1]:50913->[::1]:1522: i/o timeout
statement failed in:  30.000577417s
2024/01/30 01:03:39 Rows err: read tcp [::1]:50915->[::1]:1522: i/o timeout
statement failed in:  30.000821s
2024/01/30 01:03:40 Rows err: read tcp [::1]:50917->[::1]:1522: i/o timeout
statement failed in:  30.000631867s
2024/01/30 01:03:40 Rows err: read tcp [::1]:50919->[::1]:1522: i/o timeout
statement failed in:  30.00073339s
2024/01/30 01:03:40 Rows err: read tcp [::1]:50921->[::1]:1522: i/o timeout
statement failed in:  30.000918905s
2024/01/30 01:03:40 Rows err: read tcp [::1]:50923->[::1]:1522: i/o timeout
statement failed in:  30.000409634s
2024/01/30 01:03:41 Rows err: read tcp [::1]:50925->[::1]:1522: i/o timeout
statement failed in:  30.000118119s
statement done in:  6.16147129s
2024/01/30 01:03:41 Rows err: read tcp [::1]:50927->[::1]:1522: i/o timeout
statement failed in:  30.001304135s
2024/01/30 01:03:41 Rows err: read tcp [::1]:50929->[::1]:1522: i/o timeout
statement failed in:  30.000341995s
2024/01/30 01:03:41 Rows err: read tcp [::1]:50931->[::1]:1522: i/o timeout
statement failed in:  30.000501973s
2024/01/30 01:03:42 Rows err: read tcp [::1]:50933->[::1]:1522: i/o timeout
statement failed in:  30.000188033s
2024/01/30 01:03:42 Rows err: read tcp [::1]:50935->[::1]:1522: i/o timeout
statement failed in:  30.001535963s
2024/01/30 01:03:42 Rows err: read tcp [::1]:50937->[::1]:1522: i/o timeout
statement failed in:  30.000410544s
2024/01/30 01:03:42 Rows err: read tcp [::1]:50939->[::1]:1522: i/o timeout
statement failed in:  30.008316347s
2024/01/30 01:03:43 Rows err: read tcp [::1]:50941->[::1]:1522: i/o timeout
statement failed in:  30.000214526s
2024/01/30 01:03:43 Rows err: read tcp [::1]:50943->[::1]:1522: i/o timeout
statement failed in:  30.00075535s
2024/01/30 01:03:43 Rows err: read tcp [::1]:50945->[::1]:1522: i/o timeout
statement failed in:  30.000333552s
2024/01/30 01:03:44 Rows err: read tcp [::1]:50947->[::1]:1522: i/o timeout
statement failed in:  30.001129749s
2024/01/30 01:03:48 Rows err: read tcp [::1]:50949->[::1]:1522: i/o timeout
statement failed in:  30.001263615s
statement done in:  6.223474735s
statement done in:  6.263805552s
statement done in:  6.262821739s
^C
Process finished with the exit code 130 (interrupted by signal 2:SIGINT)

sijms commented 6 months ago

one stmt = one cursor what you do is sharing a cursor with go routines so when the cursor closed ==> query will give invalid cursor

sijms commented 6 months ago

I make some changes in the code to do more testing 1- make pool.Ping() at first so prepare stmt will take no time

sijms commented 6 months ago

change the interval of scheduler to 10 seconds

if _, err := scheduler.Every(10).Second().Do(execQuery, stmt); err != nil {
    log.Fatalf("Err job: %v", err)
}

so here the stmt cycle between go routines use stmt in 6 second then in next go-routine after 10 second use same stmt and so one

statement done in:  6.23777933s
statement done in:  6.173200325s
statement done in:  6.095747128s
statement done in:  6.132643942s
statement done in:  6.167505299s
statement done in:  6.152956421s
statement done in:  6.348165558s
statement done in:  6.289108592s
statement done in:  6.227015146s
statement done in:  6.085832225s
statement done in:  6.113325508s
statement done in:  6.256283425s
^C
Process finished with the exit code 130 (interrupted by signal 2:SIGINT)

result is good

dargi98 commented 6 months ago

So a statement shouldn't be used by two routines at the same time?

sijms commented 6 months ago

too early to decide I increase context time + timeout and use pool instead of statement I get the following error

2024/01/30 01:44:18 Rows err: EOF
statement failed in:  35.715124757s
2024/01/30 01:44:19 Rows err: EOF
statement failed in:  35.784884064s
2024/01/30 01:44:19 Rows err: EOF
statement failed in:  36.041784705s
2024/01/30 01:44:19 Rows err: EOF
statement failed in:  35.69886748s
2024/01/30 01:44:19 Rows err: EOF
statement failed in:  35.773319594s
2024/01/30 01:44:19 Rows err: EOF

the error after 35 seconds which is not enough to make new connection so I will repeat the test withoutgocron and see

dargi98 commented 6 months ago

But anyway, if the timeout is set to 30s, the query shouldn't take more than that in any case, and I was seeing timeouts at 40s. Something's off there.

dargi98 commented 6 months ago

Do you know why is the cursor being closed, if in my code I never close the statement?

dargi98 commented 6 months ago

So I've noticed that when queries are executed in parallel, they need one cursor for each thread, so cursors will be created if there are not enough. Creating a cursors is what it was taking +40s in my code. Reusing a cursor just took 5s. Everything looks good then. Thanks!

sijms commented 6 months ago

I find that server send data packet with data flag 0x40 which means final packet so I close the session and send driver.ErrBadConn which make some imporvment

statement done in:  1.139308595s
statement done in:  1.778063129s
statement done in:  1.766297968s
statement done in:  1.967601353s
statement done in:  2.259216404s
statement done in:  2.822390483s
statement done in:  2.654227856s
statement done in:  2.465289225s
statement done in:  3.160532626s
statement done in:  3.435321345s
statement done in:  2.947048625s
statement done in:  3.61750788s
statement done in:  3.548542799s
2024/01/31 02:44:49 db.Conn err: driver: bad connection
statement failed in:  1.193882661s
statement done in:  5.660202346s
statement done in:  5.102705821s
statement done in:  5.023838916s
statement done in:  4.526794712s
2024/01/31 02:44:50 db.Conn err: driver: bad connection
statement failed in:  1.60393316s
2024/01/31 02:44:50 db.Conn err: driver: bad connection
statement failed in:  1.419327006s

but after a while server stop responding and only send final packet

sijms commented 6 months ago

after change max number of open conn from 500 to 30

statement done in:  1.215149094s
statement done in:  1.412897495s
statement done in:  4.090677614s
statement done in:  1.108195s
statement done in:  3.370878617s
statement done in:  4.638568939s
statement done in:  4.932093239s
statement done in:  4.157165957s
statement done in:  4.875736283s
statement done in:  5.352759952s
statement done in:  5.737172428s
statement done in:  6.582372173s
statement done in:  8.158797964s
statement done in:  7.346206854s
statement done in:  6.239381876s
statement done in:  6.739436361s
statement done in:  6.246055732s
statement done in:  7.534909035s
statement done in:  6.110846654s
statement done in:  10.057045511s
statement done in:  7.572630859s
statement done in:  11.158363607s
statement done in:  10.670837925s
statement done in:  10.38588354s
statement done in:  14.720954339s
statement done in:  14.533476103s
statement done in:  13.315378693s
statement done in:  18.218179663s
statement done in:  15.963077109s
statement done in:  11.608412636s
statement done in:  14.753760267s
statement done in:  15.309727125s
statement done in:  12.329979088s
statement done in:  12.102954557s
statement done in:  13.410618834s
statement done in:  11.48361484s
statement done in:  16.768000984s
statement done in:  15.0221943s
statement done in:  15.464390403s
statement done in:  14.077422666s
statement done in:  21.837804845s
statement done in:  19.362774505s
statement done in:  16.883516648s
statement done in:  15.344820262s
statement done in:  18.388700669s
statement done in:  14.04999551s
statement done in:  18.838282823s
statement done in:  12.839230854s
statement done in:  18.092491141s
statement done in:  19.41892591s
statement done in:  10.405253134s
statement done in:  16.678399511s
statement done in:  13.24961344s
statement done in:  10.665653951s
statement done in:  19.468485012s
statement done in:  21.036524841s
statement done in:  12.711629722s
statement done in:  21.491073612s
statement done in:  11.741902989s
statement done in:  20.221521085s
statement done in:  21.492173214s
statement done in:  17.293201421s
statement done in:  16.343159635s
statement done in:  16.804669533s
statement done in:  14.400441646s
statement done in:  14.956597858s
statement done in:  27.535942926s
statement done in:  23.801911603s
statement done in:  26.322372603s
statement done in:  21.336852239s
statement done in:  15.342024494s
statement done in:  17.467764179s
statement done in:  16.198997026s
statement done in:  27.079797421s
statement done in:  24.396046998s
statement done in:  22.544461673s
statement done in:  24.637400234s
statement done in:  31.229253715s
statement done in:  16.505806792s
statement done in:  34.046918028s
statement done in:  21.061540335s
statement done in:  29.318266024s
statement done in:  25.888398806s
statement done in:  19.664457294s
statement done in:  17.934820192s
statement done in:  15.378547903s
statement done in:  10.861956742s
statement done in:  21.622224156s
statement done in:  16.24738432s
statement done in:  20.050764058s
statement done in:  18.883230269s
statement done in:  10.915862302s
statement done in:  11.681386786s
statement done in:  17.981194775s
statement done in:  16.695439115s
statement done in:  27.237099509s
statement done in:  31.868155952s
statement done in:  21.233326607s
statement done in:  16.528145419s
statement done in:  17.924605691s
statement done in:  36.251833838s
2024/01/31 02:58:10 Rows err: driver: bad connection
statement failed in:  39.468540828s
statement done in:  24.317574592s
statement done in:  29.976650866s
statement done in:  35.281565944s
statement done in:  34.799604684s
statement done in:  32.576943696s
statement done in:  29.179923442s
statement done in:  33.715296615s
statement done in:  12.255486536s
statement done in:  16.259255707s
statement done in:  30.021276174s
statement done in:  19.029107127s
statement done in:  32.138135729s
statement done in:  44.601921824s
statement done in:  18.445921731s
statement done in:  16.31780594s
statement done in:  36.591775577s
statement done in:  32.364189902s
statement done in:  21.894387792s
statement done in:  27.394239048s
statement done in:  39.468242367s
statement done in:  42.172286457s
statement done in:  18.730293776s
statement done in:  49.916135043s
statement done in:  17.184262383s
statement done in:  30.904592379s
statement done in:  15.692376844s
statement done in:  14.697511267s
statement done in:  48.806478097s
statement done in:  21.134551564s
statement done in:  38.387758484s
statement done in:  33.41249595s
statement done in:  19.514779444s
statement done in:  15.227657221s
statement done in:  28.757849358s
statement done in:  57.008517535s
statement done in:  17.258857138s
statement done in:  16.051285708s
statement done in:  45.987840178s
statement done in:  16.328455369s
statement done in:  15.284561017s
statement done in:  25.072067758s
statement done in:  46.591108725s
statement done in:  34.400984886s
statement done in:  28.903588289s
statement done in:  59.211197321s
statement done in:  19.784193274s
statement done in:  1m3.135891451s
statement done in:  23.38999243s
statement done in:  33.570112135s
statement done in:  47.651607834s
statement done in:  52.428409371s
statement done in:  26.725323264s
statement done in:  12.231614636s
statement done in:  20.842014754s
statement done in:  25.843890164s
statement done in:  58.932644847s
statement done in:  39.51398696s
^C
Process finished with the exit code 130 (interrupted by signal 2:SIGINT)

after decrease it to 20

pool.SetMaxOpenConns(20)

no error come

note that the last tests is done without toxiproxy because I find no difference between using or not using it

so from the above it is clear that the issue is not a client issue but related to server response (server will respond to limited number of open connection)

sijms / go-ora

Concurrency issues #506