search

gluent / goe

GOE: a simple and flexible way to copy data from an Oracle Database to Google BigQuery.
Apache License 2.0
8 stars 2 forks source link

Repo connection leak when running integration tests #48

Closed nj1973 closed 8 months ago

nj1973 commented 9 months ago

When we run integration tests we are leaking OrchestrationRepo client connections and eventually tests fail because we exceed the Oracle processes parameter.

It's not a problem at the moment because most integration tests have not been converted to the new repo.

However this is a pre-req to issue https://github.com/gluent/goe/issues/42

nj1973 commented 8 months ago

SQL to monitor sessions:

set lines 200 pages 100
column USERNAME format a10
column module format a25
column action format a45
select username,module,action,last_call_et
from v$session where username like 'G%'
order by last_call_et;

I ran a few tests concurrently:

pytest -n 3 tests/integration/scenarios/test_offload_*.py

And after a few minutes saw a build up of sessions in Oracle:

USERNAME   MODULE                    ACTION                                        LAST_CALL_ET
---------- ------------------------- --------------------------------------------- ------------
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                            1
GOE_ADM    Gluent Offload Engine     OffloadSourceTable(GOE_TEST.STORY_PBO_LIST)              1
GOE_ADM    Gluent Offload Engine     OffloadSourceTable(GOE_TEST.RPA_DATE)                    1
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                            1
GOE_ADM    Gluent Offload Engine     OffloadSourceTable(GOE_TEST.RPA_NUM)                     1
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                            1
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                            1
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                            8
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                            9
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                            9
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                            9
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           10
GOE_ADM    python3@neiljohnson.c.goo                                                         16
           glers.com (TNS V1-V3)

GOE_ADM    Gluent Offload Engine     test_offload_pbo_unicode                                18
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           19
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           26
GOE_ADM    python3@neiljohnson.c.goo                                                         31
           glers.com (TNS V1-V3)

GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           34
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           36
GOE_ADM    python3@neiljohnson.c.goo                                                         42
           glers.com (TNS V1-V3)

GOE_ADM    Gluent Offload Engine     test_offload_pbo_ts                                     44
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           44
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           48
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           48
GOE_ADM    Gluent Offload Engine     test_offload_pbo_dim                                    62
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           62
GOE_ADM    Gluent Offload Engine     test_offload_pbo_range                                  66
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           66
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           71
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           79
GOE_ADM    Gluent Offload Engine     test_offload_pbo_char_pad                               79
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           80
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                           88
GOE_ADM    python3@neiljohnson.c.goo                                                         99
           glers.com (TNS V1-V3)

GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          102
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          106
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          106
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          112
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          118
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          119
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          124
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          125
GOE_ADM    python3@neiljohnson.c.goo                                                        137
           glers.com (TNS V1-V3)

GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          139
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          139
GOE_ADM    python3@neiljohnson.c.goo                                                        141
           glers.com (TNS V1-V3)

GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          143
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          147
GOE_ADM    Gluent Offload Engine     test_offload_pbo_late_range_100_0                      147
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          147
GOE_ADM    Gluent Offload Engine     test_offload_pbo_late_list_as_range_100_0              151
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          151
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          165
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          166
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          169
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          177
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          178
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          181
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          197
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          207
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          207
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          208
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          216
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          216
GOE_ADM    Gluent Offload Engine     test_offload_pbo_late_range_90_10                      216
GOE_ADM    Gluent Offload Engine     test_offload_pbo_late_list_as_range                    217
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          217
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          217
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          218
GOE_ADM    python3@neiljohnson.c.goo                                                        234
           glers.com (TNS V1-V3)

GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          237
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          237
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          241
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          242
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          248
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          248
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          249
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          249
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          256
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          260
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          261
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          264
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          270

USERNAME   MODULE                    ACTION                                        LAST_CALL_ET
---------- ------------------------- --------------------------------------------- ------------
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          271
GOE_ADM    Gluent Offload Engine     OracleOrchestrationRepoClient                          280