Shutdown can cause engine crash if there is client which establishes dozen of different connections (via ES/EDS) and they all do DML [CORE5098]

[firebird-automations]

Submitted by: @pavel-zotov

Attachments: gdb-crashes-on-shutdown-while-active-DML-attachments-exist.zip

0. Open two console sessions, one for launching ISQL, second for move database to shutdown state.

1. Run in 1st console following script for initial DDL:

shell rm -f /var/db/fb30/e30.fdb 2>/dev/null; create database 'localhost/3333:/var/db/fb30/e30.fdb' user 'SYSDBA' password 'masterkey'; commit; recreate table test(s varchar(36) unique); commit;

show version; show database;

2. Run (also in 1st console) following script for starting ISQL which, in turn, will create new attachments with DML activity. 2.1 Prepare file (its name = "c3328-dml.sql"):

set term ^; execute block as declare v_role varchar(31); begin while (1=1) do begin v_role = left(replace( uuid_to_char(gen_uuid()), '-', ''), 31); execute statement ( 'insert into test(s) values( ? )' ) ( uuid_to_char( gen_uuid() ) ) on external 'localhost:' || rdb$get_context('SYSTEM','DB_NAME') as user 'SYSDBA' password 'masterkey' role v_role ; end end ^ set term ;^

2.2 Go:

$ /opt/fb30ss/bin/isql /3333:/var/db/fb30/e30.fdb -i /var/db/fb30/c3328-dml.sql

-- and wait several sseconds (I could not understand whether number of delay affect or no; crash can be after 2-3", but may absent even after 10-15"; it seems to me that crash has probability < 50%).

3. Change to 2nd console and run following shell script:

fbc=/opt/fb30ss/bin dbn=/var/db/fb30/e30.fdb

$fbc/fbsvcmgr localhost:service_mgr user SYSDBA password masterkey action_get_fb_log >./fblog_1.txt

$fbc/fbsvcmgr localhost:service_mgr user SYSDBA password masterkey \ action_properties prp_shutdown_mode prp_sm_full prp_shutdown_db 0 \ dbname $dbn

$fbc/fbsvcmgr localhost:service_mgr user SYSDBA password masterkey \ action_db_stats \ dbname $dbn \ sts_hdr_pages \ | grep -i attributes

$fbc/fbsvcmgr localhost:service_mgr user SYSDBA password masterkey \ action_properties prp_shutdown_mode prp_sm_full prp_shutdown_db 0 \ dbname $dbn

$fbc/fbsvcmgr localhost:service_mgr user SYSDBA password masterkey \ action_properties prp_db_online \ dbname $dbn

$fbc/fbsvcmgr localhost:service_mgr user SYSDBA password masterkey \ action_db_stats \ dbname $dbn \ sts_hdr_pages \ | grep -i attributes

$fbc/fbsvcmgr localhost:service_mgr user SYSDBA password masterkey action_get_fb_log >./fblog_2.txt

ls -l ./fblog_?.txt

REPEAT steps 2..3 several times! It's quite possible that FB will _NOT_ crashed after 5-10 attempts. Eventually one may to get on 2nd console (i.e. where we force SHUTDOWN):

$ ./c3328.sh Error reading data from the connection. ------------------------------------------------------------------- [ 1 ] Unable to complete network request to host "localhost". -Failed to establish a connection. Attributes force write -rw-r--r-- 1 firebird firebird 31086 Feb 2 23:13 ./fblog_1.txt -rw-r--r-- 1 firebird firebird 31420 Feb 2 23:13 ./fblog_2.txt

-- instead of normal output:

$ ./c3328.sh Attributes force write, full shutdown Target shutdown mode is invalid for database "/var/db/fb30/e30.fdb" Attributes force write -rw-r--r-- 1 firebird firebird 62316 Feb 2 23:48 ./fblog_1.txt -rw-r--r-- 1 firebird firebird 62316 Feb 2 23:48 ./fblog_2.txt

This mean that either "action_properties prp_shutdown_mode prp_sm_full prp_shutdown_db 0" or "action_db_stats sts_hdr_pages" caused engine to crash.

Three stacktraces of crashed FB process are in attached .zip.

PS. Console session #⁠1 (which executes "client" script) may contain message "invalid transaction handle (expecting explicit transaction start)", but it appears not regularly and is not relates to crash - may be this will be useful for investigating.

$ /opt/fb30ss/bin/isql /3333:/var/db/fb30/e30.fdb -i /var/db/fb30/c3328-dml.sql Statement failed, SQLSTATE = 42000 Execute statement error at attach : 335544528 : database /var/db/fb30/e30.fdb shutdown Data source : Firebird::localhost:/var/db/fb30/e30.fdb After line 1 in file /var/db/fb30/c3328-dml.sql Statement failed, SQLSTATE = 08003 invalid transaction handle (expecting explicit transaction start) After line 17 in file /var/db/fb30/c3328-dml.sql Statement failed, SQLSTATE = HY000 database /var/db/fb30/e30.fdb shutdown After line 17 in file /var/db/fb30/c3328-dml.sql

PPS. LI-V3.0.0.32311, build on Revision: 62907

[firebird-automations]

Modified by: @pavel-zotov

Attachment: gdb\-crashes\-on\-shutdown\-while\-active\-DML\-attachments\-exist\.zip \[ 12894 \]

[firebird-automations]

Modified by: @hvlad

assignee: Vlad Khorsun \[ hvlad \]

[firebird-automations]

Commented by: Sean Leyne (seanleyne)

1- Does the connection method really make a difference for the case?

2- What does ES/EDS refer to? It is not a term commonly used within FB support/Devel lists.

[firebird-automations]

Commented by: @hvlad

I can't reproduce it on Windows

[firebird-automations]

Commented by: @hvlad

Sean,

ES is Execute Statement, EDS is External Data Source