Decoding BLOB / CLOB resources

[phansys]

I'd like to know what is the status of https://bugs.php.net/bug.php?id=59158. Is the issue still present? Is there something I can do in order to collaborate with the fix? doctrine/dbal dropped the support for this extension because that problem, and I'd like to restore that support.

Thank you!

[NattyNarwhal]

I'm attempting to reproduce this, but it works, at least with a simple table (one made by the ibm_db2 test suite) on IBM i and LUW.

<?
$conn = new PDO("ibm:[...]", "", "");
$stmt = $conn->prepare("select * from blob_stream");
$res = $stmt->execute();
while (($row = $stmt->fetch()) != false) {
        $stream = $row["MY_BLOB"];
        $contents = stream_get_contents($stream);
        var_dump($contents);
}

$ php -d extension=$PWD/modules/pdo_ibm.so ../testpdoibmlob.php 
string(41) "THIS IS A BLOB TEST. THIS IS A BLOB TEST."

[NattyNarwhal]

(I should mention I've tested this w/ PHP 8.2 on i and PHP 8.1 on Linux+LUW)

[NattyNarwhal]

FWIW I re-read the issue and I think I saw an implication it's about the lifetimes of the LOB stream. Unfortunately, without a sample, it's hard to reproduce and determine what/why. Especially since it's PHP 5.x era, and a lot of things have changed since then.

[phansys]

Thanks for your response and sorry for the delay. I got a segfault on the doctrine/dbal test suite trying to use this extension: https://github.com/doctrine/dbal/actions/runs/4179111783/jobs/7238686229#step:9:727. This is the test producing the error. Please, let me know if it is enough or if I should try to create a reproducer here.

[NattyNarwhal]

A case that reproduces would be useful. I'm not familiar enough with Doctrine to tell what SQL it generates.

[NattyNarwhal]

I think this would be the right translation from Doctrine to raw SQL/PDO:

<?php

$conn = new PDO("ibm:DRIVER={IBM DB2 ODBC DRIVER};DATABASE=sample;HOSTNAME=localhost;PORT=60000;PROTOCOL=TCPIP", "db2inst1", "password", [PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION]);

$stmt = $conn->prepare("drop table blob_table");
$res = $stmt->execute();

$stmt = $conn->prepare("create table blob_table (id integer, clobcolumn clob, blobcolumn blob)");
$res = $stmt->execute();

$longBlob = str_repeat('x', 4 * 8192);

$stmt = $conn->prepare("insert into blob_table values (?, ?, ?)");
$stmt->bindValue(1, 1, PDO::PARAM_INT);
$stmt->bindValue(2, "ignored", PDO::PARAM_STR);
$stmt->bindValue(3, fopen('data://text/plain,' . $longBlob, 'r'), PDO::PARAM_LOB);
$res = $stmt->execute();

I don't get a crash or userland exception from this. (edit: LUW, Linux amd64, PHP 8.1)

[phansys]

Yes, that's exactly what I'm trying to reproduce, without errors: https://github.com/phansys/pecl-database-pdo_ibm/actions/runs/4186046386/jobs/7253998644. I'll try to investigate if there are some different settings affecting the results.

[NattyNarwhal]

What would be the best way to run that test from dbal? I have the branch checked out, but the setup for CI looks very complex and I haven't figured out how to untangle it and just run the DB2 relevant tests against PDO_IBM only (...and successfully).

[NattyNarwhal]

(That is, I'm running ./vendor/bin/phpunit -d extension=/home/calvin/src/pecl-database-pdo_ibm/modules/pdo_ibm.so -c ci/github/phpunit/pdo_ibm.xml --debug -vvv and I modified just db_password - do I need to change anything else or run anything to initialize it?)

[phansys]

I'd say your approach is enough to achieve a very similar setup. Only two things to consider on the doctrine/dbal CI:

• zend.assertions=1;
• PCOV is enabled.

https://github.com/doctrine/dbal/pull/5924/files#diff-c471496bcb3ed47397b73d2e3e3aa41a8a679c86accaed9e05676fd9d5987434R567-R570.

[NattyNarwhal]

Hmmm, running ./vendor/bin/phpunit -c ci/github/phpunit/pdo_ibm.xml --debug -vvv with XML such as:

<?xml version="1.0" encoding="utf-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="../../../vendor/phpunit/phpunit/phpunit.xsd"
         colors="true"
         beStrictAboutOutputDuringTests="true"
         beStrictAboutTodoAnnotatedTests="true"
         failOnRisky="true"
         failOnWarning="true"
         convertDeprecationsToExceptions="true"
>
    <php>
        <ini name="error_reporting" value="-1" />
        <ini name="extension" value="/home/calvin/src/pecl-database-pdo_ibm/modules/pdo_ibm.so" />
        <ini name="zend.assertions" value="1" />

        <var name="db_driver" value="pdo_ibm"/>
        <var name="db_host" value="127.0.0.1"/>
        <var name="db_user" value="db2inst1"/>
        <!--
        <var name="db_password" value="Doctrine2018"/>
        <var name="db_dbname" value="doctrine"/>
        -->
        <var name="db_password" value="password"/>
        <var name="db_dbname" value="sample"/>
    </php>

    <testsuites>
        <testsuite name="Doctrine DBAL Test Suite">
                <!--<directory>../../../tests</directory>-->
                <directory>../../../tests/Functional</directory>
        </testsuite>
    </testsuites>

    <coverage>
        <include>
            <directory suffix=".php">../../../src</directory>
        </include>
    </coverage>
</phpunit>

...I'm getting PDOException from everything basically. Clearly I'm not getting it to load correctly in the context of PHPUnit...

[phansys]

What if you filter only the BlobTest?

vendor/bin/phpunit -c ci/github/phpunit/pdo_ibm.xml --debug -vvv --filter BlobTest

[NattyNarwhal]

That cuts the number of tests down to a manageable number, though I still need to figure out why it's not finding PDO_IBM.

[phansys]

What is the message you're receiving? Is it from Doctrine or from PDO?

[NattyNarwhal]

Mix of both; PDO and Doctrine: PDOException: could not find driver or Doctrine\DBAL\Driver\PDO\Exception: could not find driver. I'm not entirely sure if it's loading the extension when running the tests...

[phansys]

For what you share, I guess is not being loaded. You could use var_dump(extension_loaded('pdo_ibm')); in order to confirm.

[NattyNarwhal]

Oh, you have to run PHPUnit with PHP having those options set, since its interface for that can't change PHP_INI_SYSTEM options. Of course.

Now I'm getting SQLSTATE=08001, SQLDriverConnect: -30081 [IBM][CLI Driver] SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "127.0.0.1". Communication function detecting the error: "connect". Protocol specific error code(s): "111", "*", "*". SQLSTATE=08001, so now I'm checking where I messed up the connection string...

[NattyNarwhal]

Yeah, default port wasn't proper. Now it's Doctrine\DBAL\Exception: Unknown database type decfloat requested, Doctrine\DBAL\Platforms\DB2Platform may not support it. using the existing database I have. Suspecting I need an empty one.

[phansys]

In case you need it, you may specify the db_protocol var at ci/github/phpunit/pdo_ibm.xml (ref).

[NattyNarwhal]

After remembering how to create a database in LUW (can you tell I'm an i user?), I can reproduce this with the slightly unwieldy command line php -d extension=/home/calvin/src/pecl-database-pdo_ibm/modules/pdo_ibm.so -d zend.assertions=1 vendor/bin/phpunit -d extension=/home/calvin/src/pecl-database-pdo_ibm/modules/pdo_ibm.so -d zend.assertions=1 -c ci/github/phpunit/pdo_ibm.xml --debug -vvv --filter BlobTest. The fact I only need to run the BlobTest helps too.

[NattyNarwhal]

I ran the test once and now I can't run any of them again :raised_eyebrow:

3) Doctrine\DBAL\Tests\Functional\BlobTest::testInsertProcessesStream
TypeError: preg_match(): Argument #2 ($subject) must be of type string, resource given

/home/calvin/src/dbal/src/Schema/DB2SchemaManager.php:106
/home/calvin/src/dbal/src/Schema/AbstractSchemaManager.php:1408
/home/calvin/src/dbal/src/Schema/AbstractSchemaManager.php:423
/home/calvin/src/dbal/src/Schema/DB2SchemaManager.php:43
/home/calvin/src/dbal/src/Schema/AbstractSchemaManager.php:1651
/home/calvin/src/dbal/src/Schema/AbstractSchemaManager.php:1663
/home/calvin/src/dbal/tests/TestUtil.php:107
/home/calvin/src/dbal/tests/TestUtil.php:61
/home/calvin/src/dbal/tests/FunctionalTestCase.php:40

[phansys]

Possibly that's because the value there is a stream resource created by fopen(). Let me check the flow in order to verify if I'm missing something.

[NattyNarwhal]

No, it affects the other tests too. The stack trace is hinting to me it might be something about fetching table metadata?

[phansys]

If I followed the flow correctly, I think the value passed there as argument 2 to preg_match() comes as result from \PDOStatement::fetchAll(\PDO::FETCH_ASSOC). Maybe you could use get_resource_type() in order to debug what are you receiving there.

[NattyNarwhal]

It's some kind of stream resource, but if I fetch it, it becomes NULL and returns nothing :raised_eyebrow:

If I check what $tableColumns is in AbstractSchemaManager::doListTables (the function in AbstractSchemaManager.php:423), I get:

array(1) {
  ["BLOB_TABLE"]=>
  array(3) {
    [0]=>
    array(10) {
      ["NAME"]=>
      string(10) "BLOB_TABLE"
      ["COLNAME"]=>
      string(2) "ID"
      ["TYPENAME"]=>
      string(7) "INTEGER"
      ["CODEPAGE"]=>
      string(1) "0"
      ["NULLS"]=>
      string(1) "N"
      ["LENGTH"]=>
      string(1) "4"
      ["SCALE"]=>
      string(1) "0"
      ["COMMENT"]=>
      NULL
      ["AUTOINCREMENT"]=>
      string(1) "0"
      ["DEFAULT"]=>
      NULL
    }
    [1]=>
    array(10) {
      ["NAME"]=>
      string(10) "BLOB_TABLE"
      ["COLNAME"]=>
      string(10) "CLOBCOLUMN"
      ["TYPENAME"]=>
      string(4) "CLOB"
      ["CODEPAGE"]=>
      string(4) "1208"
      ["NULLS"]=>
      string(1) "Y"
      ["LENGTH"]=>
      string(7) "1048576"
      ["SCALE"]=>
      string(1) "0"
      ["COMMENT"]=>
      NULL
      ["AUTOINCREMENT"]=>
      string(1) "0"
      ["DEFAULT"]=>
      resource(438) of type (stream)
    }
    [2]=>
    array(10) {
      ["NAME"]=>
      string(10) "BLOB_TABLE"
      ["COLNAME"]=>
      string(10) "BLOBCOLUMN"
      ["TYPENAME"]=>
      string(4) "BLOB"
      ["CODEPAGE"]=>
      string(1) "0"
      ["NULLS"]=>
      string(1) "Y"
      ["LENGTH"]=>
      string(7) "1048576"
      ["SCALE"]=>
      string(1) "0"
      ["COMMENT"]=>
      NULL
      ["AUTOINCREMENT"]=>
      string(1) "0"
      ["DEFAULT"]=>
      resource(439) of type (stream)
    }
  }
}

Per SYSCAT.COLUMNS, that (DEFAULT) looks like it's funnily enough, a clob having trouble.

[NattyNarwhal]

oh that's really funny

diff --git a/src/Schema/DB2SchemaManager.php b/src/Schema/DB2SchemaManager.php
index 0308d69b5..7f582edde 100644
--- a/src/Schema/DB2SchemaManager.php
+++ b/src/Schema/DB2SchemaManager.php
@@ -100,6 +100,9 @@ class DB2SchemaManager extends AbstractSchemaManager

         $default = null;

+        if (is_resource($tableColumn['default'])) {
+            $tableColumn['default'] = stream_get_contents($tableColumn['default']);
+        }
         if ($tableColumn['default'] !== null && $tableColumn['default'] !== 'NULL') {
             $default = $tableColumn['default'];

$ php -d extension=/home/calvin/src/pecl-database-pdo_ibm/modules/pdo_ibm.so -d zend.assertions=1 vendor/bin/phpunit -c ci/github/phpunit/pdo_ibm.xml --debug -vvv --filter BlobTest
PHPUnit 9.6.3 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.1.14
Configuration: ci/github/phpunit/pdo_ibm.xml

Test 'Doctrine\DBAL\Tests\Functional\BlobTest::testInsert' started
Segmentation fault (core dumped)

Test 'Doctrine\DBAL\Tests\Functional\BlobTest::testInsert' started

Program received signal SIGSEGV, Segmentation fault.
lob_stream_read (stream=<optimized out>, buf=0x7fffe1303000 '\377' <repeats 40 times>, "\003", count=8192) at /home/calvin/src/pecl-database-pdo_ibm/ibm_statement.c:127
127     switch (col_res->data_type) {
(gdb) where
#0  lob_stream_read (stream=<optimized out>, buf=0x7fffe1303000 '\377' <repeats 40 times>, "\003", count=8192) at /home/calvin/src/pecl-database-pdo_ibm/ibm_statement.c:127
#1  0x00005555557a234f in _php_stream_fill_read_buffer (stream=stream@entry=0x7ffff7356000, size=size@entry=8192) at /usr/src/debug/php-8.1.14-1.fc37.x86_64/main/streams/streams.c:666
#2  0x00005555557a2465 in _php_stream_read (stream=stream@entry=0x7ffff7356000, buf=<optimized out>, buf@entry=0x7fffe1364018 "", size=<optimized out>, size@entry=8192)
    at /usr/src/debug/php-8.1.14-1.fc37.x86_64/main/streams/streams.c:718
#3  0x00005555557a3aa1 in _php_stream_copy_to_mem (src=src@entry=0x7ffff7356000, maxlen=0, persistent=persistent@entry=0) at /usr/src/debug/php-8.1.14-1.fc37.x86_64/main/streams/streams.c:1518
#4  0x0000555555786687 in zif_stream_get_contents (execute_data=0x7ffff7214620, return_value=0x7ffff72145f0) at /usr/src/debug/php-8.1.14-1.fc37.x86_64/ext/standard/streamsfuncs.c:473
#5  0x00005555558618e9 in ZEND_DO_FCALL_BY_NAME_SPEC_RETVAL_USED_HANDLER () at /usr/src/debug/php-8.1.14-1.fc37.x86_64/Zend/zend_vm_execute.h:1558
#6  execute_ex (ex=0x7fffe17d7550) at /usr/src/debug/php-8.1.14-1.fc37.x86_64/Zend/zend_vm_execute.h:55795
#7  0x0000555555863dc9 in zend_execute (op_array=0x7ffff7277100, return_value=0x0) at /usr/src/debug/php-8.1.14-1.fc37.x86_64/Zend/zend_vm_execute.h:60147
#8  0x00005555557f37d0 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at /usr/src/debug/php-8.1.14-1.fc37.x86_64/Zend/zend.c:1799
#9  0x000055555578e5aa in php_execute_script (primary_file=primary_file@entry=0x7fffffffc610) at /usr/src/debug/php-8.1.14-1.fc37.x86_64/main/main.c:2541
#10 0x00005555558dd3fd in do_cli (argc=12, argv=0x555555e10b70) at /usr/src/debug/php-8.1.14-1.fc37.x86_64/sapi/cli/php_cli.c:965
#11 0x000055555563db83 in main (argc=12, argv=0x555555e10b70) at /usr/src/debug/php-8.1.14-1.fc37.x86_64/sapi/cli/php_cli.c:1367

[NattyNarwhal]

rebuilding from PHP master with --enable-debug,

(gdb) where
#0  0x00007ffff7fb195f in lob_stream_read (stream=0x7fffe5b7bdc0, buf=0x7fffe4047000 "\006\001", count=8192) at /home/calvin/src/pecl-database-pdo_ibm/ibm_statement.c:127
#1  0x0000000000a06918 in _php_stream_fill_read_buffer (stream=0x7fffe5b7bdc0, size=8192) at /home/calvin/src/php-src/main/streams/streams.c:669
#2  0x0000000000a06ad8 in _php_stream_read (stream=0x7fffe5b7bdc0, buf=0x7fffe402e018 "\200\355\277\345\377\177", size=8192) at /home/calvin/src/php-src/main/streams/streams.c:722
#3  0x0000000000a08646 in _php_stream_copy_to_mem (src=0x7fffe5b7bdc0, maxlen=0, persistent=0, __php_stream_call_depth=0, __zend_filename=0x15d4a38 "/home/calvin/src/php-src/ext/standard/streamsfuncs.c", __zend_lineno=473, 
    __zend_orig_filename=0x0, __zend_orig_lineno=0) at /home/calvin/src/php-src/main/streams/streams.c:1522
#4  0x00000000009c5531 in zif_stream_get_contents (execute_data=0x7ffff681ec60, return_value=0x7ffff681e860) at /home/calvin/src/php-src/ext/standard/streamsfuncs.c:473
#5  0x0000000000ac65ea in ZEND_DO_FCALL_BY_NAME_SPEC_RETVAL_USED_HANDLER () at /home/calvin/src/php-src/Zend/zend_vm_execute.h:1637
#6  0x0000000000b38f60 in execute_ex (ex=0x7ffff6817020) at /home/calvin/src/php-src/Zend/zend_vm_execute.h:56583
#7  0x0000000000b3d716 in zend_execute (op_array=0x7ffff687b500, return_value=0x0) at /home/calvin/src/php-src/Zend/zend_vm_execute.h:60935
#8  0x0000000000a88618 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /home/calvin/src/php-src/Zend/zend.c:1798
#9  0x00000000009eab3d in php_execute_script (primary_file=0x7fffffffd680) at /home/calvin/src/php-src/main/main.c:2482
#10 0x0000000000bf0cc8 in do_cli (argc=12, argv=0x1a449f0) at /home/calvin/src/php-src/sapi/cli/php_cli.c:964
#11 0x0000000000bf17bd in main (argc=12, argv=0x1a449f0) at /home/calvin/src/php-src/sapi/cli/php_cli.c:1333

[NattyNarwhal]

(gdb) p *data
$1 = {stmt_res = 0x7fffe5a46cd0, stmt = 0x7fffe5b8c540, colno = 9}
(gdb) p *data->stmt_res
$2 = {hstmt = 1, executing = 22, converted_statement = 0xf26d926e2920b438, lob_buffer = 0xa <error: Cannot access memory at address 0xa>, columns = 0x4457454956535953, cursor_type = (PDO_CURSOR_SCROLL | unknown: 0x5044), 
  server_ver = 110}

Looks like stuff in stmt_res is getting clobbered?

[kadler]

I wonder if valgrind would help find the clobbering

[NattyNarwhal]

Valgrind reports nothing.

I have a thick slathering of printf something like this so far:

 ** stmt_allocate_column_descriptors:   stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0
  *                                     no cols 1
  *                                     columns 0x7f78d9768120
 ** stmt_cleanup:   stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0
  * column desc?...
  * freeing col# 0 string 0x7f78d96dd540...
  * freeing columns 0x7f78d9768120
 ** stmt_allocate_column_descriptors:   stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0
  *                                     no cols 10
  *                                     columns 0x7f78d9754000
 ** create_lob_stream:  stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0 colno 9 col_res 0x7f78d9754240 data 0x7f78d9762118 php stream 0x7f78d977bdc0
 ** lob_stream_read:    php_stream 0x7f78d977bdc0 buf (nil) size 0
 **                     data 0x7f78d9762118 col_res 0x7f78d9754240 stmt_res 0x7f78d9646cd0 stmt 0x7f78d978c540
 *                      -> ctype -2
 *                      -> rc 0 read bytes -1
 *                      ret: NULL
 ** create_lob_stream:  stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0 colno 9 col_res 0x7f78d9754240 data 0x7f78d9762118 php stream 0x7f78d977bdc0
 ** lob_stream_read:    php_stream 0x7f78d977bdc0 buf (nil) size 0
 **                     data 0x7f78d9762118 col_res 0x7f78d9754240 stmt_res 0x7f78d9646cd0 stmt 0x7f78d978c540
 *                      -> ctype -2
 *                      -> rc 1 read bytes 4
 *                      ret: read bytes 0
 ** create_lob_stream:  stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0 colno 9 col_res 0x7f78d9754240 data 0x7f78d9762150 php stream 0x7f78d977c040
 ** lob_stream_read:    php_stream 0x7f78d977c040 buf (nil) size 0
 **                     data 0x7f78d9762150 col_res 0x7f78d9754240 stmt_res 0x7f78d9646cd0 stmt 0x7f78d978c540
 *                      -> ctype -2
 *                      -> rc 1 read bytes 4
 *                      ret: read bytes 0
 ** stmt_cleanup:   stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0
  * column desc?...
  * freeing col# 0 string 0x7f78d96dd180...
  * freeing col# 1 string 0x7f78d96ccf00...
  * freeing col# 2 string 0x7f78d96ccc00...
  * freeing col# 3 string 0x7f78d99f1208...
  * freeing col# 4 string 0x7f78d97531b8...
  * freeing col# 5 string 0x7f78d9757870...
  * freeing col# 6 string 0x7f78d97531e0...
  * freeing col# 7 string 0x7f78d9882c40...
  * freeing col# 8 string 0x7f78d9757a20...
  * freeing columns 0x7f78d9754000
 ** stmt_allocate_column_descriptors:   stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0
  *                                     no cols 5
  *                                     columns 0x7f78d9765f00
 ** stmt_cleanup:   stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0
  * column desc?...
  * freeing col# 0 string 0x7f78d96ccf00...
  * freeing col# 1 string 0x7f78d96dd180...
  * freeing col# 2 string 0x7f78d96dd540...
  * freeing col# 3 string 0x7f78d9757a20...
  * freeing col# 4 string 0x7f78d9757870...
  * freeing columns 0x7f78d9765f00
 ** stmt_allocate_column_descriptors:   stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0
  *                                     no cols 7
  *                                     columns 0x7f78d9747800
 ** stmt_cleanup:   stmt 0x7f78d978c540 stmt_res 0x7f78d9646cd0
  * column desc?...
  * freeing col# 0 string 0x7f78d96dd180...
  * freeing col# 1 string 0x7f78d96ccf00...
  * freeing col# 2 string 0x7f78d96ccc00...
  * freeing col# 3 string 0x7f78d96cca80...
  * freeing col# 4 string 0x7f78d96cc900...
  * freeing col# 5 string 0x7f78d9757870...
  * freeing col# 6 string 0x7f78d9757a20...
  * freeing columns 0x7f78d9747800
 ** stmt_allocate_column_descriptors:   stmt 0x7f78d978c540 stmt_res 0x7f78da1a7a50
  *                                     no cols 2
  *                                     columns 0x7f78d963aa00
 ** stmt_cleanup:   stmt 0x7f78d978c540 stmt_res 0x7f78da1a7a50
  * column desc?...
  * freeing col# 0 string 0x7f78d96cc900...
  * freeing col# 1 string 0x7f78d977bc80...
  * freeing columns 0x7f78d963aa00
 ** lob_stream_read:    php_stream 0x7f78d977bdc0 buf 0x7f78d7c47000 size 8192
 **                     data 0x7f78d9762118 col_res 0x4457454956535b93 stmt_res 0x7f78d9646cd0 stmt 0x7f78d978c540
Segmentation fault (core dumped)

...kinda hinting at lifetimes. Curiously, the LOB stream create calls LOB stream read with a null buffer - and I noticed the LOB handling code is almost completely different on i: https://github.com/php/pecl-database-pdo_ibm/blob/master/ibm_statement.c#L143-L168

...and that does handle a null buffer. The question is, why is it different for LUW vs. i?

[phansys]

Regarding the differences between LUW and i, IIUC, IBM seems to suggest different drivers for these platforms:

• LUW: https://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli/linuxx64_odbc_cli.tar.gz
• i: https://ibmi-oss-docs.readthedocs.io/en/latest/odbc/installation.html#linux

But since I'm not familiar with the IBM products, I can not be sure if these are just 2 possible variants.

[NattyNarwhal]

i has a (quirky and different) SQL/CLI implementation that's mostly compatible, which is why the #ifdef is around, but this case seems really curious to me, considering the massive behaviour difference (especially when the LUW case also seemingly triggers those behaviours).

[NattyNarwhal]

I think it's lifetimes. Putting assert(stmt_res == stmt->driver_data) in lob_stream_read indicates that they don't match, when they should. Probably because stmt has been reused/not really zeroed out, it points to a differrent stmt_handle, and that points to memory that the allocator has overwritten with other stuff.

From the looks of it, the LOB should add a reference to the statement, and we should be keeping track of statements in general.

fwiw, more printf debugging slurry, so you can kinda see the logic where I go with it:

 ** stmt_allocate_column_descriptors:   stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0
  *                                     no cols 1
  *                                     columns 0x7faccbd68120
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd68120 colno 0
 *                      col name 1 type 12 size 128 scale/precision 0
 ** stmt_cleanup:   stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0
  * column desc?...
  * freeing col# 0 string 0x7faccbcdd540...
  * freeing columns 0x7faccbd68120
 ** stmt_allocate_column_descriptors:   stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0
  *                                     no cols 10
  *                                     columns 0x7faccbd54000
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd54000 colno 0
 *                      col name NAME type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd54040 colno 1
 *                      col name COLNAME type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd54080 colno 2
 *                      col name TYPENAME type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd540c0 colno 3
 *                      col name CODEPAGE type 5 size 6 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd54100 colno 4
 *                      col name NULLS type 1 size 1 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd54140 colno 5
 *                      col name LENGTH type 4 size 11 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd54180 colno 6
 *                      col name SCALE type 5 size 6 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd541c0 colno 7
 *                      col name COMMENT type 12 size 254 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd54200 colno 8
 *                      col name AUTOINCREMENT type 4 size 11 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd54240 colno 9
 *                      col name DEFAULT type -99 size 65536 scale/precision 0
 ** create_lob_stream:  stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 colno 9 col_res 0x7faccbd54240 data 0x7faccbd62118 php stream 0x7faccbd7bdc0
 ** lob_stream_read:    php_stream 0x7faccbd7bdc0 buf (nil) size 0
 **                     data 0x7faccbd62118 col_res 0x7faccbd54240 stmt_res 0x7faccbc46cd0 stmt 0x7faccbd8c540
 *                      -> ctype -2
 *                      -> rc 0 read bytes -1
 *                      ret: NULL
 ** lob_stream_close:    stream 0x7faccbd7bdc0 data 0x7faccbd62118
 ** create_lob_stream:  stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 colno 9 col_res 0x7faccbd54240 data 0x7faccbd62118 php stream 0x7faccbd7bdc0
 ** lob_stream_read:    php_stream 0x7faccbd7bdc0 buf (nil) size 0
 **                     data 0x7faccbd62118 col_res 0x7faccbd54240 stmt_res 0x7faccbc46cd0 stmt 0x7faccbd8c540
 *                      -> ctype -2
 *                      -> rc 1 read bytes 4
 *                      ret: read bytes 0
 ** create_lob_stream:  stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 colno 9 col_res 0x7faccbd54240 data 0x7faccbd62150 php stream 0x7faccbd7c040
 ** lob_stream_read:    php_stream 0x7faccbd7c040 buf (nil) size 0
 **                     data 0x7faccbd62150 col_res 0x7faccbd54240 stmt_res 0x7faccbc46cd0 stmt 0x7faccbd8c540
 *                      -> ctype -2
 *                      -> rc 1 read bytes 4
 *                      ret: read bytes 0
 ** stmt_cleanup:   stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0
  * column desc?...
  * freeing col# 0 string 0x7faccbcdd180...
  * freeing col# 1 string 0x7faccbcccf00...
  * freeing col# 2 string 0x7faccbcccc00...
  * freeing col# 3 string 0x7faccbff1208...
  * freeing col# 4 string 0x7faccbd531b8...
  * freeing col# 5 string 0x7faccbd57870...
  * freeing col# 6 string 0x7faccbd531e0...
  * freeing col# 7 string 0x7faccbe82c40...
  * freeing col# 8 string 0x7faccbd57a20...
  * freeing columns 0x7faccbd54000
 ** stmt_allocate_column_descriptors:   stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0
  *                                     no cols 5
  *                                     columns 0x7faccbd65f00
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd65f00 colno 0
 *                      col name NAME type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd65f40 colno 1
 *                      col name KEY_NAME type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd65f80 colno 2
 *                      col name COLUMN_NAME type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd65fc0 colno 3
 *                      col name PRIMARY type 4 size 11 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd66000 colno 4
 *                      col name NON_UNIQUE type 4 size 11 scale/precision 0
 ** stmt_cleanup:   stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0
  * column desc?...
  * freeing col# 0 string 0x7faccbcccf00...
  * freeing col# 1 string 0x7faccbcdd180...
  * freeing col# 2 string 0x7faccbcdd540...
  * freeing col# 3 string 0x7faccbd57a20...
  * freeing col# 4 string 0x7faccbd57870...
  * freeing columns 0x7faccbd65f00
 ** stmt_allocate_column_descriptors:   stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0
  *                                     no cols 7
  *                                     columns 0x7faccbd47800
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd47800 colno 0
 *                      col name NAME type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd47840 colno 1
 *                      col name LOCAL_COLUMN type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd47880 colno 2
 *                      col name FOREIGN_TABLE type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd478c0 colno 3
 *                      col name FOREIGN_COLUMN type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd47900 colno 4
 *                      col name INDEX_NAME type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd47940 colno 5
 *                      col name ON_UPDATE type 12 size 8 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0 col_res 0x7faccbd47980 colno 6
 *                      col name ON_DELETE type 12 size 8 scale/precision 0
 ** stmt_cleanup:   stmt 0x7faccbd8c540 stmt_res 0x7faccbc46cd0
  * column desc?...
  * freeing col# 0 string 0x7faccbcdd180...
  * freeing col# 1 string 0x7faccbcccf00...
  * freeing col# 2 string 0x7faccbcccc00...
  * freeing col# 3 string 0x7faccbccca80...
  * freeing col# 4 string 0x7faccbccc900...
  * freeing col# 5 string 0x7faccbd57870...
  * freeing col# 6 string 0x7faccbd57a20...
  * freeing columns 0x7faccbd47800
 ** stmt_allocate_column_descriptors:   stmt 0x7faccbd8c540 stmt_res 0x7faccc7a7a50
  *                                     no cols 2
  *                                     columns 0x7faccbc3aa00
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccc7a7a50 col_res 0x7faccbc3aa00 colno 0
 *                      col name NAME type 12 size 128 scale/precision 0
 ** ibm_stmt_describer: stmt 0x7faccbd8c540 stmt_res 0x7faccc7a7a50 col_res 0x7faccbc3aa40 colno 1
 *                      col name REMARKS type 12 size 254 scale/precision 0
 ** stmt_cleanup:   stmt 0x7faccbd8c540 stmt_res 0x7faccc7a7a50
  * column desc?...
  * freeing col# 0 string 0x7faccbccc900...
  * freeing col# 1 string 0x7faccbd7bc80...
  * freeing columns 0x7faccbc3aa00
 ** lob_stream_read:    php_stream 0x7faccbd7bdc0 buf 0x7facca247000 size 8192
 **                     data 0x7faccbd62118 col_res 0x4457454956535b93 stmt_res 0x7faccbc46cd0 stmt 0x7faccbd8c540
Segmentation fault (core dumped)

[NattyNarwhal]

I have the (linked by GH) PR that makes it no longer crash, but it still has some issues, it seems...

Time: 00:05.251, Memory: 20.00 MB

There were 7 failures:

1) Doctrine\DBAL\Tests\Functional\BlobTest::testInsert
Failed asserting that 0 matches expected 1.

/home/calvin/src/dbal/tests/Functional/BlobTest.php:47

2) Doctrine\DBAL\Tests\Functional\BlobTest::testInsertNull
Failed asserting that 0 matches expected 1.

/home/calvin/src/dbal/tests/Functional/BlobTest.php:62

3) Doctrine\DBAL\Tests\Functional\BlobTest::testInsertProcessesStream
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'xxx[...]'
+''

/home/calvin/src/dbal/tests/Functional/BlobTest.php:180
/home/calvin/src/dbal/tests/Functional/BlobTest.php:87

4) Doctrine\DBAL\Tests\Functional\BlobTest::testSelect
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'test'
+''

/home/calvin/src/dbal/tests/Functional/BlobTest.php:180
/home/calvin/src/dbal/tests/Functional/BlobTest.php:102

5) Doctrine\DBAL\Tests\Functional\BlobTest::testUpdate
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'test2'
+''

/home/calvin/src/dbal/tests/Functional/BlobTest.php:180
/home/calvin/src/dbal/tests/Functional/BlobTest.php:122

6) Doctrine\DBAL\Tests\Functional\BlobTest::testUpdateProcessesStream
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'test2'
+''

/home/calvin/src/dbal/tests/Functional/BlobTest.php:180
/home/calvin/src/dbal/tests/Functional/BlobTest.php:150

7) Doctrine\DBAL\Tests\Functional\BlobTest::testBindParamProcessesStream
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'test'
+''

/home/calvin/src/dbal/tests/Functional/BlobTest.php:180
/home/calvin/src/dbal/tests/Functional/BlobTest.php:170

FAILURES!
Tests: 7, Assertions: 17, Failures: 7.
[Thu Feb 16 16:19:53 2023]  Script:  '/home/calvin/src/dbal/vendor/bin/phpunit'
/home/calvin/src/php-src/Zend/zend_objects_API.h(92) :  Freeing 0x00007f2589c8b700 (368 bytes), script=/home/calvin/src/dbal/vendor/bin/phpunit
Last leak repeated 5 times
[Thu Feb 16 16:19:53 2023]  Script:  '/home/calvin/src/dbal/vendor/bin/phpunit'
/home/calvin/src/php-src/Zend/zend_objects_API.h(92) :  Freeing 0x00007f258b655b40 (48 bytes), script=/home/calvin/src/dbal/vendor/bin/phpunit
Last leak repeated 1 time
=== Total 8 memory leaks detected ===

[phansys]

I have the (linked by GH) PR that makes it no longer crash, but it still has some issues, it seems...

As doctrine/dbal#5924 is an attempt to restore support for this extension, maybe there is something missing to consider. I make this comment in order to prevent taking that PR as a solid base for the final implementation.

[NattyNarwhal]

As https://github.com/doctrine/dbal/pull/5924 is an attempt to restore support for this extension, maybe there is something missing to consider. I make this comment in order to prevent taking that PR as a solid base for the final implementation.

I'll note that at least for IBM i, we ('we' being IBM and Seiden Group) don't recommend PDO_IBM and ibm_db2 for any new applications, but rather ODBC with the IBM i ODBC driver instead. That notice is on the PECL package for ibm_db2 at least, but I should probably make it more obvious there and here. We're still maintaining it for existing legacy users on IBM i though. Of course, I don't know how IBM feels about it for Db2 LUW users.

Not to say any issues found in PDO_IBM and ibm_db2 shouldn't be fixed, of course. Including this one.

[phansys]

The recommendation is for connections made from the same server or even for remote connections? In my use case, I need to connect to a DB2 for i database from a dockerized Debian mounted on a Centos instance :face_with_spiral_eyes:

In the ibm_db2 PECL extension, it seems to be the recommended way for any connection:

Note that for IBM i, IBM and Seiden Group recommend new applications use ODBC instead.

In the IBM support docs, it seems to recommend ODBC just for applications running on the same database server:

For users of PHP running on IBM i, the ODBC interfaces (odbc/PDO_ODBC) are the recommended technique for connecting to the database. The ODBC driver for PASE is available via "IBM i Access Client Solutions - PASE Application Package" located at the main IBM i Access - Client Solutions page. This connection type does not utilize SQL CLI.

[NattyNarwhal]

I think there's some politicking going on - using the Db2 LUW driver to connect for Db2 on i requires the Db2 Connect LPP on i, which I think got sold to Rocket Software. IBM itself maintains the ODBC driver for i, which runs on i/Winddows/Linux and doesn't need any extra licensed products. Again, for LUW client to LUW server, none of this is relevant.

[phansys]

Oh, I see. Thank you very much for your explanation.

Off topic:

I think that's why I can use ibm_db2 extension (with the CLIDRIVER) to connect the containerized app without issues to a container using the ibmcom/db2 image, but when I try to perform the same connection against a DB2 for i server, I receive this message:

[IBM][CLI Driver] SQL1598N  An attempt to connect to the database server failed because of a licensing problem.  SQLSTATE=42968 SQLCODE=-1598

BTW, using the IBM i Access ODBC approach, I can connect against the DB2 for i server, but not anymore with the Docker container:

SQLSTATE[08004] SQLDriverConnect: 10061 [IBM][System i Access ODBC Driver]Communication link failure. comm rc=10061 - CWBCO1049 - The IBM i server application  is not started, or the connection was blocked by a firewall

[kadler]

@phansys The IBM i team recommends using IBM i ODBC driver (and associated middleware, eg. pdo_odbc, etc) to connect to IBM i databases. We recommend against using ibm_db2 / pdo_ibm unless you're running on IBM i and only then for existing applications that haven't been migrated to ODBC.

it seems to recommend ODBC just for applications running on the same database server:

I'm pretty sure this doc is really just meant for PHP users who run on IBM i and not making any recommendations for PHP users on other platforms connecting to IBM i. FWIW, I think nearly all IBM i customers running PHP do so on IBM i (I don't know of a single customer that doesn't, or at least runs on a different platform and then connects to IBM i).

I'll get the wording improved to be clear that we recommend ODBC in all cases.

[phansys]

Working on the Doctrine driver, I found this suspicious definition:

public function getBlobTypeDeclarationSQL(array $column)
{
    // todo blob(n) with $column['length'];
    return 'BLOB(1M)';
}

The same behavior exists with CLOB. Just leaving the finding here, in case it could be related to this issue.