sabineWalter commented 8 months ago

Hello,

create a small table in oracle as `drop table clob_sizes_2 /

create table clob_sizes_2 (ID int generated always as identity, myClob clob);

declare myText clob := ''; begin for i in 1 .. 1000 loop myText := concat(myText, lpad(1,1024,'0')); insert into clob_sizes_2 (myClob) values (myText); end loop; commit; end; / commit;

alter table clob_sizes_2 add constraint p_id primary key (id)



Setup in PostgreSQL 

`CREATE EXTENSION oracle_fdw;
CREATE SERVER newbisdev FOREIGN DATA WRAPPER oracle_fdw  OPTIONS (dbserver '//10.162.75.236:1521/NEWBISTEST');
GRANT USAGE ON FOREIGN SERVER newbisdev TO postgres;
CREATE USER MAPPING FOR postgres SERVER newbisdev OPTIONS (user 'EMATRIX', password 'xxxxx');

CREATE FOREIGN TABLE ft_clog_sizes_2 ( 
  id     integer NOT NULL,
  myclob text ) 
SERVER newbisdev 
OPTIONS (schema 'EMATRIX', table 'CLOB_SIZES_2', 
         readonly 'yes', 
         sample_percent '5', lob_prefetch '8388608');
`

Trying to select the myClob column in PostgreSQL 
a) if length(myClob) < 21900: everythink works fin
b) if length(myClob) > 21900: I get en empty clob from oracle_fdw

`pocnewbis=#  select length(myclob) from ft_clog_sizes_2 where id=20;
 length
--------
  20480
(1 row)

pocnewbis=#  select length(myclob) from ft_clog_sizes_2 where id=21;
 length
--------
  21504
(1 row)

pocnewbis=#  select length(myclob) from ft_clog_sizes_2 where id=22;
 length
--------
      0
(1 row)`

pls help me

I'm using PostgreSQL 14.10
Oracle DB is of Version = 19.20

`                          oracle_diag
---------------------------------------------------------------
 oracle_fdw 2.6.0, PostgreSQL 14.10, Oracle client 21.11.0.0.0
(1 row)
`

Regards

laurenz commented 8 months ago

As we discussed offline: can you please send me a data pump export of the Oracle table created with expdp ... VERSION=18.1? I need that to reproduce the problem.

sabineWalter commented 8 months ago

executed expdp with parfile DIRECTORY=EXPORT DUMPFILE=dws_transfer.2024-01-31.dmp LOGFILE=2024-01-31.exp.log CONTENT=ALL FLASHBACK_TIME=systimestamp VERSION=18.1 PARALLEL=1 EXCLUDE=STATISTICS TABLES=EMATRIX.DWS_TRANSFER

will attach dmp.gz file dws_transfer.2024-01-31.dmp.gz

sabineWalter commented 8 months ago

to my mind the testcase sql script will create easy data to reproduce the issue ...

laurenz commented 8 months ago

I couldn't import the dump, but I tried your clob_sizes_2 with

Oracle XE 18.4, PostgreSQL 17devel, oracle_fdw 2.7.0devel on Fedora Linux 39 (my development environment)
Oracle Free 23c, PGEE 14.10, oracle_fdw 2.7.0devel on Rocky Linux 8

On the first system it works like a charm, on the second system I get random crashes for larger LOBs, for example

#0  0x00007f211b1aab8f in raise () from /lib64/libpthread.so.0
#1  0x00007f20fcafdd0f in skgesigOSCrash () from /opt/oracle/product/23c/dbhomeFree/lib/libclntsh.so.23.1
#2  0x00007f20fd2b43f7 in kpeDbgSignalHandler () from /opt/oracle/product/23c/dbhomeFree/lib/libclntsh.so.23.1
#3  0x00007f20fcafe018 in skgesig_sigactionHandler () from /opt/oracle/product/23c/dbhomeFree/lib/libclntsh.so.23.1
#4  <signal handler called>
#5  0x00007f20fb144c10 in __intel_avx_rep_memcpy () from /opt/oracle/product/23c/dbhomeFree/lib/libclntsh.so.23.1
#6  0x00007f20fb414b74 in kpulbc_get_pref_data () from /opt/oracle/product/23c/dbhomeFree/lib/libclntsh.so.23.1
#7  0x00007f20f9e3bf6b in kpulbcr () from /opt/oracle/product/23c/dbhomeFree/lib/libclntsh.so.23.1
#8  0x00007f20f9e2b962 in kpulfrdarr () from /opt/oracle/product/23c/dbhomeFree/lib/libclntsh.so.23.1
#9  0x00007f20f9e29eee in kpulfrd () from /opt/oracle/product/23c/dbhomeFree/lib/libclntsh.so.23.1
#10 0x00007f20fb20c983 in OCILobRead2 () from /opt/oracle/product/23c/dbhomeFree/lib/libclntsh.so.23.1
#11 0x00007f20fe653144 in oracleGetLob (session=0x3028bd8, locptr=0x3025538, type=ORA_TYPE_CLOB, value=0x7ffda4c222d0, value_len=0x7ffda4c222d8, 
    trunc=0) at oracle_utils.c:2397
#12 0x00007f20fe64be18 in convertTuple (fdw_state=0x3024d80, index=1, values=0x3024228, nulls=0x3024238, trunc_lob=false) at oracle_fdw.c:6703
#13 0x00007f20fe63bcaa in oracleIterateForeignScan (node=0x3023c00) at oracle_fdw.c:1401
#14 0x00000000006a5a83 in ForeignNext (node=node@entry=0x3023c00) at nodeForeignscan.c:62
#15 0x0000000000696202 in ExecScanFetch (recheckMtd=0x6a5ad0 <ForeignRecheck>, accessMtd=0x6a5a10 <ForeignNext>, node=0x3023c00)
    at execScan.c:132
#16 ExecScan (node=0x3023c00, accessMtd=0x6a5a10 <ForeignNext>, recheckMtd=0x6a5ad0 <ForeignRecheck>) at execScan.c:198
#17 0x000000000068d603 in ExecProcNode (node=0x3023c00) at ../../../src/include/executor/executor.h:260
#18 ExecutePlan (execute_once=<optimized out>, dest=0x2eb7bf0, direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>, 
    operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0x3023c00, estate=0x3023988) at execMain.c:1551
#19 standard_ExecutorRun (queryDesc=0x2e0a178, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:361
#20 0x0000000000802beb in PortalRunSelect (portal=0x2e59f38, forward=<optimized out>, count=0, dest=<optimized out>) at pquery.c:921
#21 0x0000000000803efd in PortalRun (portal=0x2e59f38, count=9223372036854775807, isTopLevel=<optimized out>, run_once=<optimized out>, 
    dest=0x2eb7bf0, altdest=0x2eb7bf0, qc=0x7ffda4c226c0) at pquery.c:765
#22 0x000000000080022b in exec_simple_query ()
#23 0x0000000000801a99 in PostgresMain ()
#24 0x000000000077e147 in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4958
#25 BackendStartup (port=<optimized out>) at postmaster.c:4680
#26 ServerLoop (receive_encryption_key=receive_encryption_key@entry=false) at postmaster.c:1901
#27 0x000000000077f244 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x2de18e0) at postmaster.c:1533
#28 0x0000000000500383 in main (argc=3, argv=0x2de18e0) at main.c:209

I am really not sure what to make of this. Perhaps it is a bug in oracle_fdw (but then, why does it work on my development machine?), perhaps it is a bug in Oracle 23c, perhaps it is some kind of resource problem on the second system (OOM that is not properly handled?), perhaps there is a subtly incompatibility in PGEE.

The actual crashes happen somewhere deep in Oracle code, but that is certainly no proof that it is Oracle's fault.

I honestly don't know what to make of this.

I never observe the truncated LOBs; they seep to point at some memory management bug, but that is only a guess.

I am afraid I'll have to give up on this for now, until I get some clue or inspiration.

sabineWalter commented 8 months ago

I'm using 19.20 EE and 19.18 EE of oracle Inside of both version I can execute select min(length(myclob)) as MINLEN from clob_sizes_2; and get then expected return value 1024, no error no Return value 0

in postgres then same sql will return 0

laurenz commented 8 months ago

I get crashes with PostgreSQL v14 and v16 on the Rocky 8 test VM, but not on my machine.

I'm pretty sure it is a bug in oracle_fdw, but I cannot seem to fix it.

sabineWalter commented 8 months ago

How can I help / assist you to fix the bug / identify which components are working fine / will crashes ?

On pg v14.10 I do not have any crash, but when clob is to large, I do not get any data from oracle to pg ... see above

laurenz commented 8 months ago

I don't think you can do more, unless you can find the bug.

sabineWalter commented 8 months ago

if you want to check blob as well I created a testcase - based on blob_sizes_2

create in oracle the following function create or replace function clob2blob(AClob CLOB) return BLOB is Result BLOB; o1 integer; o2 integer; c integer; w integer; begin o1 := 1; o2 := 1; c := 0; w := 0; DBMS_LOB.CreateTemporary(Result, true); DBMS_LOB.ConvertToBlob(Result, AClob, length(AClob), o1, o2, 0, c, w); return(Result); end clob2blob; /

create a new table blob_sizes_2: create table ematrix.blob_sizes_2(id number, myblob blob);

and fill the table: insert into blob_sizes_2(id, myblob) select id, clob2blob(myclob) from clob_sizes_2 order by id; commit;

sabineWalter commented 8 months ago

what do you think / assume where the bug will be?

in which file? in which function ?

laurenz commented 7 months ago

I don't know, but rest assured that I have not given up on this. It has all the earmarks of a memory management bug, which are notoriously hard to find. That I have to deal with closed source software doesn't hake it any easier.

Steps that I have planned:

See if it works with an older version of oracle_fdw (2.5.0). If yes, git bisect to identify where it stopped working.
Try with an older version of Oracle client to see if it is related to some change in Oracle code.

I cannot promise a fast resolution.

sabineWalter commented 7 months ago

to my mind it should be a memory issue and I agree to you it is very hard to find. therefor I asked the last question to assist you find the root cause. If you have any idea where I can create other testcases, do in different setup like other Oracle client let me know what exactly I should test ...

sabineWalter commented 7 months ago

analyzing your oracle_fdw and the stack trace I come to same result: it is a bug in Oracle 23c,

try to use a client max 21c, better 19.xx (23c is too new, => to many bugs are not removed, today)

sabineWalter commented 7 months ago

when I use Oracle Client 21.c I get no crash but 0 bytes retunred when Size of clobs are greater > 21k

Can you generate a smal c program retrieving one lob with id = 30 using 23c => should crash using 21c => will not get any data

so I can open a ticket at Oracle Support

laurenz commented 7 months ago

I am not certain that that is a bug in Oracle - it could be that the bug just manifests differently in different versions. If nothing else gets me further, I will eventually write such a test program.

laurenz commented 7 months ago

The problem does not occur with release 2.5.0. I ran git bisect and the commit that broke the code was d679a73d02057fd674fc78fe5f25183e5c9a172b That is tough, because it is a large commit. But at least it is pretty clear now that it is an oracle_fdw bug.

laurenz commented 7 months ago

Issue #614 is probably the same problem.

laurenz commented 7 months ago

I wrote a standalone test program to reproduce the crash: crash.c.gz.

It failed to crash on either system, but the behavior is slightly different:

Oracle 18.4:

...
execute the statement
get the LOB length
LOB length = 51200
read the first chunk of the LOB
got 10000 bytes
read the next chunk of the LOB
got 10000 bytes
read the next chunk of the LOB
got 10000 bytes
read the next chunk of the LOB
got 10000 bytes
read the next chunk of the LOB
got 10000 bytes
read the next chunk of the LOB
got 1200 bytes
received LOB length = 51200
...

Oracle 23c:

...
execute the statement
get the LOB length
LOB length = 51200
read the first chunk of the LOB
got 40001 bytes
read the next chunk of the LOB
got 11199 bytes
received LOB length = 51200
...

The 23c behavior is actually more correct; I guess Oracle fixed a bug in OCILobRead2(), and the new behavior tickles a bug in my code.

sabineWalter commented 7 months ago

1) pls can you assist me to compile and run this small c program under Linux RHEL So I will it also again 19.18

2) do I understand you correct it seems not to be bug of oracle_fdw it seems to be a bug of the oracle client ? if yes, which oracle Client do you have to run again a) again 18c database b) again 23c database

laurenz commented 7 months ago

I still think that the bug is in oracle_fdw, and this is confirmed by the fact that my test program works with Oracle 23c.

The exact build method will depend on whether you are using Instant Client or not.

With Instant Client:

gcc -Wall -o crash crash.c -I $ORACLE_HOME/sdk/include -L $ORACLE_HOME -l clntsh -Wl,-rpath,$ORACLE_HOME

With and installation by the Oraclr Universal Installer:

gcc -Wall -o crash crash.c -I $ORACLE_HOME/rdbms/public -L $ORACLE_HOME/lib -l clntsh -Wl,-rpath,$ORACLE_HOME/lib

laurenz commented 7 months ago

More data points: With Oracle Instant Client 21.13, oracle_fdw on my machine returns zero length LOBs, and my little test program that works just fine with Oracle 18.4, Oracle 19.18 and Oracle 23 crashes with

#11 0x00007f0f6525e7d0 in __libc_message (fmt=fmt@entry=0x7f0f653db56a "%s\n") at ../sysdeps/posix/libc_fatal.c:150
#12 0x00007f0f652d17a5 in malloc_printerr (str=str@entry=0x7f0f653d90c0 "malloc(): corrupted top size") at malloc.c:5765
#13 0x00007f0f652d51cc in _int_malloc (av=av@entry=0x7f0f6540fac0 <main_arena>, bytes=bytes@entry=16384) at malloc.c:4440
#14 0x00007f0f652d6e61 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3747
#15 0x00007f0f627b35a0 in nsviniv () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#16 0x00007f0f6278f748 in nsvntx_ini () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#17 0x00007f0f6278708b in nsvrecv () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#18 0x00007f0f627ed0ac in niovrc () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#19 0x00007f0f614ce145 in ttciovconv () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#20 0x00007f0f6129a507 in kpulbcr () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#21 0x00007f0f643383bb in ttcdrv () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#22 0x00007f0f642f79ad in nioqwa () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#23 0x00007f0f642b4e6c in upirtrc () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#24 0x00007f0f642cc4e1 in kpurcsc () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#25 0x00007f0f6128c46c in kpulrcsc () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#26 0x00007f0f61288129 in kpulfrdarr () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#27 0x00007f0f6128753e in kpulfrd () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#28 0x00007f0f622593df in OCILobRead2 () from /home/laurenz/instantclient_21_13/libclntsh.so.21.1
#29 0x00000000004019cf in main (argc=1, argv=0x7ffdbec1f018) at crash.c:200

Now that may well be a bug in Oracle 21.13...

This is tough. I am getting no lead, and Oracle's software quality isn't helping.

philflorent commented 7 months ago

Hi,

11.2.0.4 was a LTR (long term release) but premier, extended and market driven support have ended (2022). It is still widely used (sustaining support). 18c is an innovation release and it is EOL (2021) 19c is the current LTR. Waived extended support will end on 2026, extended support will end on 2027. 21c is an innovation release and it will be EOL on 2025. 23c is also a LTR but it is not generally available on premise, only Free edition and client are available on some platforms. It's a bit too early to use it on production imo.

=> 2024-02...I would still use 19c client with oracle-fdw in production if I were you. 21c has notably 0 interest since it will be EOL earlier than 19c !

That said, I will definitely have a look asap with various LTR (are or were) supported combos..

Best regards, Phil

laurenz commented 7 months ago

Ha! I think I found the solution - with random trial and error.

Could you try this version and see if it fixes the problem?

sabineWalter commented 7 months ago

extract the file
run "make"
run "make install"

pls commit

laurenz commented 7 months ago

pls commit

Does that mean that you can confirm that my patch fixes the bug on your system?

sabineWalter commented 7 months ago

pls confirm that these three steps are fine to create a lib / extension to implement in postgresql ? if yes, I will do these steps and test it on myy system a) with clob_sizes_2 b) with another table (real data)

sabineWalter commented 7 months ago

in the README Section 6 Installation I found

You will need PostgreSQL headers and PGXS installed (if your PostgreSQL was installed with packages, install the development package). You need to install Oracle's C header files as well (SDK package for Instant Client). If you use the Instant Client ZIP files provided by Oracle, you will have to create a symbolic link from libclntsh.so to the actual shared library file yourself.

a) I installed postgres with rpms what is name of the devel rpm I have to install?

b) I installed oracle-instantclient-basic-21.11.0.0.0-1.el8.x86_64 which link do I have t create concerning libclntsh.so?

thx

sabineWalter commented 7 months ago

/usr/lib/oracle/21/client64/lib /usr/lib/oracle/21/client64/lib/libclntsh.so /usr/lib/oracle/21/client64/lib/libclntsh.so.10.1 /usr/lib/oracle/21/client64/lib/libclntsh.so.11.1 /usr/lib/oracle/21/client64/lib/libclntsh.so.12.1 /usr/lib/oracle/21/client64/lib/libclntsh.so.18.1 /usr/lib/oracle/21/client64/lib/libclntsh.so.19.1 /usr/lib/oracle/21/client64/lib/libclntsh.so.20.1 /usr/lib/oracle/21/client64/lib/libclntsh.so.21.1 /usr/lib/oracle/21/client64/lib/libclntshcore.so /usr/lib/oracle/21/client64/lib/libclntshcore.so.21.1 /usr/lib/oracle/21/client64/lib/libnnz21.so /usr/lib/oracle/21/client64/lib/libocci.so /usr/lib/oracle/21/client64/lib/libocci.so.10.1 /usr/lib/oracle/21/client64/lib/libocci.so.11.1 /usr/lib/oracle/21/client64/lib/libocci.so.12.1 /usr/lib/oracle/21/client64/lib/libocci.so.18.1 /usr/lib/oracle/21/client64/lib/libocci.so.19.1 /usr/lib/oracle/21/client64/lib/libocci.so.20.1 /usr/lib/oracle/21/client64/lib/libocci.so.21.1 /usr/lib/oracle/21/client64/lib/libocci_gcc53.so /usr/lib/oracle/21/client64/lib/libocci_gcc53.so.21.1 /usr/lib/oracle/21/client64/lib/libociei.so /usr/lib/oracle/21/client64/lib/libocijdbc21.so /usr/lib/oracle/21/client64/lib/liboramysql.so

sabineWalter commented 7 months ago

currently installed:

postgresql14-contrib-14.10-1PGDG.rhel8.x86_64
postgresql-jdbc-javadoc-42.6.0-1.rhel8.noarch
postgresql14-devel-14.10-1PGDG.rhel8.x86_64
postgresql14-libs-14.10-1PGDG.rhel8.x86_64
postgresql14-server-14.10-1PGDG.rhel8.x86_64
postgresql14-plpython3-14.10-1PGDG.rhel8.x86_64
postgresql14-odbc-13.02.0000-1PGDG.rhel8.x86_64
postgresql-jdbc-42.6.0-1.rhel8.noarch
postgresql14-pltcl-14.10-1PGDG.rhel8.x86_64
postgresql14-tcl-3.0.0-1.rhel8.x86_64
postgresqltuner-1.0.1-4.el7.noarch
postgresql14-14.10-1PGDG.rhel8.x86_64
postgresql14-plperl-14.10-1PGDG.rhel8.x86_64
postgresql14-docs-14.10-1PGDG.rhel8.x86_64

laurenz commented 7 months ago

I see. Try this:

extract the archive and cd into it

build with

make ORACLE_HOME=/usr/lib/oracle/21/client64 PATH=/usr/pgsql-14/bin:$PATH

install with

sudo make install ORACLE_HOME=/usr/lib/oracle/21/client64 PATH=/usr/pgsql-14/bin:$PATH

sabineWalter commented 7 months ago

as $ echo $PATH /export/sw_netapp_db/POSTGRES/BIN:/home/postgres/.local/bin:/home/postgres/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/pgsql-14/bin

I started

$ make ORACLE_HOME=/usr/lib/oracle/21/client64

and get

Makefile:30: /usr/lib64/pgsql/pgxs/src/makefiles/pgxs.mk: No such file or directory make: *** No rule to make target '/usr/lib64/pgsql/pgxs/src/makefiles/pgxs.mk'. Stop.

Howto solve this issue ?

sabineWalter commented 7 months ago

extract the archive and cd into it

=> done

build with

make ORACLE_HOME=/usr/lib/oracle/21/client64 PATH=/usr/pgsql-14/bin:$PATH

get following error:

$ make ORACLE_HOME=/usr/lib/oracle/21/client64 PATH=/usr/pgsql-14/bin:$PATH gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -I"/usr/lib/oracle/21/client64/sdk/include" -I"/usr/lib/oracle/21/client64/oci/include" -I"/usr/lib/oracle/21/client64/rdbms/public" -I"/usr/lib/oracle/21/client64/" -I. -I./ -I/usr/pgsql-14/include/server -I/usr/pgsql-14/include/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include -c -o oracle_fdw.o oracle_fdw.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -I"/usr/lib/oracle/21/client64/sdk/include" -I"/usr/lib/oracle/21/client64/oci/include" -I"/usr/lib/oracle/21/client64/rdbms/public" -I"/usr/lib/oracle/21/client64/" -I. -I./ -I/usr/pgsql-14/include/server -I/usr/pgsql-14/include/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include -c -o oracle_utils.o oracle_utils.c oracle_utils.c:22:10: fatal error: oci.h: No such file or directory

include

      ^~~~~~~

compilation terminated. make: *** [: oracle_utils.o] Error 1

I've installed the instant client, therefore

$ ls -l /usr/lib/oracle/21/client64 total 8 drwxr-xr-x 2 root root 4096 Nov 15 13:42 bin drwxr-xr-x 3 root root 4096 Nov 15 13:42 lib

in detail:

$ ls -lR /usr/lib/oracle/21/client64 /usr/lib/oracle/21/client64: total 8 drwxr-xr-x 2 root root 4096 Nov 15 13:42 bin drwxr-xr-x 3 root root 4096 Nov 15 13:42 lib

/usr/lib/oracle/21/client64/bin: total 104 -rwxr-xr-x 1 root root 41624 Jun 28 2023 adrci -rwxr-xr-x 1 root root 59160 Jun 28 2023 genezi

/usr/lib/oracle/21/client64/lib: total 243128 lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root -rwxr-xr-x 1 root root lrwxrwxrwx 1 root root -rwxr-xr-x 1 root root -rwxr-xr-x 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root -rwxr-xr-x 1 root root lrwxrwxrwx 1 root root -rwxr-xr-x 1 root root -rwxr-xr-x 1 -rwxr-xr-x 1 root root -rwxr-xr-x 1 root root drwxr-xr-x 3 root root -rw-r--r-- 1 root root -rw-r--r-- 1 root root 17 Jun 28 2023 libclntsh.so -> libclntsh.so.21.1 17 Jun 28 2023 libclntsh.so.10.1 -> libclntsh.so.21.1 17 Jun 28 2023 libclntsh.so.11.1 -> libclntsh.so.21.1 17 Jun 28 2023 libclntsh.so.12.1 -> libclntsh.so.21.1 17 Jun 28 2023 libclntsh.so.18.1 -> libclntsh.so.21.1 17 Jun 28 2023 libclntsh.so.19.1 -> libclntsh.so.21.1 17 Jun 28 2023 libclntsh.so.20.1 -> libclntsh.so.21.1 83674312 Jun 28 2023 libclntsh.so.21.1 21 Jun 28 2023 libclntshcore.so -> libclntshcore.so.21.1 8055264 Jun 28 2023 libclntshcore.so.21.1 5706440 Jun 28 2023 libnnz21.so 15 Jun 28 2023 libocci.so -> libocci.so.21.1 15 Jun 28 2023 libocci.so.10.1 -> libocci.so.21.1 15 Jun 28 2023 libocci.so.11.1 -> libocci.so.21.1 15 Jun 28 2023 libocci.so.12.1 -> libocci.so.21.1 15 Jun 28 2023 libocci.so.18.1 -> libocci.so.21.1 15 Jun 28 2023 libocci.so.19.1 -> libocci.so.21.1 15 Jun 28 2023 libocci.so.20.1 -> libocci.so.21.1 2370680 Jun 28 2023 libocci.so.21.1 21 Jun 28 2023 libocci_gcc53.so -> libocci_gcc53.so.21.1 954984 Jun 28 2023 libocci_gcc53.so.21.1 root root 142717328 Jun 28 2023 libociei.so 153328 Jun 28 2023 libocijdbc21.so 115584 Jun 28 2023 liboramysql.so 4096 Nov 15 13:42 network 5153052 Jun 28 2023 ojdbc8.jar 31869 Jun 28 2023 xstreams.jar

/usr/lib/oracle/21/client64/lib/network: total 4 drwxr-xr-x 2 root root 4096 Nov 15 13:42 admin

/usr/lib/oracle/21/client64/lib/network/admin: total 4 -rw-r--r-- 1 root root 502 Jun 28 2023 README

Howto solve this comile issue ?

laurenz commented 7 months ago

You will have to install the Oracle SDK.

With Instant Client you have to install the SDK package, and I don't know what exactly you have to install with Universal Installer: something like "SDK" or "C headers".

Or you could try this binary that I built for RHEL 8, Oracle 21 and PostgreSQL v14.

sabineWalter commented 7 months ago

thx for compiling

pls confirm that I have to replace the existing oracle_fdw.so (2.6.0) by this new one

laurenz commented 7 months ago

That's the idea, yes.

sabineWalter commented 7 months ago

the current oracle_fdw.so is lying in dir = /usr/pgsql-14/lib, size = 171760 bytes, Perms = -rwxr-xr-x 1 root root
the new compiled has size = 748048

sabineWalter commented 7 months ago

drop extension oracle_fdw cascade;
create extension oracle_fdw;
# select oracle_diag();
                                                   oracle_diag
-----------------------------------------------------------------------------------------------------------------
 oracle_fdw 2.7.0devel, PostgreSQL 14.10, Oracle client 21.11.0.0.0, ORACLE_HOME=/usr/lib/oracle/21/client64/lib
(1 row)

sabineWalter commented 7 months ago

CREATE SERVER newbisdev FOREIGN DATA WRAPPER oracle_fdw  OPTIONS (dbserver '//10.162.75.236:1521/NEWBISTEST');
GRANT USAGE ON FOREIGN SERVER newbisdev TO postgres;
CREATE USER MAPPING FOR postgres SERVER newbisdev OPTIONS (user 'EMATRIX', password 'xxxxxxx');
CREATE FOREIGN TABLE ft_clog_sizes_2 ( 
  id     integer NOT NULL,
  myclob text ) 
SERVER newbisdev 
OPTIONS (schema 'EMATRIX', table 'CLOB_SIZES_2');

 select length(myclob) from ft_clog_sizes_2 where id=1;
 length
--------
      0
(1 row)

=> it does not work anymore

laurenz commented 7 months ago

Right, I can confirm that. I forgot that Oracle 21 client is broken; see my comment above.

Can you try with some other Oracle client version?

sabineWalter commented 7 months ago

that mean

installing 19c client instead of 21c ?

and/or must I recompile it other client version ?

laurenz commented 7 months ago

Here is a binary built with Oracle Client 19.

sabineWalter commented 7 months ago

replace oracle_client 21 by oracle_client 19 and it works

# select count(*) from ft_clog_sizes_2;
 count
-------
  1000
(1 row)

Time: 244.464 ms

# select count(*),min(length(myclob)), max(length(myclob)) from ft_clog_sizes_2;
 count | min  |   max
-------+------+---------
  1000 | 1024 | 1024000
(1 row)

Time: 84532.621 ms (01:24.533)

question if I replace the oracle client 19 by oracle client 23

does it works fine too? (from oracle client side?)
need I a new compiled binary ?

laurenz commented 7 months ago

Very nice. Yes, you can use that binary with Oracle 23. According to Oracle, you can use an executable built with Oracle Client X with Oracle Client Y, as long as Y >= X. I don't know if there are exceptions, but that's the general rule.

sabineWalter commented 7 months ago

so I will test the binary with installed oracle 23 client (after oracle 19 client deinstallation)

for the test cases
for the original table we run into this issue

coming back with the results

sabineWalter commented 7 months ago

oracle client 23 : I did not found to download => tried with 21.13

but get the following error:

# select count(*) from ft_clog_sizes_2;
ERROR:  could not load library "/usr/pgsql-14/lib/oracle_fdw.so": libclntsh.so.19.1: cannot open shared object file: No such file or directory

when execute

DROP SERVER newbisdev CASCADE;
CREATE SERVER newbisdev FOREIGN DATA WRAPPER oracle_fdw  OPTIONS (dbserver '//10.162.75.236:1521/NEWBISTEST');

hitting the same message:

ERROR: could not load library "/usr/pgsql-14/lib/oracle_fdw.so": libclntsh.so.19.1: cannot open shared object file: No such file or directory

ls -l /usr/pgsql-14/lib/oracle_fdw.so
-rwxr-xr-x 1 root root 748048 Feb  9 09:18 /usr/pgsql-14/lib/oracle_fdw.so

sabineWalter commented 7 months ago

final testcase with oracle client 19 passed

# select count(*) ANZ,min(length(XML)) MINXML,max(length(xml)) MAXXML from ft_dws_transfer;
 anz | minxml | maxxml
-----+--------+---------
 521 |    149 | 3956305
(1 row)

while

select count(*) ANZ,min(length(XML)) MINXML,max(length(xml)) MAXXML from ematrix.DWS_TRANSFER;

       ANZ     MINXML     MAXXML
---------- ---------- ----------
       521        149    3956305

sabineWalter commented 7 months ago

please explain the problem / issue with oracle client 21 to me in more details?

I have to install on this server oracle 21.12 or above to install ora2pg too (for some internal requests ....)

sabineWalter commented 7 months ago

sorry - I do not want to close this issue

laurenz commented 7 months ago

I don't know what exactly the problem with Oracle Client 21 is, but the symptoms are as follows:

different from all other client versions I tested (18, 19, and 23), my test program crashed with it
different from all other client versions I tested, my fix does not work with it

@philflorent provided some helpful information above, where he described version 21 as an "innovation release", which to me sounds like a beta test program where you have to pay to test.

Now I am open to the possibility that my code still contains a bug that is at fault, and that the other Oracle versions only work by accident for me, but I think that the suspicion that there is a bug in Oracle 21 is founded. If you have access to their bug database, you could search if they have any bugs about reading CLOBs with OCILobRead2().

Yes, Oracle 23 is only available as "Free" release so far (well, it's only 2024, so why hurry). I looked at the page where they advertise their new features, and it's things like a boolean data type and CREATE/DROP IF [NOT] EXISTS :^)

philflorent commented 7 months ago

Hi,

Lob_prefetch 32768 prefetch 50 clob_sizes_2 table (DDL is a bit modified for 11.2 since identity columns are not available but same data)

Commit 6f1f7c9 / Instant Client 19.22 / Database 11.2.0.4 => OK Commit 6f1f7c9 / Instant Client 19.22 / Database 19.13 => OK Commit 6f1f7c9 / Instant Client 19.22 / Database 23.2.0 => OK

Commit 6f1f7c9 / Instant Client 21.13 / Database 11.2.0.4 => KO (unsupported combo) Commit 6f1f7c9 / Instant Client 21.13 / Database 19.13 => KO (supported , should work but KO) Commit 6f1f7c9 / Instant Client 21.13 / Database 23.2.0 => KO (supported , should work but KO)

Commit d62b876 / Instant Client 19.22 / Database 11.2.0.4 => OK Commit d62b876/ Instant Client 19.22 / Database 19.13 => OK Commit d62b876 / Instant Client 19.22 / Database 23.2.0 => OK

Commit d62b876 / Instant Client 21.13 / Database 11.2.0.4 => KO (unsupported combo) Commit d62b876 / Instant Client 21.13 / Database 19.13 => KO (supported , should work but KO) Commit d62b876 / Instant Client 21.13 / Database 23.2.0 => KO (supported , should work but KO)

Same Debian 12 VM for pg , 3 different RHEL VM for Oracle DB (6 for 11.2, 7 for 19c, 8 for 23c)

I was not able to reproduce the issue with a LTR combo. Issue with the 21c client is the 0 length one.

"I have to install on this server oracle 21.12 or above to install ora2pg too"

The Dalibo tool ora2pg does not require 21c client to work. 21c is an innovation release not to be used in production. 19c client is compatible with versions 11.2.0.3<=>23c.

Best regards, Phil

laurenz / oracle_fdw

oracle_fdw cannnot load clobs larger than 21kbytes #643

include