search

oracle / python-cx_Oracle

Python interface to Oracle Database now superseded by python-oracledb
https://oracle.github.io/python-cx_Oracle
Other
888 stars 361 forks source link

ORA-03124 Two-task internal error running Django migrations #306

Closed danizen closed 5 years ago

danizen commented 5 years ago

1. What is your version of Python? Is it 32-bit or 64-bit?

Python 3.6.3 (64-bit) on CentOS 6.10. This is from CentOS SCL repository:

$ rpm -q rh-python36-python 
rh-python36-python-3.6.3-3.el6.x86_64

2. What is your cx_Oracle version?

cx-Oracle 6.4.1 - we have reproduced with both the manylinux1 build and a build created as follows:

pip uninstall -y cx-Oracle
pip install --no-cache --index https://pypi.org/simple --no-binary cx-Oracle cx-Oracle==6.4.1

We found that this no longer occurs with home-built versions of cx-Oracle 5.2.1 and cx-Oracle 5.3.

3. What exact command caused the problem (e.g. what command did you try to install with)?

./manage.py migrate

Who were you logged in as?

davisda4, but this is also occurring in our CI/CD environment.

4. What error(s) you are seeing?

During Django migrations, we see a traceback from django.db.utils.DatabaseError: ORA-03124: two-task internal error. This Django exception class is a wrapper for multiple database vendor specific errors. We are assuming the Oracle error number is of sufficient detail. See the attached file for full log information:

fulldebug.log

The pid/tid come from %(process)d and %(thread)d in the python logging formatter. Analyzing django/db/backends/utils.py confirms that the last SQL statement in this full log was the one on which the problem occurred. Note that there are no migrations, and so this is stock Django 1.11.20.

5. What OS (and version) is Python executing on?

CentOS 6.10

6. What is your version of the Oracle client (e.g. Instant Client)? How was it installed? Where is it installed?

12.1.0.1

Note that Django 1.11.20 was used. I cannot easily update to cx_Oracle 7+ without also updating to Django 2.2, which my organization is not yet ready to do.

I am not sure how it was installed, but the instantclient rpms are not installed /usr/nlm/share/oracle/12.1.0.1 - this is a read-only NFS mount

7. What is your Oracle Database version?

Oracle 18c

8. What is the PATH environment variable (on Windows) or LD_LIBRARY_PATH (on Linux) set to? On macOS, what is in ~/lib?

$ echo $LD_LIBRARY_PATH
/opt/rh/rh-python36/root/usr/lib64:/usr/nlm/share/oracle/12.1.0.1/lib

9. What Oracle environment variables did you set? How exactly did you set them?

This is an expert from my .bashrc:

ORACLE_HOME=/usr/nlm/share/oracle/12.1.0.1
TNS_ADMIN=/usr/nlm/share/etc/tns_admin
NLS_LANG="AMERICAN_AMERICA.UTF8"

# Put oracle's pelr libraires into PERL5LIB
if [ -z "$ORAPERL" ]; then
    ORAPERL=/usr/nlm/share/oracle/12.1.0.1/perl
    PERL5LIB="$ORAPERL/lib/5.14.1:$ORAPERL/lib/site_perl/5.14.1";
    export PERL5LIB
fi
export ORACLE_SID
export ORACLE_HOME
export ORACLE_PERL
export PERL5LIB
export TNS_ADMIN
export NLS_LANG

if [[ -z "$LD_LIBRARY_PATH" ]]; then
    LD_LIBRARY_PATH=$ORACLE_HOME/lib
else
    [[ $LD_LIBRARY_PATH =~ $ORACLE_HOME ]] || LD_LIBRARY_PATH="$ORACLE_HOME/lib:$LD_LIBRARY_PATH"
fi
export LD_LIBRARY_PATH

10. Do you have a small, single Python script that immediately runs to show us the problem?

Not as yet.

11. Additional Notes

We also submitted through our vendor support channel as SR 3-20049808601, with title "Client connection failing on with ORA-03124 Two-task internal error".

cjbj commented 5 years ago

@danizen this will require a parallel approach to solve since you are working with multiple open source projects.

The ORA error number necessarily is too broad to identity a specific cause. The trace file is for Django, and doesn't mean a lot to us. In particular it's not clear to me whether those SQL statements are executed when they are printed, and what the column and bind types are.

Even if Oracle Support won't handle (the open source) cx_Oracle, and certainly don't look after the 3rd-party Django, they should be able to help you create Oracle client, server and Oracle Net trace files that may help identify the failing statement, or sequence of statements. I see a comment in the SR about what you've tried; work with them on the best way to handle this. See MOS Doc ID 203967.1 for client side tracing. The traces might identify the base issue.

I would also turn on DPI tracing by setting the environment variable DPI_DEBUG_LEVEL to 16. and seeing if this also identifies the failing statement.

If you can give us a pure cx_Oracle test case, then we (cx_Oracle dev) can take it to the relevant Oracle Development groups.

There was a big refactor in cx_Oracle 6, which would explains the behavior compared with 5.

Could you update the original comment answers and mention the Instant Client versions you've tried - I see the SR has more info. BTW, OCI is always 'thick'.

danizen commented 5 years ago

@cjbj, my DBAs said that Oracle's advice was that using the same connection in multiple processes was a bad idea. I played "OPTIONS": { 'threaded' True }. Later, I turned up the logging to verify there was only on pid/tid involved.

BTW, OCI is always 'thick'

That is odd. I assumed thin, because when I use PyCharm IDE from Home to a database server over VPN, choosing "Thick" doesn't work. But my python test cases do... I just assumed that only thin was used by cx_Oracle. I think instead that only thick is used by cx_Oracle.

danizen commented 5 years ago

@cjbj, Here is the DPI tracing file, which I will also pass on to Oracle support:

dpi-tracing.txt

anthony-tuininga commented 5 years ago

Please redo the logging, but with DPI_DEBUG_LEVEL set to 24. The value of 8 will highlight any errors and where those errors took place. It is possible that the error is taking place before the execution takes place, so the offending SQL is not logged.

danizen commented 5 years ago

@anthony-tuininga , attached is a SQL*Net client trace of the error with tracing level 16. Running with DPI_DEBUG_LEVEL set to 24 did not emit any additional output; possibly because we are still using cx_Oracle 6.4.1. I am working to update some internal software which needs to be updated so we can update to Django 2.2 (generally), so that we can try again with the latest cx_Oracle.

Attached below is a SQL*Net client trace with tracing level 16: sqlnettrace.zip

cjbj commented 5 years ago

@danizen please work with Oracle Support on analyzing the SQL*Net trace. And also get a client trace, which is different. This is their area of expertise.

danizen commented 5 years ago

After upgrading some other libraries, I was finally able to upgrade to cx_Oracle 7.1.3, and can confirm that this problem does occur with the latest cx_Oracle. I still need to try the latest Oracle instantclient.

cjbj commented 5 years ago

@danizen thanks for the update. We still are relying on you and Support to identify any failing SQL, or other, statements.

danizen commented 5 years ago

@cjbj, I am uncertain how to proceed. In the worst case, you guys will need to educate me on how to run python and cx_Oracle and some python code through gdb.

I have the following thoughts:

danizen commented 5 years ago

Let me at least be clearer about where the problem is occurring. I see the root exception being thrown from line 497 of the file django/db/backends/oracle/base.py in Django 1.11.20, running the following query:

SELECT
  "DJANGO_CONTENT_TYPE"."ID",
  "DJANGO_CONTENT_TYPE"."APP_LABEL",
  "DJANGO_CONTENT_TYPE"."MODEL"
FROM "DJANGO_CONTENT_TYPE" 
WHERE "DJANGO_CONTENT_TYPE"."APP_LABEL" = :arg0

The parameters passed look like this from the python side:

{':arg0': 'auth'}

I also see that I was incorrect about the environment variables at the time of the error, because Django overrides some environment variables, and the Python debugger reports the following:

(Pdb) os.environ['NLS_LANG']
'.AL32UTF8'
(Pdb) os.environ['ORA_NCHAR_LITERAL_REPLACE']
'TRUE'
cjbj commented 5 years ago

I would start with your third point "Code the same SQL statements in a plain python file with just cx_Oracle". To get to that point you may need to do miscellaneous tracing (Oracle client, ODPI-C).

danizen commented 5 years ago

@cjbj , I will try that today. We have that dpi-tracing.txt file, which includes the generated error:

ODPI [14474] 2019-05-20 13:08:40.756: OCI error ORA-03124: two-task internal error (dpiStmt_execute / execute)
danizen commented 5 years ago

Here is a more complete DPI tracing log with DPI_DEBUG_LEVEL=63:

dpi-tracing-63.txt

I've also updated the file django/db/backends/oracle/base.py to log an error with the statement, params, and exception information when the issue occurs, and that output is also in the file.

This more complete tracing should allow me to mirror what Django is doing with the cursor after executing it and see if I can reproduce this with a single file python script.

danizen commented 5 years ago

Looking at src/dpiStmt.c, I do not see where dpiError_t error is initialized to 0.

In ISO/IEC 9899:TC3 WG14/N1256 (C99 standard) section 6.7.8 clause 10:

If an object that has automatic storage duration is not initialized explicitly, its value is indeterminate.

This might be a difference between Solaris and Linux in practice. Most of the time, this would not be an issue, because you check the return code and then return, but there may be a vulnerability somewhere here - maybe a place where you need to check for errno == EINTR and continue to wait on a socket or something.

I'm not so far grasping at straws that I'm setting it to zero and compiling it; it is just a thought.

anthony-tuininga commented 5 years ago

Can you be more precise about the location in src/dpiStmt.c? The error structure is initialised at the start of every public function.

danizen commented 5 years ago

@anthony-tuininga - line 1173 of odpi/src/dpiStmt.c, looking at the tarball of cx_Oracle 6.4.1. One of our custom libraries has a dependency on Django 1.11 for the moment, but I worked around that and did verify that this still occurs with cx_Oracle 7.1.3 - so, I'm only now working with an older version for convenience.

I cannot say that uninitialized memory feels right, however. All I know is that Oracle support tells me that there is no error in my client/server trace. So, my DBA thinks that the error is coming from the cx_Oracle library.

The structure of your code looks good to me so far though - I'm really not doing C development at all any longer, until this issue :)

I need to figure out how the numerical value of ORA-03124 corresponds to anything DPI would return, knowing very little of either.

danizen commented 5 years ago

@anthony-tuininga - the other question is how any specific error is returned by cx_Oracle when there is a DPI error. It seems like the DPI error in this case is neither logged nor raised up. So, the two-task internal error must be generated by cx_Oracle somewhere - where is the question.

danizen commented 5 years ago

Next step might be to instrument the cx_Oracle code to generate a backtrace when this happens using backtrace library. To do this, I have to figure out how to build cx_Oracle without stripping symbols, and to disable optimization.

Can you guys help with the latter?

danizen commented 5 years ago

Can you guys help with the latter?

No need. It is compiled with debugging symbols and optimizations. I now know I can capture the error while doing stuff like this, but I have not translated these into symbols as yet:

[2019-05-29 14:42:15,003] INFO nlm.occs.dbp retrieved password for mplusmon_dev@medplus_dev
Operations to perform:
  Apply all migrations: auth, casauth, contenttypes, django_cas_ng, mplusmon, sessions
Running migrations:
  No migrations to apply.
DPI error at 1192, with following 20 stack frames:
    0x7f2166c703dd
    0x7f2166c50130
    0x7f2174b687fc
    0x7f2174bdd27e
    0x7f2174bd1e9b
    0x7f2174bdc1b4
    0x7f2174bdd4f1
    0x7f2174bd1e9b
    0x7f2174bdc1b4
    0x7f2174bdd4f1
    0x7f2174bd1e9b
    0x7f2174bdc1b4
    0x7f2174bdd4f1
    0x7f2174bd2fef
    0x7f2174b41274
    0x7f2174b4cc1c
    0x7f2174b4d05e
    0x7f2174b7d46c
    0x7f2174b26c0d
    0x7f2174bdd45a
danizen commented 5 years ago

This apparently is translated to:

/home/davisda4/python-envs/mplusmon/lib/python3.6/site-packages/cx_Oracle.cpython-36m-x86_64-linux-gnu.so(dpiStmt_execute+0xdf)[0x7f88d4c183ff]
/home/davisda4/python-envs/mplusmon/lib/python3.6/site-packages/cx_Oracle.cpython-36m-x86_64-linux-gnu.so(+0x1a1d0)[0x7f88d4bf81d0]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(_PyCFunction_FastCallDict+0x22c)[0x7f88e2b107fc]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0x15327e)[0x7f88e2b8527e]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(_PyEval_EvalFrameDefault+0x6fb)[0x7f88e2b79e9b]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0x1521b4)[0x7f88e2b841b4]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0x1534f1)[0x7f88e2b854f1]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(_PyEval_EvalFrameDefault+0x6fb)[0x7f88e2b79e9b]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0x1521b4)[0x7f88e2b841b4]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0x1534f1)[0x7f88e2b854f1]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(_PyEval_EvalFrameDefault+0x6fb)[0x7f88e2b79e9b]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0x1521b4)[0x7f88e2b841b4]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0x1534f1)[0x7f88e2b854f1]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(_PyEval_EvalFrameDefault+0x184f)[0x7f88e2b7afef]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0xb7274)[0x7f88e2ae9274]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0xc2c1c)[0x7f88e2af4c1c]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0xc305e)[0x7f88e2af505e]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0xf346c)[0x7f88e2b2546c]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(_PyObject_FastCallDict+0xcd)[0x7f88e2acec0d]
/opt/rh/rh-python36/root/usr/lib64/libpython3.6m.so.rh-python36-1.0(+0x15345a)[0x7f88e2b8545a]
danizen commented 5 years ago

So that was a good effort, but lacking the symbol just above dpiStmt_execute isn't helpful.

danizen commented 5 years ago

It is no doubt the cxoCursor__execute(), because that's where it was in the python code.

danizen commented 5 years ago

So, tracing the other way with fprintf, I've verified that this error is coming from dpiOci__stmtExecute, which means it is past where it makes much sense for me to go farther in looking for a simple error. I've also verified the problem still occurs even if I add a pedantic memset - so that's not the problem.

I will attempt now to reproduce with a simple python program again.

anthony-tuininga commented 5 years ago

@anthony-tuininga - line 1173 of odpi/src/dpiStmt.c, looking at the tarball of cx_Oracle 6.4.1. One of our custom libraries has a dependency on Django 1.11 for the moment, but I worked around that and did verify that this still occurs with cx_Oracle 7.1.3 - so, I'm only now working with an older version for convenience.

So dpiStmt__checkOpen() calls dpiGen__startPublicFn() which is what initialises the error structure.

@anthony-tuininga - the other question is how any specific error is returned by cx_Oracle when there is a DPI error. It seems like the DPI error in this case is neither logged nor raised up. So, the two-task internal error must be generated by cx_Oracle somewhere - where is the question.

What makes you think the error is neither logged nor raised up? I see it in your ODPI-C tracing file. It also makes it clear in that file that the error occurred with dpiStmt_execute() and more specifically the actual execution (OCIStmtExecute()), as you discovered.

All I know is that Oracle support tells me that there is no error in my client/server trace. So, my DBA thinks that the error is coming from the cx_Oracle library.

I can see that there wouldn't necessarily be an error on the server side but there definitely should be on the client side of things! That error is not normal by any means. Make sure you're getting the right trace file.

danizen commented 5 years ago

@anthony-tuininga , thanks for your attention to this. Good points. I see the flow of the code now - after the return from dpiStmt_execute, then cxoCursor_execute will call cxoError_raiseAndReturnNull which will request the latest error information.

I've also reproduced this now with a simple Python script, without any Django or any other add-on packages except cx_Oracle imported. So, I'm getting closer to a test case. I will now attempt to reproduce in another schema.

danizen commented 5 years ago

I've received this from Oracle support:

As per the client/server trace files we have not find any issue with client and database .

We need to check from the application prospective...

We have reviewed both client/server trace files .

There is no ORA-03124 errors reported in the trace files .

Just user canceled the operation from the client

(3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 20 FE 4B 23 B6 7F |....K#..| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 00 00 00 00 00 00 |........| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 00 00 00 00 00 00 |........| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 00 00 00 00 00 00 |........| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 00 00 00 00 00 00 |........| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 00 00 00 00 00 00 |........| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 00 00 00 00 00 00 |........| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 00 00 00 00 F5 03 |........| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 02 00 00 00 00 00 |........| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 00 00 36 4F 52 41 2D 30 |..6ORA-0| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 31 30 31 33 3A 20 75 73 |1013:.us| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 65 72 20 72 65 71 75 65 |er.reque| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 73 74 65 64 20 63 61 6E |sted.can| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 63 65 6C 20 6F 66 20 63 |cel.of.c|========================>user canceled the operation (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 75 72 72 65 6E 74 20 6F |urrent.o| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 70 65 72 61 74 69 6F 6E |peration| (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: 0A |. | (3497322240) [29-MAY-2019 12:59:46:778] nsbasic_brc: exit: oln=0, dln=199, tot=209, rc=0 (3497322240) [29-MAY-2019 12:59:46:778] nioqrc: exit

Directly before the cancellation they mention, I see the following:

(3497322240) [29-MAY-2019 12:59:46:777] nserror: entry (3497322240) [29-MAY-2019 12:59:46:777] nserror: nsres: id=0, op=77, ns=12630, ns2=0; nt[0]=0, nt[1]=0, nt[2]=0; ora[0]=0, ora[1]=0, ora[2]=0 (3497322240) [29-MAY-2019 12:59:46:777] nioqer: entry (3497322240) [29-MAY-2019 12:59:46:777] nioqer: incoming err = 0 (3497322240) [29-MAY-2019 12:59:46:777] nioqce: entry (3497322240) [29-MAY-2019 12:59:46:777] nioqce: exit (3497322240) [29-MAY-2019 12:59:46:777] nioqer: returning err = 0 (3497322240) [29-MAY-2019 12:59:46:777] nioqer: exit (3497322240) [29-MAY-2019 12:59:46:777] nioqrs: exit (3497322240) [29-MAY-2019 12:59:46:777] nioqrc: entry (3497322240) [29-MAY-2019 12:59:46:777] nsbasic_brc: entry: oln/tot=0 (3497322240) [29-MAY-2019 12:59:46:778] nttfprd: entry (3497322240) [29-MAY-2019 12:59:46:778] nttfprd: socket 11 had bytes read=209 (3497322240) [29-MAY-2019 12:59:46:778] nttfprd: exit

I see some retry logic in dpiStmt__execute in odpi/src/dpiStmt.c, and by printf diagnostic instrumentation I can see that the operation is not retried. Is there any way for you guys to see which condition would need to be checked to retry in this case.

danizen commented 5 years ago

So, Oracle support will not support cx_Oracle, and cloning the schema does not reproduce the error.:

We have no choice but to face the long dark of Moria.

I read through all of https://pythonextensionpatterns.readthedocs.io/ yesterday, and I have asked Oracle support to treat this as if we were writing our own OCI code. To that end, I have instrumented dpiError__check to see what this error looks like. If you guys can help me figure out what the error looks like above, and whether it should be recoverable, and point me towards the most accessible form of OCI documentation, I'd appreciate it.

danizen commented 5 years ago

My instrumentation shows the error code as 3124 at the C-level, and not recoverable.

anthony-tuininga commented 5 years ago

The retry logic in dpiStmt__execute() is only for specific errors and ORA-3124 is definitely not one of them. The only thing that error message tells you for an action is "contact your customer support representative". :-) I also wouldn't classify ORA-3124 as recoverable. The cancellation you mention may be occurring due to ORA-3124. The error may be related to this:

(3497322240) [29-MAY-2019 12:59:46:777] nserror: entry (3497322240) [29-MAY-2019 12:59:46:777] nserror: nsres: id=0, op=77, ns=12630, ns2=0; nt[0]=0, nt[1]=0, nt[2]=0; ora[0]=0, ora[1]=0, ora[2]=0

Note that there is ns=12630 which may be an error number. That error number states:

12630, 00000, "Native service operation not supported" // Cause: An operation requested by a user is not supported by the native // services component. // Action: This may be an internal error if the operation should have been // supported.

Perhaps draw their attention to that and ask about that line?

danizen commented 5 years ago

Another way to do this is to attempt to duplicate this error simply on the level of ODPI. If I download and build ODPI-C and link with that, and reproduce the error, that might be easier.

I also have questions about how to build cx_Oracle against a Python compiled --with-pydebug and other python extension debugging flags.

What do you think of these options?

danizen commented 5 years ago

Another way to do this is to attempt to duplicate this error simply on the level of ODPI. If I download and build ODPI-C and link with that, and reproduce the error, that might be easier.

I can now reproduce the error linking with my own built version of ODPI, which allows me to add the diagnostics without changing the cx_Oracle source code. Using the dpi tracing, I could attempt to reproduce without the use of Python or cx_Oracle.

anthony-tuininga commented 5 years ago

Yes, you could do that. It will take some effort but you can do so. If you are unable to replicate with another schema, would it be possible to replace your "bad" schema with a "good" one? I'm not sure what the difference in time would be, but perhaps its something to consider?

danizen commented 5 years ago

Yes, you could do that. It will take some effort but you can do so. If you are unable to replicate with another schema, would it be possible to replace your "bad" schema with a "good" one? I'm not sure what the difference in time would be, but perhaps its something to consider?

This analagous thing has happened to 3 other projects here, and probably would have happened to more except I told everyone to stick with cx_Oracle==5.3 My project is ideal to stay "bad" because it was a prototype we never released. That happens to me a lot because I'm the "architect" who brought us from ColdFusion to Python/Django over the past 6 years. So, despite the fact that I can dive in too deep when the issue is a simple thing, when it is a complex thing (or simply a thing that requires someone who understands RPATH, LD_LIBRARY_PATH, and dlopen/dlsym and Python/Django), then it is my issue.

One of the projects where this happened built several schemas and reproduced the problem on its own, so I could also copy DDL statements and create a schema until I can make it happen.

anthony-tuininga commented 5 years ago

If you're able to find a way to reliably replicate the issue (meaning that I can replicate it, too) that will be very helpful. Otherwise, I'd still ask support about the ORA error I mentioned earlier.

cjbj commented 5 years ago

@danizen I had previously asked Support to treat it as an OCI issue. Feel free to mention my name when you talk to them.

cjbj commented 5 years ago

@danizen what's the status?

anthony-tuininga commented 5 years ago

I am assuming this question has been adequately answered due to lack of activity.

danizen commented 5 years ago

@danizen what's the status?

We never really got a satisfactory answer from Oracle support, but I am satisfied it is not a cx_Oracle problem. In our move to cloud, we are also move to a mix of PostgreSQL and MySQL, and so we are using cx_Oracle on premise. I am very careful to build cx_Oracle with 12.1.0.1, because of Compatibility matrix, and have buckled down and rebuilt it, even for Windows, with Python 3.6.

This means that we need to be careful about LD_LIBRARY_PATH, and on-premise, we can handle that.

It is unfortunate, but that's the way it has to be.

Thanks for all your help, guys.