Closed liudonghua123 closed 2 years ago
cjbj
commented
2 years ago init_oracle_client(lib_dir=...) instead of PATH? https://cx-oracle.readthedocs.io/en/latest/user_guide/initialization.htm
liudonghua123
commented
2 years ago
- Please clarify what you mean about running the pyinstaller exe. Did you run that on the same machine?
I tried to use pyinstaller to package the python app. Yes I run on the same machine.
- Have you tried using
init_oracle_client(lib_dir=...)instead of PATH? https://cx-oracle.readthedocs.io/en/latest/user_guide/initialization.htm
I only use PATH, If I use init_oracle_client(lib_dir=...), then no problem.
- Do you need to reboot or restart a terminal window to get PATH set?
I restarted the terminal window, and I also checked PATH and where oci.dll in my code.
sharadraju
commented
2 years ago @liudonghua123,
What I understand is: Your apps works if you use the init_oracle_client(lib_dir=...)function to initialize the Oracle client libraries, but not when you set the client library path on the PATH environment variable.
Is my understanding right?
If so, is there any reason you do not want to use the init_oracle_client function?
cjbj
commented
2 years ago I'm also interested to know why init_oracle_client() isn't being used.
D:\apps\oracle\instantclient_19_8?D:\apps\oracle\instantclient_19_8 first in PATHUnrelated to your problem: you should upgrade to the latest Release Update, which is 19.14
liudonghua123
commented
2 years ago What I understand is: Your apps works if you use the init_oracle_client(lib_dir=...) function to initialize the Oracle client libraries, but not when you set the client library path on the PATH environment variable. Is my understanding right? If so, is there any reason you do not want to use the init_oracle_client function?
@sharadraju Yes, you understanding is right, I want to deploy the app to other machine which configured oracle instantclient in their own location. So I did not want to use init_oracle_client manully.
@cjbj
Do you have any other oci.dll files on your machine, perhaps that are in your PATH and therefore get loaded before D:\apps\oracle\instantclient_19_8?
Yes, I have tried a few different version of oracle instant client. And I can confirm that I used the right architecture with my python.
Try putting D:\apps\oracle\instantclient_19_8 first in PATH
I don't think that is matters, From the output of the above logs. The PATH when I execute the app via python script.py or script.exe is the same. I wonder why script.exe worked and python script.py did not worked.
cjbj
commented
2 years ago Please try changing PATH as suggested. I can see the PATH look the same, but ....
liudonghua123
commented
2 years ago I have changed the PATH as you sugguested, and the results is the same.
I have also included the following test code.
def check_env():
print(f'os.environ["PATH"]: {os.environ["PATH"]}')
print(f'os.system("where oci.dll"): {os.system("where oci.dll")}')
check_env()However, from the code in https://github.com/oracle/odpi/blob/f73a7c13d643b3fe252614bafc930afbd8e287dd/src/dpiOci.c#L1733-L1755, the order in the PATH do not matters.
cjbj
commented
2 years ago The order in PATH would matter if you have other oci.dll's on your machine. Did you check how many are installed.
Keep experimenting. Your problem doesn't reproduce for us!
liudonghua123
commented
2 years ago I tested with the query.py in https://cx-oracle.readthedocs.io/en/latest/user_guide/introduction.html#getting-started. And it was the same error info. I have one and only one oci.dll which is D:\apps\oracle\instantclient_19_8 in PATH, you could see the output of where oci.dll in the logs.
The problem is why exe worked while python script.py did not! It is really strange. I read the code in odpi, but could not find some clues until now. Maybe I can add some print code to see what's happened into odpi.
D:\code\python\ecard-app>cat query.py
# query.py
import os
def check_env():
print(f'os.environ["PATH"]: {os.environ["PATH"]}')
print(f'os.system("where oci.dll"): {os.system("where oci.dll")}')
check_env()
import cx_Oracle
# Establish the database connection
connection = cx_Oracle.connect(user="hr", password="userpwd",
dsn="dbhost.example.com/orclpdb1")
# Obtain a cursor
cursor = connection.cursor()
# Data for binding
manager_id = 145
first_name = "Peter"
# Execute the query
sql = """SELECT first_name, last_name
FROM employees
WHERE manager_id = :mid AND first_name = :fn"""
cursor.execute(sql, mid=manager_id, fn=first_name)
# Loop over the result set
for row in cursor:
print(row)
D:\code\python\ecard-app>I added some debugging info in https://github.com/oracle/odpi/blob/f73a7c13d643b3fe252614bafc930afbd8e287dd/src/dpiOci.c#L1768-L1788.
static int dpiOci__loadLibWithName(dpiOciLoadLibParams *loadParams,
const char *name, dpiError *error)
{
dpiDebug__print("dpiOci__loadLibWithName\n");
DWORD errorNum;
dpiDebug__print("dpiOci__loadLibWithName->name %s\n", name);
// attempt to load the library
loadParams->handle = LoadLibrary(name);
if (loadParams->handle) {
dpiDebug__print("dpiOci__loadLibWithName->LoadLibrary DPI_SUCCESS\n");
return DPI_SUCCESS;
}
dpiDebug__print("dpiOci__loadLibWithName->LoadLibrary DPI_FAILURE\n");
// if DLL is of the wrong architecture, attempt to locate the DLL that was
// loaded and use that information if it can be found
errorNum = GetLastError();
dpiDebug__print("dpiOci__loadLibWithName->errorNum == ERROR_BAD_EXE_FORMAT: %d\n", errorNum == ERROR_BAD_EXE_FORMAT);
if (errorNum == ERROR_BAD_EXE_FORMAT &&
dpiOci__findAndCheckDllArchitecture(loadParams, name, error) == 0)
return DPI_SUCCESS;
// otherwise, attempt to get the error message
return dpiUtils__getWindowsError(errorNum, &loadParams->errorBuffer,
&loadParams->errorBufferLength, error);
}And the console logs is the following.
D:\code\python\ecard-app>python query.py
......
ODPI [01012] 2022-02-25 14:19:26.676: load with OS search heuristics 123
ODPI [01012] 2022-02-25 14:19:26.676: load with name oci.dll
ODPI [01012] 2022-02-25 14:19:26.676: dpiOci__loadLibWithName
ODPI [01012] 2022-02-25 14:19:26.676: dpiOci__loadLibWithName->name oci.dll
ODPI [01012] 2022-02-25 14:19:26.676: dpiOci__loadLibWithName->LoadLibrary DPI_FAILURE
ODPI [01012] 2022-02-25 14:19:26.676: dpiOci__loadLibWithName->errorNum == ERROR_BAD_EXE_FORMAT: 0
ODPI [01012] 2022-02-25 14:19:26.676: load by OS failure: The specified module could not be found
......D:\code\python\ecard-app>dist\query\query.exe
......
ODPI [28464] 2022-02-25 14:23:09.967: load with OS search heuristics 123
ODPI [28464] 2022-02-25 14:23:09.969: load with name oci.dll
ODPI [28464] 2022-02-25 14:23:09.971: dpiOci__loadLibWithName
ODPI [28464] 2022-02-25 14:23:09.973: dpiOci__loadLibWithName->name oci.dll
ODPI [28464] 2022-02-25 14:23:09.977: dpiOci__loadLibWithName->LoadLibrary DPI_SUCCESS
ODPI [28464] 2022-02-25 14:23:09.980: load by OS successful
ODPI [28464] 2022-02-25 14:23:09.981: validating loaded library
......So the root issue is on loadParams->handle = LoadLibrary(name);! It worked in exe but not python script.py.
liudonghua123
commented
2 years ago So I just tested LoadLibrary as the following code (mentioned in https://stackoverflow.com/questions/13265046/could-not-load-c-dll-in-python/13268880). This behavior is different in exe and python script.py!
from ctypes import cdll
cdll.LoadLibrary('oci.dll')I also tested the follow code.
import traceback
import snoop
@snoop
def test_ctypes_loadlibrary():
from ctypes import cdll
print("ctypes.cdll.LoadLibrary")
try:
oci=cdll.LoadLibrary('oci.dll')
print("ctypes.cdll.LoadLibrary", oci)
except Exception as e:
traceback.print_exc()
@snoop
def test_pywin32_loadlibrary():
import win32api
print("pywin32 win32api.LoadLibrary")
try:
oci=win32api.LoadLibrary('oci.dll')
print("pywin32 win32api.LoadLibrary", oci)
except Exception as e:
traceback.print_exc()
test_ctypes_loadlibrary()
test_pywin32_loadlibrary()The oci.dll could not load in python script.
cjbj
commented
2 years ago As a non Windows user, it's not clear to me whether win32 was/is only 32 bit.
Do you have 32-bit or 64-bit Oracle Instant Client.
liudonghua123
commented
2 years ago @cjbj Hi, I have checked my envs again. Python, oracle instant client, the exe packaged via pyinstaller, all of them are 64-bit.
Microsoft Windows [Version 10.0.22000.527]
(c) Microsoft Corporation. All rights reserved.
Clink v1.2.42.d4506e
Copyright (c) 2012-2018 Martin Ridgers
Portions Copyright (c) 2020-2021 Christopher Antos
https://github.com/chrisant996/clink
C:\Users\Liu.D.H>where oci.dll
D:\apps\oracle\instantclient_19_8\oci.dll
C:\Users\Liu.D.H>linux_tools
C:\Users\Liu.D.H>file D:\apps\oracle\instantclient_19_8\oci.dll
D:\apps\oracle\instantclient_19_8\oci.dll: PE32+ executable (DLL) (GUI) x86-64, for MS Windows
C:\Users\Liu.D.H>d:
D:\>cd code\python\ecard-app\
D:\code\python\ecard-app>where python
C:\Users\Liu.D.H\AppData\Local\Microsoft\WindowsApps\python.exe
C:\Users\Liu.D.H\anaconda3\python.exe
D:\code\python\ecard-app>file C:\Users\Liu.D.H\AppData\Local\Microsoft\WindowsApps\python.exe
C:\Users\Liu.D.H\AppData\Local\Microsoft\WindowsApps\python.exe: symbolic link to /c/Program Files/WindowsApps/PythonSoftwareFoundation.Python.3.10_3.10.752.0_x64__qbz5n2kfra8p0/python3.10.exe
D:\code\python\ecard-app>file "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.752.0_x64__qbz5n2kfra8p0\python3.10.exe"
C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.752.0_x64__qbz5n2kfra8p0\python3.10.exe: PE32+ executable (console) x86-64, for MS Windows
D:\code\python\ecard-app>file dist\query\query.exe
dist\query\query.exe: PE32+ executable (console) x86-64, for MS Windows
D:\code\python\ecard-app>And the dependences of oci.dll seems ok according to the comments mentioned in https://stackoverflow.com/questions/11717312/loadlibrary-project-dll-failed-the-specified-module-could-not-be-found.
The problem I interested is why python script.py not works while packaged_via_pyinstaller.exe works.
And I do not think it was a bad architecture error, because if this is the case, then loadParams->handle = LoadLibrary(name); would got ERROR_BAD_EXE_FORMAT and then dpiOci__findAndCheckDllArchitecture would invoked.
If I use python 3.10 32-bit to run the above load_oci.py test. I will get %1 is not a valid Win32 application error instead of The specified module could not be found.
D:\code\python\ecard-app>C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310-32\python.exe load_oci.py
09:46:17.78 >>> Call to test_ctypes_loadlibrary in File "D:\code\python\ecard-app\load_oci.py", line 5
09:46:17.78 5 | def test_ctypes_loadlibrary():
09:46:17.78 6 | from ctypes import cdll
09:46:17.78 .......... cdll = <ctypes.LibraryLoader object at 0x01DC5E98>
09:46:17.78 7 | print("ctypes.cdll.LoadLibrary")
ctypes.cdll.LoadLibrary
09:46:17.78 8 | try:
09:46:17.78 9 | oci=cdll.LoadLibrary('oci.dll')
09:46:17.80 !!! FileNotFoundError: Could not find module 'oci.dll' (or one of its dependencies). Try using the full path with constructor syntax.
09:46:17.80 !!! When calling: cdll.LoadLibrary('oci.dll')
09:46:17.80 11 | except Exception as e:
09:46:17.80 .......... e = FileNotFoundError("Could not find module 'oci.dl...ry using the full path with constructor syntax.")
09:46:17.80 12 | traceback.print_exc()
Traceback (most recent call last):
File "D:\code\python\ecard-app\load_oci.py", line 9, in test_ctypes_loadlibrary
oci=cdll.LoadLibrary('oci.dll')
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310-32\lib\ctypes\__init__.py", line 452, in LoadLibrary
return self._dlltype(name)
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310-32\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'oci.dll' (or one of its dependencies). Try using the full path with constructor syntax.
09:46:17.80 <<< Return value from test_ctypes_loadlibrary: None
09:46:17.80 >>> Call to test_pywin32_loadlibrary in File "D:\code\python\ecard-app\load_oci.py", line 15
09:46:17.80 15 | def test_pywin32_loadlibrary():
09:46:17.80 16 | import win32api
09:46:17.81 .......... win32api = <module 'win32api' from 'C:\\Users\\Liu.D.H\\AppData\\Local\\Programs\\Python\\Python310-32\\lib\\site-packages\\win32\\win32api.cp310-win32.pyd'>
09:46:17.81 17 | print("pywin32 win32api.LoadLibrary")
pywin32 win32api.LoadLibrary
09:46:17.81 18 | try:
09:46:17.81 19 | oci=win32api.LoadLibrary('oci.dll')
09:46:17.81 !!! pywintypes.error: (193, 'LoadLibrary', '%1 is not a valid Win32 application.')
09:46:17.81 !!! When calling: win32api.LoadLibrary('oci.dll')
09:46:17.81 21 | except Exception as e:
09:46:17.81 .......... e = error(193, 'LoadLibrary', '%1 is not a valid Win32 application.')
09:46:17.81 22 | traceback.print_exc()
Traceback (most recent call last):
File "D:\code\python\ecard-app\load_oci.py", line 19, in test_pywin32_loadlibrary
oci=win32api.LoadLibrary('oci.dll')
pywintypes.error: (193, 'LoadLibrary', '%1 is not a valid Win32 application.')
09:46:17.81 <<< Return value from test_pywin32_loadlibrary: None
D:\code\python\ecard-app>How are you going with this?
The fact that init_oracle_client() works for you, indicates to me there is something in your path or environment that is causing this.
liudonghua123
commented
2 years ago I know I can use init_oracle_client to make it work, however, I just want to find why python script not work while pyinstaller_packaged_exe work using the PATH method in my current environment. It's really strange.
liudonghua123
commented
2 years ago I tried the same code and tests on another computer without this error, and I reboot my computer and tested again without this error either.
Closing this issue due to can not reproduce.
@cjbj @sharadraju Thanks for your warmful helps.
If I encountered the same errors. Maybe I can use the following code to test and find "why".
// test_loadlibrary.cpp
// compile using vs2019 x64 prompt tools. cl test_loadlibrary.cpp
// running test_loadlibrary.exe
#include <windows.h>
#include <iostream>
int main() {
HINSTANCE hOCIDLL = LoadLibrary("oci.dll");
if (hOCIDLL == NULL) {
std::cout << "Cannot LoadLibrary oci.dll" << std::endl;
return -1;
}
std::cout << "LoadLibrary oci.dll successfully" << std::endl;
return 0;
}@liudonghua123 thanks for sharing that code snippet. I'm glad the issue resolved itself.
cjbj
commented
2 years ago For anyone finding this issue via a search, try the latest cx_Oracle release (now called python-oracledb). It doesn't need Oracle Client libraries for most things. See https://oracle.github.io/python-oracledb/. Install with a command like python -m pip install oracledb
Give your database version. Oracle 12c
Also run Python and show the output of:
And:
Is it an error or a hang or a crash? crash
What error(s) or behavior you are seeing?
I use this package in my python app, and I have configured oracle instant client by adding
D:\apps\oracle\instantclient_19_8to the path environment. When I tried to connect to the remote oracle database, I gotCannot locate a 64-bit Oracle Client library: "The specified module could not be found"error. However If I packaged the app using pyinstaller and execute the exe, the it worked.The following is some logs of these steps.
I also add some code to check the
oci.dllin PATH. Both of them passed.