ibmdb / python-ibmdb-django

IBM DB2 Driver for the Django application Framework
Apache License 2.0
29 stars 42 forks source link

Performing system checks... System check identified no issues (0 silenced). Hangs.... #95

Closed florisjkruger4 closed 1 year ago

florisjkruger4 commented 1 year ago

Hello Everyone,

So I have been trying to connect to an already existing AS400 iSeries Database using a Django project. It is important to know that this database I am trying to connect to already has a lot of data on it... but I am not sure if that would explain why my command line "hangs" or freezes whenever I try to run the following commands:

python manage.py makemigrations
python manage.py migrate

However when I run python manage.py runserver, I get this response: `(env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\test\plzwrk>python manage.py runserver Watching for file changes with StatReloader Performing system checks...

System check identified no issues (0 silenced).`

Usually you would get another message after this stating that the development has started on a server at http://127.0.0.1:8000/ or something like that. But my terminal just stops at system checks and I am not able to access the web page.

I have followed all the steps listed in this github documentation and the following is what my current 'pip list' is within the virtual environment my Django Project is configured with: `(env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\test>pip list Package Version


asgiref 3.7.2 Django 3.2 ibm-db 3.1.4 ibm-db-django 1.5.2.0 pip 23.0.1 pytz 2023.3 regex 2023.6.3 setuptools 65.5.0 six 1.16.0 sqlparse 0.4.4 typing_extensions 4.7.1`

I am also using python version 3.10.11 (env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\test\plzwrk>python --version Python 3.10.11

Furthermore, I believe I have my settings.py database settings and installed apps settings configured correctly. `INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles',

'django.contrib.flatpages',
'django.contrib.redirects',
'django.contrib.sites',

]`

DATABASES = { 'default': { 'ENGINE' : 'ibm_db_django', 'NAME' : 'DB_name', 'USER' : 'USERNAME', 'PASSWORD' : 'PASSWORD', 'HOST' : 'IP_ADDRESS', 'PORT' : 'PORT', 'PCONNECT' : True, } } And I know all the values for these Database settings are in fact connect since I am able to connect to the AS400 system when using the PYODBC library which requires a connection string with the same data in order to access the AS400 system. connection_string = 'DRIVER={IBM i Access ODBC Driver};SYSTEM=_____;UID=_____;PWD=_____'

However, I would like to connect properly using the actual database settings since this is much more efficient vs using the PYODBC method.

I was also able to get all three drivers installed: Client Access ODBC Driver (32-bit) IBM i Access ODBC Driver iSeries Access ODBC Driver

The default driver ibm_db uses is the Client Access Driver. However, I am still faced with the same problem and not getting any feedback when running the commands to connect. Also, my trace facility is not writing anything to the files... This must mean that the ODBC driver is not really being used? I'm completely stumped...

imavo commented 1 year ago

You understand that the ibm_db_django will not use the pyodbc interface, but instead will use the DBI interface.

Which bitness of python are you using? If you use 64-bit python you should use a 64-bit CLI driver along with the 64bit version of odbcad32.exe gui on MS-Windows. If you use a 32-bit python you must use a 32-bit CLI driver along with the 32-bit version of odbcad32.exe gui.

ibm_db_django uses the DBI interface to connect to the database, which in turn uses the underlying ibm_db interface, which in turn uses a CLI driver. But which CLI (odbc) driver will it actually use, that depends on your configuration, your PATH , your choices before installing ibm_db_django, and your connection-string.

By default the chosen CLI (odbc) driver will be the tiny "clidriver" from IBM (which needs a licence file when connecting to as400), but you can arrange BEFORE installing ibm_db to specify to not install clidriver , which causes ibm_db later to try to use a pre-existing CLI driver as long as that pre-existing CLI driver is already fully configured, and already on the PATH, and already operating with the odbcad32.exe and already with a working DSN (data source name) corresponding to your database that matches your connection-string and PATH.

If you did not set environment variable IBM_DB_HOME to point to your working odbc driver BEFORE installing ibm_db_django (or ibm_db) then perform a full uninstall of ibm_db_django and ibm_db (with pip uninstall ibm_db_django , pip uninstall ibm_db) , set the IBM_DB_HOME variable correctly to point to the executables directory containing your working and configured odbc driver, verify the chosen odbc driver is on the PATH, and then run pip install ibm_db_django... (with whatever options are required for your django version, as per the README on the ibm_db_django github page).

Next, It is wise to verify that your stack is able to connect with DBI explicitly, before you try the django framework. Doing it that way allows for easier problem determination. If you cannot connect with simple DBI, then django and ibm_db_django will also not connect.

The simplest way to test the DBI connection explicitly is with a tiny python script , something like this:

import ibm_db_dbi
conn_string='...'   # put your known working connection string here, it can identify the odbc driver via DRIVER=....
user='...'  # whatever your userid for accessing the as400
userPassword='...' # the matching password for that account
try:
    conn = ibm_db_dbi.connect(conn_string , user, userPassword)
except Exception as e:
    print(e)

If you get an exception, report all its details in full. If you do not get an exception, it means your DBI is successfully connecting using your specified odbc driver.

florisjkruger4 commented 1 year ago

Yes I am aware that the ibm_db_django will not use the pyodbc interface, but instead will use the DBI interface. I just wanted to show that I can in fact connect using PYODBC so the database connection credentials are correct. The python version I am using is a 64 bit version, along with a 64 bit odbcad.exe.

I am quite confused since I downloaded the CLI Access ODBC Driver from this link on the IBM website. The readme states the following: `3.6 64-bit hardware installation considerations

When installed on a supported 64-bit Windows operating system:

So this CLI Access ODBC Driver should support both 64 and 32 bit? However when looking in the actual ODBC Administrator (64-bit) interface I get these results in my Drivers tab with the Client Acess ODBC Driver's name specifically being: "Client Access ODBC Driver (32-bit)" Is there a specific 64 bit version I need to use? odbcad-64bit

I am also confused as to what you mean by configuring the PATH in my environment variable, specifically when you mention the IBM_DB_HOME variable. I do not have this variable in my PATH anywhere. Here is how I have it set up env_path

I also wrote a small python program to try and connect via import ibm_db_dbi and configuring my connection string with the example you provided above. But the same exact thing is happening... If I change the PORT connection string to a value I know is incorrect like 22 and try connect I get this as the exception message django.db.utils.OperationalError: [IBM][CLI Driver] SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "10.11.5.12". Communication function detecting the error: "recv". Protocol specific error code(s): "*", "*", "0". SQLSTATE=08001 SQLCODE=-30081

However when I correct the settings and try to connect I do not get an exception, but my command line and program just hangs like before... I have also tried printing out a message after the connection string to see if the program actually gets past the connection but it never does.

I hope this can give some clarification as to why I am running into the "hanging" issue, I appreciate your quick response. Do you have any other suggestions?

imavo commented 1 year ago

Note that I do not have odbc access to any as400 system so my advice is general in nature.

The environnment variable IBM_DB_HOME is a different environment variable from PATH.

When that IBM_DB_HOME variable is not set (i.e does not exist), at the time you pip install ibm_db_django then this results in default behaviour (i.e causes installation and use of clidriver).

Programming is all about details. Show your test code including the connection string details (just put XXX for the userid and XXX for the password). Do not use graphics/pictures, use only plain text.

Look at the exception you get (with the deliberately bad port number), on the first line it begins "django.db.utils.OperationalError: [IBM][CLI Driver]" . That shows you the odbc-driver that is actually being used, in this case clidriver.

Only use clidriver with as400 only if one of these conditions are true (1) you have a license file for db2 connect deployed in the clidriver\license subdirectory, or (2) your i-series administrator has enabled server-side licensing via db2connectactivate. If both of these are false then you should not use clidriver. You have not stated whether these are true/false. So therefore I'm assuming that both are false for you, in which case try to use instead the i series access odbc driver (64-bit).i.e non default behaviour for ibm_db.

Read the advice in my first response again until you comprehend , especially about uninstalling, setting IBM_DB_HOME correctly, then re-installing ibm_db_django with that variable correctly set.

After making the above change, to use (with python) the i-series access odbc driver (64-bit version), ensure it is installed, configured, working for your DSN with the 64-bit odbcad32.exe, and 64-bit version is present on your PATH variable, and mention its driver name (from the odbcad32.exe (64-bit) gui) in your connection string Driver={...}; in your test-code for the DBI connect.

If you can get the ibm_db_dbi connecting with the IBM i series access odbc driver (64-bit) then you have a chance...

praveen-db2 commented 1 year ago

@florisjkruger4 ibm_db doesn't use any of below drivers you specified , Client Access ODBC Driver (32-bit) IBM i Access ODBC Driver iSeries Access ODBC Driver

It uses "IBM DB2 ODBC driver" also called as "IBM Data Server Driver for ODBC and CLI " or in short called as clidriver . Try installing the same with license and connect.

imavo commented 1 year ago

@praveen-db2 Can you explain further, what is the technical reason that ibm_db is unable to use the i access odbc driver?

bimalkjha commented 1 year ago

@imavo @florisjkruger4 ibm_db_django uses ibm_db for connectivity and ibm_db is developed to work with only clidriver, not with iAccess driver. You can see in ibm_db header file that ibm_db.h is dependent on sqlcli1.h which comes with clidriver only: https://github.com/ibmdb/python-ibmdb/blob/master/ibm_db.h#L15C1-L15C21

We never claim that ibm_db or ibm_db_django works with iAccess ODBC Driver. It works with IBM Db2 Clients only. If you want to use iAccess, then ibm_db is not the right module but you need to looks for some other module. ibm_db uses clidriver which requires db2connect license to access AS400. Hope it clarifies. Thanks.

imavo commented 1 year ago

@bimalkjha , , this does not answer my question re clarification.

I'm not asking about the compilation or building of the ibm_db extension. Instead I am only asking about the run time situation, because I see that ibm_db uses dynamic linking on MS-Windows.

Instead, I'm asking what is the technical reason that a DLL exporting all of required cli SQL APIs used by ibm_db (with the matching API names etc), and is first on the PATH when IBM_DB run, and those APIs were functionally at the odbc v3 level or above, what technical reason would actively __prevent*__ ibm_db from working with that DLL ?

( It is my assumption that the IBM i access odbc DLLs do provide all the required odbc v3 APIs as used by ibm_db ).

Hope you understand the difference. When I read the code, I'm not finding such dependencies. Hence the question. I do appreciate that the rocketsoftware people might not have tested such a configuration, but if there are technical reasons, I believe they should be clearly explained.

bimalkjha commented 1 year ago

@imavo ibm_db comes with a native binary file (ibm_db.cpython-36m-x86_64-linux-gnu.so on linux) which get generated by compiling ibm_db.c file. Similar binary we get on all distributed platforms. This binary dynamically links with libdb2.so. Check output of ldd ibm_db.cpython-36m-x86_64-linux-gnu.so on Linux. That means ibm_db dynamically loads only DB2 ODBC driver library not iAccess library. So, anything about iAccess is unknown for ibm_db. The dsn created under ODBC Data Source Administrator for iAccess is unknown for ibm_db as db2app.dll do not read iAccess DSN but IBM ODBC dsn. On Windows, we get ibm_db.cp310-win_amd64.pyd under site_packages after installation of ibm_db and ibm_db.cp310-win_amd64.pyd links with db2cli64.lib dynamically which links with db2app64.dll, it do not link with i access odbc dll. You can see library name as db2cli64 in ibm_db code here: https://github.com/ibmdb/python-ibmdb/blob/master/setup.py#L471C18-L471C18 and it eventually get passed to setup() as ext_modules argument. Hope it clarifies. Thanks.

imavo commented 1 year ago

@bimalkjha So, on MS-Windows x64, if there was a DLL present on the PATH named db2app64.dll, that exported all of the required APIs with the expected names/functionality etc, what would prevent ibm_db from loading and using it ?

bimalkjha commented 1 year ago

@imavo It is loading and using it. There is no issues with that. The problem is wit the way user is trying to work with ibm_db. See the connection string used: connection_string = 'DRIVER={IBM i Access ODBC Driver};SYSTEM=_____;UID=_____;PWD=_____'. It should be connection_string = 'DRIVER={IBM DB2 ODBC Driver};Database=_____;Host=__;Port=___;UID=_____;PWD=_____' . If user want to use DSN in connection string as DSN=___;UID=___;PWD=___; then user need to create valid DNS by selecting IBM DB2 ODBC DRIVER as in below screen shot, not a dsn under IBM i Access ODBC Driver. It is the wrong driver selected by user while creating a dsn. image

bimalkjha commented 1 year ago

If IBM DB2 ODBC DRIVER is not found to create a new Data Source in Microsoft ODBC Data Source Administrator, then user need to run below commands from Admin Command prompt:

db2cli install -cleanup -force
db2cli install -setup

For db2cli command to work, user need to add full path of clidriver/bin in PATH and clidriver/lib in LIB system level environment variable. clidriver can be found under site_packages directory if IBM_DB_HOME is not set before installation of ibm_db or a preinstalled db2 clients path is not added in PATH already. Thanks.

bimalkjha commented 1 year ago

We can see the file name for iSeries Access ODBC Driver in the screen shot shared by user. It is CWBODBC.DLL. For IBM DB2 ODBC Driver the file name under Drivers tab of ODBC Data Source Adminstrator, would be DB2CLIO.DLL as below: image

imavo commented 1 year ago

@bimalkjha , ok, i am satisfied that the "loading and using" bit is theoretically possible (apart from all the crypto stuff), as regards my request for clarification.

I suggest that the README file be improved to explicitly exclude the i access odbc driver (that is to say, to mention that ibm_db is unable to use that i series access odbc driver).

You understand that IBM customers might baulk at having to pay twice (i.e once for the IBM i series access, and again for the db2-connect license file...). But that is not a matter for ibm_db .

bimalkjha commented 1 year ago

@imavo Yes, it is possible by modifying the code of ibm_db. pyodbc already support all odbc drivers. We created ibm_db to remove dependency on installing a separate odbc driver as clidriver get automatically installed when we install ibm_db. Code of ibm_db is tuned to work only with db2 odbc driver. One of the purpose is to promote use of clidriver and db2connect license. We do not have any experience of working with iAccess driver. Thanks.

bimalkjha commented 1 year ago

@florisjkruger4 Could you please confirm the command db2level works for you from command prompt? If not, do you see clidriver under site_packages directory? If yes, add full path of site_packages\clidirver\bin in PATH and full path of site_packages\clidriver\lib in LIB, system level environment variable. Then open a new Admin Command Prompt and run below commands:

db2cli install -cleanup -force
db2cli install -setup

Now, you can create Data Source in ODBC Data Source Administrator for IBM DB2 ODBC Driver and use the created data source in connection string as: connection_string = 'DRIVER={IBM DB2 ODBC Driver};DSN=_____;UID=_____;PWD=_____'. Yes, you need db2connect license for connection to succeed. Alternatively, you can verify the database connectivity using below db2cli command from command prompt:

db2cli validate -database "<dbname>:<hostname or ipaddress>:<drda port no>" -user <dbuser> -passwd <dbpass> -connect

Let us know if you need any other info. Thanks.

SabaKauser commented 1 year ago

FYI, there is separate branch that has the driver to connect to IBM i, and possibly using iSeries Access ODBC driver https://github.com/ibmdb/python-ibmdb/tree/ibmi better to check with @kadler if this is being maintained.

bimalkjha commented 1 year ago

@SabaKauser I think code in ibmi branch is to install ibm_db on AS400 system, i.e. native support for AS400 using db400 library: https://github.com/ibmdb/python-ibmdb/blob/ibmi/IBM_DB/ibm_db/setup.py#L214 Here user want to connect to AS400 from Windows system. Thanks.

SabaKauser commented 1 year ago

oh I see, thanks @bimalkjha I do not see how we can do that considering that the focus as you already mentioned is to offer connectivity from our ODBC driver interface(both for promoting as well as from the maintenance perspective, we own end-to-end for bug fixes).

kadler commented 1 year ago

@florisjkruger4 you probably want to look at https://github.com/IBM/django-ibmi which uses the IBM i Access driver.

florisjkruger4 commented 1 year ago

@kadler is django-ibmi just a newer version of ibm-db-django? I will definitely give it a shot. I notice that a lot of the documentation is still TODO, did you just need to do a simple "pip install django-ibmi" command and configure it to use the IBM i Access Driver?

kadler commented 1 year ago

It's originally based off ibm-db-django, but only supports PyODBC + IBM i Access driver. It could use some polish yet.

kadler commented 1 year ago

@florisjkruger4 if you have more questions about it, probably best to open issues on that repo instead of talking about it here. There's also an IBM i Ryver community you can join here: http://ibm.biz/ibmioss-chat-join and more IBM i open source resources listed here: https://ibm.github.io/ibmi-oss-resources/

florisjkruger4 commented 1 year ago

IBM i Access ODBC Driver

@bimalkjha @praveen-db2

I appreciate all the responses and help! I am new to working with drivers and unfortunately I am still not successful with getting this connection going. I understand now that it looks like I do not have the correct driver installed to allow access to an AS400 database. The driver needed is IBM DB2 ODBC Driver and the command prompt db2level does not work for me, I get this response: 'db2level' is not recognized as an internal or external command, operable program or batch file.

I looked into installing this driver but I am not getting it to show up in my ODBC Administrator interface... I have tried downloading the correct driver named as "IBM Data Server Driver for ODBC and CLI (64-bit)" as seen in this screenshot.

IBMDATASERVERDRIVERFORODBCANDCLI64BIT

This redirected me to this page where I was able to download a compressed Zip file called v11.5.8_ntx64_odbc_cli.zip.

Next I extracted this folder and followed the steps from this stacked overflow page to try and install the driver. However, after navigating to the directory with my command prompt and using the commands `db2oreg1 –i

db2oreg1 –setup`

I can see the db2oreg1 Application within the file that I extracted, but running the command above simply does nothing and my command prompt looks like the following:

C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\v11.5.8_ntx64_odbc_cli (1)\clidriver\bin>db2oreg1 -i

C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\v11.5.8_ntx64_odbc_cli (1)\clidriver\bin>

I also tried a different version of the driver with the same commands like this:

C:\Users\floris.kruger\Downloads\v11.5.4_ntx64_odbc_cli\clidriver\bin>db2oreg1 -i

C:\Users\floris.kruger\Downloads\v11.5.4_ntx64_odbc_cli\clidriver\bin>

I am still not seeing this IBM DB2 ODBC Driver registered within my ODBC Administrator 64bit Interface. Could you please clarify how I can properly install this driver?

florisjkruger4 commented 1 year ago

@florisjkruger4 Could you please confirm the command db2level works for you from command prompt? If not, do you see clidriver under site_packages directory? If yes, add full path of site_packages\clidirver\bin in PATH and full path of site_packages\clidriver\lib in LIB, system level environment variable. Then open a new Admin Command Prompt and run below commands:

db2cli install -cleanup -force
db2cli install -setup

Now, you can create Data Source in ODBC Data Source Administrator for IBM DB2 ODBC Driver and use the created data source in connection string as: connection_string = 'DRIVER={IBM DB2 ODBC Driver};DSN=_____;UID=_____;PWD=_____'. Yes, you need db2connect license for connection to succeed. Alternatively, you can verify the database connectivity using below db2cli command from command prompt:

db2cli validate -database "<dbname>:<hostname or ipaddress>:<drda port no>" -user <dbuser> -passwd <dbpass> -connect

Let us know if you need any other info. Thanks.

I tried following the steps in your reply above, and I think I've made some progress? After trying the intitial command db2level and not being successful. I finally found the site_packages\clidirver\bin directory within my project. I added this path in PATH within my system level environment variables as well as site_packages\clidriver\lib since this page states that On the Windows operating system, both the library path and PATH environment variables are represented by the PATH system environment variable. aaa

Next I went to the file directory that leads to this path and I ran the following command and received a success message:

C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\test\env\Lib\site-packages\clidriver\bin>db2cli install -setup
IBM DATABASE 2 Interactive CLI Sample Program
(C) COPYRIGHT International Business Machines Corp. 1993,1996
All Rights Reserved
Licensed Materials - Property of IBM
US Government Users Restricted Rights - Use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

===============================================================================
"IBM Data Server Driver for ODBC and CLI" is registered successfully.

Necessary configuration folders and sample files are created successfully.
===============================================================================

However.... I am still not seeing the Driver located in my ODBC Data Source Administrator Interface, So I am unable to configure a Data Source in ODBC Data Source Administrator for IBM DB2 ODBC Driver Do you have any suggestions?

bimalkjha commented 1 year ago

@florisjkruger4 1. Yes, you need to create a new system level environment variable LIB and set its value as full path of site_packages\clidriver\lib directory.

  1. Please delete the downloaded file v11.5.8_ntx64_odbc_cli.zip and directory C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\v11.5.8_ntx64_odbc_cli (1)\clidriver from your system. It is not used by ibm_db. ibm_db uses one from site_packages.
  2. It seem you have not executed db2cli install -cleanup -force command from Administrator command prompt. First you need to execute db2cli install -cleanup -force and then db2cli install -setup command. After it, you can see IBM DB2 ODBC Driver in ODBC Data Source Administrator.

Do you have any specific need to create Data Source under ODBC Data Source Administrator? Alternate way is you can just add DSN in config file of clidriver using below commands and use DSN in connection string same as it is in Data Source administrator. Once, you have updated PATH and LIB system level environment variable, open new command window and execute below commands:

db2cli writecfg add -dsn mydsn -database <dbname> -host <hostname or ipaddress>  -port <dbport> -parameters "userid=<dbuser>;password=<dbpasswd>;"
db2cli validate -dsn mydsn -connect

If above two commands succeeds, you can just use "DSN=mydsn:" as connection string in you application. Thanks.

florisjkruger4 commented 1 year ago

@florisjkruger4 1. Yes, you need to create a new system level environment variable LIB and set its value as full path of site_packages\clidriver\lib directory. 2. Please delete the downloaded file v11.5.8_ntx64_odbc_cli.zip and directory C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\v11.5.8_ntx64_odbc_cli (1)\clidriver from your system. It is not used by ibm_db. ibm_db uses one from site_packages. 3. It seem you have not executed db2cli install -cleanup -force command from Administrator command prompt. First you need to execute db2cli install -cleanup -force and then db2cli install -setup command. After it, you can see IBM DB2 ODBC Driver in ODBC Data Source Administrator.

Do you have any specific need to create Data Source under ODBC Data Source Administrator? Alternate way is you can just add DSN in config file of clidriver using below commands and use DSN in connection string same as it is in Data Source administrator. Once, you have updated PATH and LIB system level environment variable, open new command window and execute below commands:

db2cli writecfg add -dsn mydsn -database <dbname> -host <hostname or ipaddress>  -port <dbport> -parameters "userid=<dbuser>;password=<dbpasswd>;"
db2cli validate -dsn mydsn -connect

If above two commands succeeds, you can just use "DSN=mydsn:" as connection string in you application. Thanks.

@bimalkjha I have my Environment variables configured as the following.... Path: C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\test\env\Lib\site-packages\clidriver\bin LIB: C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\test\env\Lib\site-packages\clidriver\lib

When I try and run the commands described above I get the following response:

terminal

No it is not important for me to set up a specific DSN, I just want to be able to use the IBM DB2 ODBC Driver to connect to the AS400 system. Unfortunately I still cannot see this driver inside my ODBC Data Source Administrator Interface. I have no clue what could be causing this issue?

bimalkjha commented 1 year ago

@florisjkruger4 Sorry for copy paste error. The second command should be just db2cli install -setup after db2cli install -cleanup -force. Error from db2cli install -cleanup -force can be ignored. From your command prompt, cd to C:\Users\floris.kruger and run db2level command. Let me know the output of db2level command. Then I can share next step. Thanks.

bimalkjha commented 1 year ago

Also, if db2level command works, then please share the complete output of command db2cli validate -database "<dbname>:<hostname or ipaddress>:<drda port no>" -user <dbuser> -passwd <dbpass> -connect. Update the database connection info in this command. Thanks.

florisjkruger4 commented 1 year ago

@bimalkjha The output for running db2level is the following: C:\Users\floris.kruger>db2level DB21085I This instance or install (instance name, where applicable: "*") uses "64" bits and DB2 code release "SQL11058" with level identifier "0609010F". Informational tokens are "DB2 v11.5.8000.3209", "special_26260", "DYN2301190517WIN64_26260", and Fix Pack "0". Product is installed at "C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\test\env\Lib\SITE-P~1\CLIDRI~1" with DB2 Copy Name "IBM Data Server Driver For ODBC and CLI".

The second output gives me this:

`C:\Users\floris.kruger>db2cli validate -database "_______:_______" -user ________ -passwd ________ -connect

===============================================================================
Client information for the current copy (copy name: C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_test_env_Lib_SITE-P~1_CLIDRI~1):
===============================================================================

Client Package Type       : IBM Data Server Driver For ODBC and CLI
Client Version (level/bit): DB2 v11.5.8000.3209 (special_26260/64-bit)
Client Platform           : NT 64BIT
Install/Instance Path     : C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\test\env\Lib\SITE-P~1\CLIDRI~1

Common App Data Path      : C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_test_env_Lib_SITE-P~1_CLIDRI~1
DB2DSDRIVER_CFG_PATH value: <not-set>
db2dsdriver.cfg Path      : <Common App Data Path>\cfg\db2dsdriver.cfg
DB2CLIINIPATH value       : <not-set>
db2cli.ini Path           : <Common App Data Path>\cfg\db2cli.ini
db2diag.log Path          : <Common App Data Path>\db2diag.log

===============================================================================
List of all IBM Data Server client packages on the current workstation:
===============================================================================

Copyname                                                                Version Package Language Installed-Location
-------------------------------------------------------------------------------
C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_test_env_Lib_SITE-P~1_CLIDRI~1[C] 11.5.0800 CLI     EN       C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\test\env\Lib\SITE-P~1\CLIDRI~1
-------------------------------------------------------------------------------
 CLI: IBM Data Server Driver for ODBC and CLI
-------------------------------------------------------------------------------

===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================

Note: The validation utility could not find the configuration file
db2dsdriver.cfg. The file is searched at
"C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_test_env_Lib_SITE-P~1_CLIDRI~1\cfg\db2dsdriver.cfg".`
bimalkjha commented 1 year ago

@florisjkruger4 You have not shared the last part of output which has result of -connect option as below. That is important and I want to know you are getting SUCCESS as below or not?


===============================================================================
Connection attempt for database "sample:myhost.com:60000":
===============================================================================

[SUCCESS]

===============================================================================
The validation is completed.
===============================================================================
bimalkjha commented 1 year ago

If you are getting SUCCESS, then execute below two commands and use "DSN=mydsn;" as connection string in your test programs as suggested in previous post and verify the connection from test application.

db2cli writecfg add -dsn mydsn -database <dbname> -host <hostname or ipaddress>  -port <dbport> -parameters "userid=<dbuser>;password=<dbpasswd>;"
db2cli validate -dsn mydsn -connect

Thanks.

florisjkruger4 commented 1 year ago

@bimalkjha Unfortunately I am not getting that success message and that was the end of my output where my command prompt hangs again and I have to ^C to stop it.

Note: The validation utility could not find the configuration file
db2dsdriver.cfg. The file is searched at
"C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_test_env_Lib_SITE-P~1_CLIDRI~1\cfg\db2dsdriver.cfg".

Is what the last bit of the output says... I would think this is what is blocking the connection to complete perhaps?

The following path: C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_V115~1.8_N_CLIDRI~1\cfg

Leads me to the following two files: ibmPath

It looks like the db2dsdriver.cfg file the process is looking for is db2dsdriver.cfg.sample?

However the path in the return message from my command prompt: C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_test_env_lib_SITE-P~1_CLIDRI~1\cfg

Leads me to an empty file

florisjkruger4 commented 1 year ago

@bimalkjha After running the command db2cli install -setup within my virtual environment for my project I now also have the two files: db2cli.ini.sample db2dsdriver.cfg.sample

Inside the path directory: C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_test_env_lib_SITE-P~1_CLIDRI~1\cfg

This seems to be where the command line is looking for the configuration file when it is talking about

The file is searched at
"C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_test_env_Lib_SITE-P~1_CLIDRI~1\cfg\db2dsdriver.cfg".

IBM's website states: "Ensure that the db2dsdriver.cfg file is created in your environment. You can use the contents of the db2dsdriver.cfg.sample file to create the db2dsdriver.cfg file."

Am I supposed to use these files in some way so that the connection may work?

bimalkjha commented 1 year ago

@florisjkruger4 No need to do anything with db2cli.ini.sample or db2dsdriver.cfg.sample file. Your main aim should be to get SUCCESS message from db2cli validate -database "<dbname>:<hostname or ipaddress>:<drda port no>" -user <dbuser> -passwd <dbpass> -connect command. Since, you are inside a virtual environment, it seems your virtual environment is blocking a connection request to go outside from your virtual environment, something like a firewall blocks a socket to open. clidriver communicates with server via socket over tcpip. As the behavior you observe is a hang, I suspect your virtual env is not allowing clidriver to open a socket. Please look into system log or env log to see where it is blocked? Try to execute the db2cli validate -database "<dbname>:<hostname or ipaddress>:<drda port no>" -user <dbuser> -passwd <dbpass> -connect command from Administrator command prompt and see if it works or you are getting any prompt to give permission to db2cli to open a socket? Finally you can test the connectivity outside of your virtual environment by just installing python3 and ibm_db with django, find clidriver, set PATH and LIB and then execute db2cli validate command and check if it works outside of virtual environment. Thanks.

florisjkruger4 commented 1 year ago

@bimalkjha @praveen-db2 @imavo @SabaKauser Alright, so I followed this page on IBM's website step by step. I am not inside a virtual environment as you can see from the following path: C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\IBMDB2\CLIDRIVER\v11.5.8_ntx64_odbc_cli\clidriver\bin>

I also have successfully registered this driver after running: db2cli install -setup

===============================================================================
"IBM Data Server Driver for ODBC and CLI" is registered successfully.

Necessary configuration folders and sample files are created successfully.
===============================================================================

This created my sample configuration files in the following application data path C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1\cfg

This directory contains the following SAMPLE files: db2cli.ini db2dsdriver.cfg

I also have two environment variables with the following paths related to the clidriver: Path: C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\IBMDB2\CLIDRIVER\v11.5.8_ntx64_odbc_cli\clidriver\bin LIB: C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\IBMDB2\CLIDRIVER\v11.5.8_ntx64_odbc_cli\clidriver\lib

Once again when I try and run the following command: C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\IBMDB2\CLIDRIVER\v11.5.8_ntx64_odbc_cli\clidriver\bin>db2cli validate -database "_______:_______:_______" -user _______ -passwd _______ -connect

This is my output:

===============================================================================
Client information for the current copy (copy name: C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1):
===============================================================================

Client Package Type       : IBM Data Server Driver For ODBC and CLI
Client Version (level/bit): DB2 v11.5.8000.317 (s2209201700/64-bit)
Client Platform           : NT 64BIT
Install/Instance Path     : C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1

Common App Data Path      : C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1
DB2DSDRIVER_CFG_PATH value: <not-set>
db2dsdriver.cfg Path      : <Common App Data Path>\cfg\db2dsdriver.cfg
DB2CLIINIPATH value       : <not-set>
db2cli.ini Path           : <Common App Data Path>\cfg\db2cli.ini
db2diag.log Path          : <Common App Data Path>\db2diag.log

===============================================================================
List of all IBM Data Server client packages on the current workstation:
===============================================================================

Copyname                                                                     Version Package Language Installed-Location
-------------------------------------------------------------------------------
C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1[C] 11.5.0800 CLI     EN       C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1
-------------------------------------------------------------------------------
 CLI: IBM Data Server Driver for ODBC and CLI
-------------------------------------------------------------------------------

===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================

Note: The validation utility could not find the configuration file
db2dsdriver.cfg. The file is searched at
"C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1\cfg\db2dsdriver.cfg".

If I deliberately change my PORT number to something that I know is incorrect and I try to execute the command above then I get the following output:

[FAILED]: [IBM][CLI Driver] SQL30081N  A communication error has been detected. Communication protocol being used: "TCP/IP".  Communication API being used: "SOCKETS".  Location where the error was detected: "10.11.5.12".  Communication function detecting the error: "recv".  Protocol specific error code(s): "*", "*", "0".  SQLSTATE=08001

===============================================================================
Error: The validation operation failed.
===============================================================================

===============================================================================
The validation is completed.
===============================================================================

I looked into my system log and cannot find anything related to the port being blocked... I am honestly not sure what a lot of this information means but I will past the system log for this connection process below:

 Command Line: db2cli  validate -database "_______:_______:_______" -user _______ -passwd _______ -connect
 Process Id: 6428
 Parent Process Id: 29912
 Workstyle: High Flexibility
 Application Group: (Default) Any Application
 Reason: <None>
 File Name: c:\users\floris.kruger\onedrive - dot foods, inc\desktop\ibmdb2\clidriver\v11.5.8_ntx64_odbc_cli\clidriver\bin\db2cli.exe
 Hash: 4F38E66E55E6390C8AFCB3492DC90BCBA32EF331
 Certificate: <None>
 Description: IBM(R) DB2(R)
 Application Type: exe
 Product Name: IBM(R) DB2(R)
 Product Code: <None>
 Upgrade Code: <None>
 Product Version: 11.5.8000.317
 File Version: 11.5.8000.317
 Application Group Description: This will match for every application type Privilege Management supports
 Workstyle Description: Workstyle that applies to users who have a lot of flexibility
 Token Assignment Id: f4637e57-24b5-445a-9cc5-2ac9b0397b6f
 Token Assignment Is Shell: false
 Token Id: f8d4ce02-e95d-4700-b69a-957dc5c1de6f
 Message Id: <None>
 Token: Passive (No Change)
 Token Description: <None>
 Message Name: <None>
 Message Description: <None>
 Unique Process ID: f11a1f58-f422-4182-bf4d-1a873ea19cc9
 Workstyle ID: 301b8962-67e3-4ad8-b585-a97ad05c409b
 Application Group ID: 0b9efdf9-a490-4efc-80f1-482c5361ba5e
 User SID: S-1-5-21-33307682-826612263-262303683-152868
 User Name: floris.kruger
 User Domain SID: S-1-5-21-33307682-826612263-262303683
 User Domain Name: DOTFOODS
 User Domain Name NetBIOS: DOTFOODS
 Host SID: S-1-5-21-33307682-826612263-262303683-111082
 Host Name: 1-KRUGERF-ZBG9
 Host Name NetBIOS: 1-KRUGERF-ZBG9
 Host Domain SID: S-1-5-21-33307682-826612263-262303683
 Host Domain Name: dotfoods.com
 Host Domain Name NetBIOS: DOTFOODS
 Event ID: 12BA998A-FEA1-470F-BE62-75A82DC30EC4
 Process Start Time: 133365951637354780
 Process End Time: 0
 Event Time: 133365951637354780
 Authorizing User SID: <None>
 Authorizing User Name: <None>
 Authorizing User Domain SID: <None>
 Authorizing User Domain Name: <None>
 Authorizing User Domain Name NetBios: <None>
 Client IPV4: <None>
 Client Name:  <None>
 UAC Triggered: false
 File Owner SID: S-1-5-21-33307682-826612263-262303683-152868
 File Owner Name: floris.kruger
 File Owner Domain SID: S-1-5-21-33307682-826612263-262303683
 File Owner Domain Name: DOTFOODS
 File Owner Domain Name NetBIOS: DOTFOODS
 Parent Process Unique ID: 64294595-b832-4f38-8cce-c599901a41dd
 Parent Process File Name: c:\windows\system32\cmd.exe
 COM CLSID: <None>
 COM AppID: <None>
 COM Display Name: <None>
 Source URL: <None>
 Authorization Challenge: <None>
 Windows Store App Name: <None>
 Windows Store App Publisher: <None>
 Windows Store App Version: <None>
 Drive Type: Fixed Disk
 Challenge Response Status: <None>
 PowerShell Command: <None>
 Application Workstyle Description: Any Executable
 Application Workstyle Id: cc4758c6-09b7-4e62-9ffd-e0870cb9e313
 Message Type: <None>
 IE Zone Tag: <None>
 MD5: F0CE3D595202233E702732FAF0E6DE9D
 Host Local SID: S-1-5-21-2868025956-968045739-1026022543
 Trusted Application Name: <None>
 Trusted Application Version: <None>
 Uninstall Action: <None>
 Rule Script File Name: <None>
 Rule Script Name: <None>
 Rule Script Version: <None>
 Rule Script Publisher: <None>
 Rule Script Rule Affected: false
 Rule Script Result: <None>
 Rule Script Output: <None>
 Rule Script Status: <None>
 Auth Methods: <None>
 IdP Authentication User Name: <None>
 Configuration ID: 14739553-651e-4320-9f09-a1bd11f0f23f
 Configuration Revision Number: 965
 SHA256: 5B43BA3897768DF0CC874E9AAB6E87BAB4A0342A8A018BE6BE22A1A30CBF5E8C

Any help and advice would greatly be appreciated... I really want to get this connection working. I also have the following pip list:

C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\IBMDB2\CLIDRIVER\v11.5.8_ntx64_odbc_cli\clidriver\bin>pip list
Package           Version
----------------- --------
asgiref           3.7.2
Django            4.2.4
ibm-db            3.1.4
ibm-db-django     1.5.2.0
regex             2023.8.8
six               1.16.0
sqlparse          0.4.4
typing_extensions 4.7.1
tzdata            2023.3
bimalkjha commented 1 year ago

@florisjkruger4 Please execute below commands from Administrator command prompt and share all generated files. If db2cli validate command hangs, kill it using ctrl+c as before and continue running other commands.

db2trc on -f 1.trc
db2cli validate -database "<dbname>:<hostname or ipaddress>:<drda port no>" -user <dbuser> -passwd <dbpass> -connect
ctrl+c
db2trc off
db2trc flw 1.trc 1.flw
db2trc fmt 1.trc 1.fmt
db2trc fmt -cli 1.trc 1.cli

zip all generated 1.* files and share here. We'll look into these log files to find the root cause. Thanks.

florisjkruger4 commented 1 year ago

@bimalkjha I am having an issue with the first command:

C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\IBMDB2\CLIDRIVER>db2trc on -f 1.trc
Trace could not allocate the necessary IPC resources.  Please retry
using a smaller buffer size.
C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\IBMDB2\CLIDRIVER>db2trc on -t -l 8m -f 1.trc
Trace could not allocate the necessary IPC resources.  Please retry
using a smaller buffer size.
C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\IBMDB2\CLIDRIVER>db2trc on -t -l 2m -f 1.trc
Warning: The requested file size is < 8 MB. The filesize has been set to 8 MB.
Trace could not allocate the necessary IPC resources.  Please retry
using a smaller buffer size.

Seems like my tracer wont start and therefore the rest of the commands will not be of use...

bimalkjha commented 1 year ago

@florisjkruger4 db2trc on command should work without any message, otherwise tracing do not get enabled and no data get captured. Such messages comes when you run db2trc on command from non-administrator command prompt. Please check your command prompt and make sure it has Administrative priviledge. Try below commands:

db2trc on -t -l 8m
db2cli validate -database "<dbname>:<hostname or ipaddress>:<drda port no>" -user <dbuser> -passwd <dbpass> -connect
ctrl+c
db2trc dump 1.trc
db2trc off
db2trc flw 1.trc 1.flw
db2trc fmt 1.trc 1.fmt
db2trc fmt -cli 1.trc 1.cli

If db2trc on -t -l 8m do not work, then try db2trc on -t -l 4m or just db2trc on. Thanks.

florisjkruger4 commented 1 year ago

@bimalkjha I am in fact in an Administrator command prompt as displayed in the top left of this picture... unfortunately none pf these commands are working for me, should I perhaps try and open up db2trc directly from the clidriver/bin/ directory and run the application? administrator

florisjkruger4 commented 1 year ago

@bimalkjha @praveen-db2 Ok I may have made some progress? So I tried making my own db2dsdriver.cfg and db2cli.ini files within the directory where my driver is installed. I just used the contents of the sample db2dsdriver and db2cli files with my .ini file edited with my uid and pwd variables to log into the database.

[tstcli1x]
uid=_______
pwd=_______
autocommit=0
TableType="'TABLE','VIEW','SYSTEM TABLE'"

[tstcli2x]
; Assuming dbalias2 is a database in DB2 for MVS.
SchemaList="'OWNER1','OWNER2',CURRENT SQLID"

[MyVeryLongDBALIASName]
dbalias=dbalias3
SysSchema=MYSCHEMA

After running the following command I get this output with a success message at the bottom, however my command line does not terminate and I still need to ^C after the success message.

>db2cli validate -database "_______:_______:_______" -user FKRUGER -passwd _______ -connect

===============================================================================
Client information for the current copy (copy name: C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1):
===============================================================================

Client Package Type       : IBM Data Server Driver For ODBC and CLI
Client Version (level/bit): DB2 v11.5.8000.317 (s2209201700/64-bit)
Client Platform           : NT 64BIT
Install/Instance Path     : C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1

Common App Data Path      : C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1
DB2DSDRIVER_CFG_PATH value: <not-set>
db2dsdriver.cfg Path      : <Common App Data Path>\cfg\db2dsdriver.cfg
DB2CLIINIPATH value       : <not-set>
db2cli.ini Path           : <Common App Data Path>\cfg\db2cli.ini
db2diag.log Path          : <Common App Data Path>\db2diag.log

===============================================================================
List of all IBM Data Server client packages on the current workstation:
===============================================================================

Copyname                                                                     Version Package Language Installed-Location
-------------------------------------------------------------------------------
C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1[C] 11.5.0800 CLI     EN       C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1
-------------------------------------------------------------------------------
 CLI: IBM Data Server Driver for ODBC and CLI
-------------------------------------------------------------------------------

===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================

Success: The system db2dsdriver.cfg schema validation completed successfully without any errors.

Does this success message mean anything? Once again this command does not terminate, I have to ^C to stop the terminal.

The following sample dsn validation command gives me the following response and does in fact terminate:

(env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\dj>db2cli validate -dsn sampledsn

===============================================================================
Client information for the current copy (copy name: C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1):
===============================================================================

Client Package Type       : IBM Data Server Driver For ODBC and CLI
Client Version (level/bit): DB2 v11.5.8000.317 (s2209201700/64-bit)
Client Platform           : NT 64BIT
Install/Instance Path     : C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1

Common App Data Path      : C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1
DB2DSDRIVER_CFG_PATH value: <not-set>
db2dsdriver.cfg Path      : <Common App Data Path>\cfg\db2dsdriver.cfg
DB2CLIINIPATH value       : <not-set>
db2cli.ini Path           : <Common App Data Path>\cfg\db2cli.ini
db2diag.log Path          : <Common App Data Path>\db2diag.log

===============================================================================
List of all IBM Data Server client packages on the current workstation:
===============================================================================

Copyname                                                                     Version Package Language Installed-Location
-------------------------------------------------------------------------------
C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1[C] 11.5.0800 CLI     EN       C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1
-------------------------------------------------------------------------------
 CLI: IBM Data Server Driver for ODBC and CLI
-------------------------------------------------------------------------------

===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================

Success: The system db2dsdriver.cfg schema validation completed successfully without any errors.

===============================================================================
db2cli.ini validation for data source name "sampledsn":
===============================================================================

Note: The requested data source name or a database name cannot be found in
db2cli.ini file. The file is searched at
"C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1\cfg\db2cli.ini".

===============================================================================
db2dsdriver.cfg validation for data source name "sampledsn":
===============================================================================

Note: The requested data source name or a database name cannot be found in
db2dsdriver.cfg file. The file is searched at
"C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1\cfg\db2dsdriver.cfg".

===============================================================================
The validation is completed.
===============================================================================

(env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\dj>

Next I ran this command which configured and changed my db2dsdriver.cfg command and I got the following response:

(env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\dj>db2cli writecfg add -dsn mydsn -database _______ -host _______  -port __ -parameters "userid=_______;password=_______;"

===============================================================================
db2cli writecfg completed successfully.
===============================================================================

Although it seems like the cli driver is installed and registered and I am getting some success messages... I am still not able to see this IBM Data Server Driver for ODBC and CLI driver in my ODBC Administrator for either 64bit or 32bit. Any more suggestions would be greatly appreciated.

bimalkjha commented 1 year ago

@florisjkruger4 The connection is getting hang tells that client is waiting for response from server or server is not opening the socket for communication. Please check with your AS400 sysadmin that server is listening this DRDA port or not. Also, tell them to check when server received request for connection from client, then why server is not responding back? Since, by changing port, you are getting SQL30081 error, so driver is able to send connection request to server, but server is not responding back! Thanks.

florisjkruger4 commented 1 year ago

@bimalkjha I have finally gotten a successful connection! I believe the problem was me trying to use a TELNET port instead of the DRDA port.

C:\Users\floris.kruger>db2cli validate -database "XXXXX:XXXXX:XXX" -user XXXXX -passwd XXXXX -connect

===============================================================================
Client information for the current copy (copy name: C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1):
===============================================================================

Client Package Type       : IBM Data Server Driver For ODBC and CLI
Client Version (level/bit): DB2 v11.5.8000.317 (s2209201700/64-bit)
Client Platform           : NT 64BIT
Install/Instance Path     : C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1

Common App Data Path      : C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1
DB2DSDRIVER_CFG_PATH value: C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1\cfg
System db2dsdriver.cfg Path: <Common App Data Path>\cfg\db2dsdriver.cfg
DB2CLIINIPATH value       : <not-set>
db2cli.ini Path           : <Common App Data Path>\cfg\db2cli.ini
db2diag.log Path          : <Common App Data Path>\db2diag.log

===============================================================================
List of all IBM Data Server client packages on the current workstation:
===============================================================================

Copyname                                                                     Version Package Language Installed-Location
-------------------------------------------------------------------------------
C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1[C] 11.5.0800 CLI     EN       C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1
-------------------------------------------------------------------------------
 CLI: IBM Data Server Driver for ODBC and CLI
-------------------------------------------------------------------------------

===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================

Success: The system db2dsdriver.cfg schema validation completed successfully without any errors.

===============================================================================
db2cli.ini validation for database "XXXXX:XXXXX:XXX":
===============================================================================

Note: The requested data source name or a database name cannot be found in
db2cli.ini file. The file is searched at
"C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1\cfg\db2cli.ini".

===============================================================================
db2dsdriver.cfg validation for database "Arctic:10.11.5.12:446":
===============================================================================

Keywords                  Valid For     Value
---------------------------------------------------------------------------
GlobalParam               **UNKNOWN**

===============================================================================
Connection attempt for database "XXXXX:XXXXX:XXX":
===============================================================================

[SUCCESS]

===============================================================================
The validation is completed.
===============================================================================

C:\Users\floris.kruger>

In a previous reply from you @bimalkjha, you started to do the following if I get a success message:

db2cli writecfg add -dsn mydsn -database <dbname> -host <hostname or ipaddress>  -port <dbport> -parameters "userid=<dbuser>;password=<dbpasswd>;"
db2cli validate -dsn mydsn -connect

Whenever I run db2cli validate -dsn mydsn -connect my program hangs again.... not sure why?

I initially started the thread since my goal is to be able to connect a Django Project I am working on to this AS400 database. From the information provided by @praveen-db2, I am not able to establish this connection without the driver: IBM Data Server Driver for ODBC and CLI

My ODBC Administrator Interface still only has the following ODBC drivers:

Client Access ODBC Driver (32-bit)
IBM i Access ODBC Driver
iSeries Access ODBC Driver

What should my next steps be in order to use idm_dband ibm_db_djangocorrectly so that they do not use the Access drivers above but instead uses this cli IBM Data Server Driver for ODBC and CLI driver to establish a connection?

The Django settings.py Database configuration is as follows:

DATABASES = {
    'default': {
        'ENGINE'     : 'ibm_db_django',
        'NAME'       : 'XXXXX',
        'USER'       : 'XXXXX',
        'PASSWORD'   : 'XXXXX',
        'HOST'       : 'XXXXX',
        'PORT'       : 'XXXXX',
        'PCONNECT'   : True,
    }
}
bimalkjha commented 1 year ago

@florisjkruger4 Its really good news that you found the correct port for validate command. Same drda port should be used at other places too not the telnet port. Please rerun below commands again once db2cli validate command works for you from normal command prompt:

db2cli writecfg add -dsn mydsn -database <dbname> -host <hostname or ipaddress>  -port <dbport> -parameters "userid=<dbuser>;password=<dbpasswd>;"
db2cli validate -dsn mydsn -connect

Please revalidate the connection info in above command and make sure it is same as used in validate command, especially the port number. The second validate command should also work fine. In place of mydsn you can use any name as per your choice but other 5 connection info should be as you used in successful validate command. If still you see hand for second validate command, open file C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_IBMDB2_CLIDRI~1_V115~1.8_N_CLIDRI~1\cfg\db2dsdriver.cfg in editor and check the value of connection info is correct in this file or not. Check port number too. Finally, check the PORT value in settings.py, it should be 446. Thanks.

florisjkruger4 commented 1 year ago

Both of these command worked!!!!

db2cli writecfg add -dsn mydsn -database <dbname> -host <hostname or ipaddress>  -port <dbport> -parameters "userid=<dbuser>;password=<dbpasswd>;"
db2cli validate -dsn mydsn -connect

I can go ahead and configure the following values in my settings.py database settings and make them the same as the values I specified in the parameters above.

DATABASES = {
    'default': {
        'ENGINE'     : 'ibm_db_django',
        'NAME'       : 'XXXXX',
        'USER'       : 'XXXXX',
        'PASSWORD'   : 'XXXXX',
        'HOST'       : 'XXXXX',
        'PORT'       : '446',
        'PCONNECT'   : True,
    }
}

I am not sure exactly how to configure this Django project for this connection to work when I try to run the server on my localhost... I have installed all the packages as specified in the ibm_db_django documentation. This is my pip list output:

(env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\dj>pip list
Package              Version
-------------------- --------
asgiref              3.7.2
Django               4.2.4
django-debug-toolbar 4.2.0
ibm-db               3.1.4
ibm-db-django        1.5.2.0
pip                  23.0.1
pyodbc               4.0.39
regex                2023.8.8
setuptools           65.5.0
six                  1.16.0
sqlparse             0.4.4
typing_extensions    4.7.1
tzdata               2023.3

The documentation for doing pip install ibm_db states that this command also installs the clidriver within site packages within my Django project. I am not sure if I am supposed to replace this clidriver file with the clidriver file located in the following directory since this is where the dbcli validate commands accesses the driver it seems based on this directory: Install/Instance Path : C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1

Here is a picture of my Django projects file structure, I notice there is also a db2dsdriver.xsd file that has a similar structure to the db2dsdriver.cfg. Not sure if this has any significance and needs to be configured similarly to the db2dsdriver file?

DJfiles

Right now if I try and run my Django project with the database settings in settings.py configured with the command: python manage.py runserver

I get the following response:

django.core.exceptions.ImproperlyConfigured: 'ibm_db_django' isn't an available database backend or couldn't be imported. Check the above exception. To use one of the built-in backends, use 'django.db.backends.XXX', where XXX is one of:
    'mysql', 'oracle', 'postgresql', 'sqlite3'
bimalkjha commented 1 year ago

@florisjkruger4 The db2dsdriver.xsd is standard file whipped with clidriver and it is not supposed to be edited or modified by user. It should be read only and used internally by clidriver for xml syntax validation of db2dsdriver.cfg file. There should be only one copy of clidriver in the system. Multiple copy create confusion and problem. How you got your clidriver as C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1? It should be under site_pacakges and I do not see site_package anywhere in path C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRI~1\V115~1.8_N\CLIDRI~1. Please delete directory C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRIVER and update system level environment variables to have C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\dj\env\lib\site_packages\clidiver instead of C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\IBMDB2\CLIDRIVER in both PATH and LIB. Then you can open a new command prompt and run your test program to verify. Make sure you do not have another copy of settings.py with incorrect port. Thanks.

bimalkjha commented 1 year ago

Also, if possible, move full DJ directory out of OneDrive. I observed some issue when application is hosted on OneDrive in my system. You can compare both case i.e. inside OneDrive and outside it and if both works, then you can keep on onedrive as issue may be specific to my system. Thanks.

florisjkruger4 commented 1 year ago

@bimalkjha Alright, so I have deleted the directory specified above along with all other copies of the clidriver on my system. I created a new directory to host a Django project called "plzwrk" with the specified site packages including the clidriver. From running the command db2cli validate -dsn mydsn -connect I still do get a successful connection as shown below. Also note the path:

Copyname                                                                  Version Package Language Installed-Location
-------------------------------------------------------------------------------
C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_plzwrk_env_Lib_SITE-P~1_CLIDRI~1[C] 11.5.0800 CLI     EN       C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\plzwrk\env\Lib\SITE-P~1\CLIDRI~1
-------------------------------------------------------------------------------
 CLI: IBM Data Server Driver for ODBC and CLI
-------------------------------------------------------------------------------

This path shows that the clidriver is inside my Django project in the site packages and the output below shows the the connection test is working.

(env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk>db2cli validate -dsn mydsn -connect

===============================================================================
Client information for the current copy (copy name: C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_plzwrk_env_Lib_SITE-P~1_CLIDRI~1):
===============================================================================

Client Package Type       : IBM Data Server Driver For ODBC and CLI
Client Version (level/bit): DB2 v11.5.8000.317 (s2209201700/64-bit)
Client Platform           : NT 64BIT
Install/Instance Path     : C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\plzwrk\env\Lib\SITE-P~1\CLIDRI~1

Common App Data Path      : C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_plzwrk_env_Lib_SITE-P~1_CLIDRI~1
DB2DSDRIVER_CFG_PATH value: C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_plzwrk_env_Lib_SITE-P~1_CLIDRI~1\cfg
System db2dsdriver.cfg Path: <Common App Data Path>\cfg\db2dsdriver.cfg
DB2CLIINIPATH value       : <not-set>
db2cli.ini Path           : <Common App Data Path>\cfg\db2cli.ini
db2diag.log Path          : <Common App Data Path>\db2diag.log

===============================================================================
List of all IBM Data Server client packages on the current workstation:
===============================================================================

Copyname                                                                  Version Package Language Installed-Location
-------------------------------------------------------------------------------
C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_plzwrk_env_Lib_SITE-P~1_CLIDRI~1[C] 11.5.0800 CLI     EN       C:\Users\FLORIS~1.KRU\ONEDRI~1\Desktop\plzwrk\env\Lib\SITE-P~1\CLIDRI~1
-------------------------------------------------------------------------------
 CLI: IBM Data Server Driver for ODBC and CLI
-------------------------------------------------------------------------------

===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================

Success: The system db2dsdriver.cfg schema validation completed successfully without any errors.

===============================================================================
db2cli.ini validation for data source name "mydsn":
===============================================================================

Note: The requested data source name or a database name cannot be found in
db2cli.ini file. The file is searched at
"C:\ProgramData\IBM\DB2\C_Users_FLORIS~1.KRU_ONEDRI~1_Desktop_plzwrk_env_Lib_SITE-P~1_CLIDRI~1\cfg\db2cli.ini".

===============================================================================
db2dsdriver.cfg validation for data source name "mydsn":
===============================================================================

[ Parameters used for the connection ]

Keywords                  Valid For     Value
---------------------------------------------------------------------------
DATABASE                  CLI,.NET,ESQL ARCTIC
HOSTNAME                  CLI,.NET,ESQL 10.11.5.12
PORT                      CLI,.NET,ESQL 446
USERID                    CLI,.NET      FKRUGER
PASSWORD                  CLI,.NET      ************

GlobalParam               **UNKNOWN**

===============================================================================
Connection attempt for data source name "mydsn":
===============================================================================

[SUCCESS]

===============================================================================
The validation is completed.
===============================================================================

(env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk>

I also have my database settings in settings.py configured with the same values as under the[ Parameters used for the connection ] section above with the engine set to ibm_db_django. I would think that everything is set and I should be able to connect and be able to execute the python manage.py runserver command now? Upon trying to execute this command I get the following output:

(env) C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk>python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\ibm_db_dbi.py", line 871, in tables
    row = ibm_db.fetch_assoc(stmt)
Exception: Column information cannot be retrieved: [IBM][CLI Driver][AS] SQL0332N  Character conversion from the source code page "937" to the target code page "1252" is not supported.  SQLCODE=-332

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\floris.kruger\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "C:\Users\floris.kruger\AppData\Local\Programs\Python\Python310\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\utils\autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\core\management\commands\runserver.py", line 121, in inner_run
    self.check_migrations()
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\core\management\base.py", line 486, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\db\migrations\executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\db\migrations\loader.py", line 53, in __init__
    self.build_graph()
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\db\migrations\loader.py", line 220, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\db\migrations\recorder.py", line 77, in applied_migrations
    if self.has_table():
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\db\migrations\recorder.py", line 56, in has_table
    tables = self.connection.introspection.table_names(cursor)
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\db\backends\base\introspection.py", line 52, in table_names
    return get_names(cursor)
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\django\db\backends\base\introspection.py", line 47, in get_names
    return sorted(ti.name for ti in self.get_table_list(cursor)
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\ibm_db_django\introspection.py", line 115, in get_table_list
    for table in cursor.connection.tables( cursor.connection.get_current_schema() ):
  File "C:\Users\floris.kruger\OneDrive - Dot Foods, Inc\Desktop\plzwrk\env\lib\site-packages\ibm_db_dbi.py", line 879, in tables
    raise _get_exception(inst)
ibm_db_dbi.DatabaseError: ibm_db_dbi::DatabaseError: Column information cannot be retrieved: [IBM][CLI Driver][AS] SQL0332N  Character conversion from the source code page "937" to the target code page "1252" is not supported.  SQLCODE=-332

According to the output above the error occurs in my ibm_db_dbi.py file on line 876: row = ibm.db.fetch_assoc(stmt) in this code block:

try:
            stmt = ibm_db.tables(self.conn_handler, None, schema_name, table_name)
            row = ibm_db.fetch_assoc(stmt)
            i = 0
            while (row):
                result.append( row )
                i += 1
                row = ibm_db.fetch_assoc(stmt)
            ibm_db.free_result(stmt)
        except Exception as inst:
            raise _get_exception(inst)

This SQLCODE = -322 has something to do with WHEN EBCDIC OR ASCII GRAPHIC DATA BEING USED IF THE SYSTEM PARAMETER MIXED IS SET TO NO. The solution on this page does not really help me since my project is Django specific and I am not really understanding what it means? Do you have any ideas?

Thanks.

bimalkjha commented 1 year ago

@florisjkruger4 Its good to know that now you are able to connect to AS400 using django. To overcome this codepage mismatch error, create a system level environment variable DB2CODEPAGE and set its value as 1208. Then open a new command prompt and run your application. Thanks.

florisjkruger4 commented 1 year ago

@bimalkjha First of all I just want to say THANK YOU SO MUCH! You are an absolute legend for helping me this far and I have learnt so much through this process. I now have a successful connection to an AS400 database within my Django project.

Finally I was hoping you could help me with one more final step? The AS400 database I am connecting to has a lot of tables and data already on it. My goal was to be able to create Django models to represent some of these tables and be able to use the following migrate commands to allow my models to have access to the data in these tables so that I can then use Django ORM's to access the data.

python manage.py makemigrations
python manage.py migrate

Currently I do not have any models in my Django project so I just want to do an initial migration with thepython manage.py migrate command which leads me to this error: django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table (Statement Execute Failed: [IBM][CLI Driver][AS] SQL0204N "QSYS.FKRUGER" is an undefined name. SQLSTATE=42704 SQLCODE=-204)

QSYS is the root library of the AS400 system, and from it there branches a lot of different paths but "FKRUGER" is not one of them which makes sense why I am getting this error message.

From looking at the following view you can see that I am able to query data from the AS400 system from a table in my personal library called "XPFLORIS" which resides in QSYS.

import ibm_db
from ibm_db_dbi import Connection

def Dashboard(request):

    ibm_db_conn = ibm_db.pconnect("mydsn","FKRUGER","XXXXXXX")
    conn = Connection(ibm_db_conn)
    cursor = conn.cursor()
    cursor.execute(
        "SELECT * FROM XPFLORIS.CLIENTPF"
    )
    for x in cursor:
        print(x)

Is there a way for me to configure the current Schema so that instead of QSYS.FKRUGER it looks for QSYS.XPFLORIS or some other library I know is in the AS400 system under QSYS? I tried doing this in the db2cli.ini and db2dsdriver.cfg files but that did not seems to work.

Also I would assume that upon a successful migration my Django project work also create the 12 default tables in the AS400 database library XPFLORIS like auth_user, auth_permission, django_admin_log, django_sessions ... ect. Are you able to confirm this?

Ultimately I just want to be able to access some of these tables in the AS400 using Django Models that have matching fields and column names. I have done this before with a simple MySQL database but is this possible to do with connecting to an AS400 database?

praveen-db2 commented 1 year ago

@florisjkruger4 for your first query you can execute "set current schema ....." for 2nd query, yes Django creates default tables needed for its framework support.

for 3rd query Yes you should be able to access AS400 tables using Django Models.