missing pyisam.backend.cffi._isam_cffi

[qippur]

Hi, I know this repo is not updated since a lot, but I am trying anyway to ask.

I cloned the repo, compiled vbsam and tried to run some of the utils. I am stuck at this:

pyisam$ python3 run_wing.py
.
.
.
ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi'

Could you point me to what to do to fix this issue?

Thanks in advance

[rpmoseley]

Hi, I thank you for cloning this, I realise that I have not pushed anything in sometime. I've recently been updating my local copy.

The run_wing.py script is there to be able to debug within the WingIDE (which I assume you're doing). Otherwise, if you execute, 'python3 -m pyisam -h' you will get a list of arguments and the current tests that can be run.

The missing library 'pyisam.backend.cffi._isam_cffi' is generated by the script 'bldlibisam.py'. You can switch to using ctypes if you edit the init.py file and change the value of use_cffi to False. In order to use the 'ctypes' backend, you need to create a library named 'libpyisam' that needs the libvbisam library.

The version of the repository has it stands has a script named 'isam_build.py' within the 'utils' subdirectory which will build the necessary extension using the appropriate libraries, though, it has not yet been tested using the libvbisam library as I had an issue with it when I first started.

I will aim to push a change before the end of the year with the latest changes once I've updated my master copy.

There has been some re-organisation of the source tree, and there is now a separate configuration file. I will also push both the 32- and 64-bit original Informix (now IBM) C-ISAM libraries as well, as they are required to run programs linked to use the CISAM support.

Richard Moseley

On Fri, 29 Dec 2023, 11:28 qippur, @.***> wrote:

Hi, I know this repo is not updated since a lot, but I am trying anyway to ask.

I cloned the repo, compiled vbsam and tried to run some of the utils. I am stuck at this:

pyisam$ python3 run_wing.py . . . ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi'

Could you point me to what to do to fix this issue?

Thanks in advance

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6UJZFVZWPC7ARLUXFDRADYL2SMLAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TSMZVGU4TKOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[qippur]

Hi Richard,

Thank you for your quick response!

I will wait for your update and then try to follow your directions. I will try with both vbisam (preferred for licensing issues) and IBM libraries once they are pushed. I am more proficient in using the libraries in python than in preparing them for usage, but I'll do my best.

Thanks again, and happy new year!

roberto resecco

Il giorno ven 29 dic 2023 alle ore 17:32 Richard Moseley < @.***> ha scritto:

Hi, I thank you for cloning this, I realise that I have not pushed anything in sometime. I've recently been updating my local copy.

The run_wing.py script is there to be able to debug within the WingIDE (which I assume you're doing). Otherwise, if you execute, 'python3 -m pyisam -h' you will get a list of arguments and the current tests that can be run.

The missing library 'pyisam.backend.cffi._isam_cffi' is generated by the script 'bldlibisam.py'. You can switch to using ctypes if you edit the init.py file and change the value of use_cffi to False. In order to use the 'ctypes' backend, you need to create a library named 'libpyisam' that needs the libvbisam library.

The version of the repository has it stands has a script named 'isam_build.py' within the 'utils' subdirectory which will build the necessary extension using the appropriate libraries, though, it has not yet been tested using the libvbisam library as I had an issue with it when I first started.

I will aim to push a change before the end of the year with the latest changes once I've updated my master copy.

There has been some re-organisation of the source tree, and there is now a separate configuration file. I will also push both the 32- and 64-bit original Informix (now IBM) C-ISAM libraries as well, as they are required to run programs linked to use the CISAM support.

Richard Moseley

On Fri, 29 Dec 2023, 11:28 qippur, @.***> wrote:

Hi, I know this repo is not updated since a lot, but I am trying anyway to ask.

I cloned the repo, compiled vbsam and tried to run some of the utils. I am stuck at this:

pyisam$ python3 run_wing.py . . . ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi'

Could you point me to what to do to fix this issue?

Thanks in advance

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AB6UJZFVZWPC7ARLUXFDRADYL2SMLAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TSMZVGU4TKOA>

. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1872205200, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUKB3JPVLXPTT45YZATYL3WABAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGIYDKMRQGA . You are receiving this because you authored the thread.Message ID: @.***>

[qippur]

Sorry to bother, I cannot find the script bldlibisam.py

What am I missing?

Roberto

--

Il Ven 29 Dic 2023, 17:32 Richard Moseley @.***> ha scritto:

Hi, I thank you for cloning this, I realise that I have not pushed anything in sometime. I've recently been updating my local copy.

The run_wing.py script is there to be able to debug within the WingIDE (which I assume you're doing). Otherwise, if you execute, 'python3 -m pyisam -h' you will get a list of arguments and the current tests that can be run.

The missing library 'pyisam.backend.cffi._isam_cffi' is generated by the script 'bldlibisam.py'. You can switch to using ctypes if you edit the init.py file and change the value of use_cffi to False. In order to use the 'ctypes' backend, you need to create a library named 'libpyisam' that needs the libvbisam library.

The version of the repository has it stands has a script named 'isam_build.py' within the 'utils' subdirectory which will build the necessary extension using the appropriate libraries, though, it has not yet been tested using the libvbisam library as I had an issue with it when I first started.

I will aim to push a change before the end of the year with the latest changes once I've updated my master copy.

There has been some re-organisation of the source tree, and there is now a separate configuration file. I will also push both the 32- and 64-bit original Informix (now IBM) C-ISAM libraries as well, as they are required to run programs linked to use the CISAM support.

Richard Moseley

On Fri, 29 Dec 2023, 11:28 qippur, @.***> wrote:

Hi, I know this repo is not updated since a lot, but I am trying anyway to ask.

I cloned the repo, compiled vbsam and tried to run some of the utils. I am stuck at this:

pyisam$ python3 run_wing.py . . . ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi'

Could you point me to what to do to fix this issue?

Thanks in advance

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AB6UJZFVZWPC7ARLUXFDRADYL2SMLAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TSMZVGU4TKOA>

. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1872205200, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUKB3JPVLXPTT45YZATYL3WABAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGIYDKMRQGA . You are receiving this because you authored the thread.Message ID: @.***>

[qippur]

Hi Richard,

I created a pipenv virtual environment and I was able to build the library

$ ll ./libisam/64/libpyisam.so -rwxr-xr-x 1 roberto roberto 9064 30 dic 12.07 ./libisam/64/libpyisam.so

but it seems that some part of pyisam is missing:

$ python3 -m pyisam -h Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/roberto/devel/python/pyisam/pyisam/main.py", line 6, in

from .dumprec import dump_record_imp, dump_record_exp_eq *ModuleNotFoundError: No module named 'pyisam.dumprec'* Do you have any suggestions? roberto resecco Il giorno sab 30 dic 2023 alle ore 10:52 Roberto Resecco < ***@***.***> ha scritto: > Sorry to bother, I cannot find the script bldlibisam.py > > What am I missing? > > Roberto > > -- > > > Il Ven 29 Dic 2023, 17:32 Richard Moseley ***@***.***> ha > scritto: > >> Hi, I thank you for cloning this, I realise that I have not pushed >> anything >> in sometime. I've recently been updating my local copy. >> >> The run_wing.py script is there to be able to debug within the WingIDE >> (which I assume you're doing). Otherwise, if you execute, 'python3 -m >> pyisam -h' you will get a list of arguments and the current tests that >> can >> be run. >> >> The missing library 'pyisam.backend.cffi._isam_cffi' is generated by the >> script 'bldlibisam.py'. You can switch to using ctypes if you edit the >> __init__.py file and change the value of use_cffi to False. In order to >> use the 'ctypes' backend, you need to create a library named 'libpyisam' >> that needs the libvbisam library. >> >> The version of the repository has it stands has a script named >> 'isam_build.py' within the 'utils' subdirectory which will build the >> necessary extension using the appropriate libraries, though, it has not >> yet >> been tested using the libvbisam library as I had an issue with it when I >> first started. >> >> I will aim to push a change before the end of the year with the latest >> changes once I've updated my master copy. >> >> There has been some re-organisation of the source tree, and there is now >> a >> separate configuration file. I will also push both the 32- and 64-bit >> original Informix (now IBM) C-ISAM libraries as well, as they are >> required >> to run programs linked to use the CISAM support. >> >> Richard Moseley >> >> On Fri, 29 Dec 2023, 11:28 qippur, ***@***.***> wrote: >> >> > Hi, I know this repo is not updated since a lot, but I am trying anyway >> to >> > ask. >> > >> > I cloned the repo, compiled vbsam and tried to run some of the utils. >> > I am stuck at this: >> > >> > pyisam$ python3 run_wing.py >> > . >> > . >> > . >> > ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi' >> > >> > Could you point me to what to do to fix this issue? >> > >> > Thanks in advance >> > >> > — >> > Reply to this email directly, view it on GitHub >> > , or unsubscribe >> > < >> https://github.com/notifications/unsubscribe-auth/AB6UJZFVZWPC7ARLUXFDRADYL2SMLAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TSMZVGU4TKOA> >> >> > . >> > You are receiving this because you are subscribed to this >> thread.Message >> > ID: ***@***.***> >> > >> >> — >> Reply to this email directly, view it on GitHub >> , >> or unsubscribe >> >> . >> You are receiving this because you authored the thread.Message ID: >> ***@***.***> >> >

[rpmoseley]

Hi, I have updated the repository to put back the dumprec module in pyisam, which I had moved into the tests/ directory as part of moving away from having the tests within the main package. If you the 'python -m pyisam -h' you should get a help message giving the available tests. If you then run 'python -m pyisam -t12', and you have the sample data available under data/ you should get the following:

TEST: 12 ASSUME: TableIndex(name='comp', dups=False, desc=False, colinfo=[TableIndexCol(name='comp')]) FILL_BEF: Record_utool_dec_decomp(comp='', comptyp='', sys='', prefix='', user='', database='', release='', timeup=0, specup=0) Record_utool_dec_decomp(comp='defile', comptyp='F', sys='mx', prefix='def', user='adf', database='utool', release='Y', timeup=820246079, specup=0)

I am working on getting the support for vbisam added to the package, and will push it when ready.

However, here are the details for building the extension, though you seemed to have done this already. From a fresh repository with no _isam_cffi support under pyisam/backend/cffi, running the following command from the root directory of the repository will generate the module for the version of python you are running:

$ python utils/isam_build.py

You will then find a _isam_cffi*.so file in pyisam/backend/cffi, this should depend on libifisam.so and libifisamx.so, which should have been copied into pyisam/backend/lib. You can check the dependencies using 'ldd' or 'patchelf --print-needed' if installed.

Richard

[rpmoseley]

Hi, further to my last comment, if you wish to switch the backend to ctypes, edit the file pyisam/backend/init.py and change the variable 'conf' to "'ctypes'". This will change to be overridable with the environment variable 'PYISAM_BACKEND' once I've pushed the latest changes.

[qippur]

Thank you Richard,

the test ran successfully.

$ python3 -m pyisam -t 12 TEST: 12 ASSUME: TableIndex(name='comp', dups=False, desc=False, colinfo=[TableIndexCol(name='comp')]) FILL_BEF: Record_utool_dec_decomp(comp='', comptyp='', sys='', prefix='', user='', database='', release='', timeup=0, specup=0) Record_utool_dec_decomp(comp='defile', comptyp='F', sys='mx', prefix='def', user='adf', database='utool', release='Y', timeup=820246079, specup=0)

Then I tried to open some C-ISAM tables (attached) created with IBM C-ISAM libraries. I created a "tables" folder, copied my tables there, placed and edited the util lstisam.py, but I get this:

$ python3 tables/lstisam.py Traceback (most recent call last): File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 37, in

lstisam('cgtotst1', 'tables') File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 31, in lstisam ti, ii = fetch_isam_info(tabname, tabpath) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 17, in fetch_isam_info tabinfo = tabobj.isdictinfo() ^^^^^^^^^^^^^^^^^^^ File "/home/roberto/devel/python/pyisam/pyisam/backend/cffi/__init__.py", line 311, in isdictinfo dinfo = ffi.new('struct dictinfo *') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ffi.error: struct dictinfo: wrong size for field 'di_nrecords' (cdef says 8, but C compiler says 4). fix it or use "...;" as the last field in the cdef for struct dictinfo to make it flexible The .fld file in the tar is a field descriptor text file used by the language to map the fields in the table. As you probably understood, the underlying C stuff is not my bread and butter. I found the 2.1.1 version of vbisam HERE . It compiles a libvbisam.so.3.0.0 file. roberto Il giorno sab 30 dic 2023 alle ore 15:21 Richard Moseley < ***@***.***> ha scritto: > Hi, > I have updated the repository to put back the dumprec module in pyisam, > which I had moved into the tests/ directory as part of moving away from > having the tests within the main package. If you the 'python -m pyisam -h' > you should get a help message giving the available tests. If you then run > 'python -m pyisam -t12', and you have the sample data available under data/ > you should get the following: > > TEST: 12 > ASSUME: TableIndex(name='comp', dups=False, desc=False, > colinfo=[TableIndexCol(name='comp')]) > FILL_BEF: Record_utool_dec_decomp(comp='', comptyp='', sys='', prefix='', > user='', database='', release='', timeup=0, specup=0) > Record_utool_dec_decomp(comp='defile', comptyp='F', sys='mx', > prefix='def', user='adf', database='utool', release='Y', timeup=820246079, > specup=0) > > I am working on getting the support for vbisam added to the package, and > will push it when ready. > > However, here are the details for building the extension, though you > seemed to have done this already. From a fresh repository with no > _isam_cffi support under pyisam/backend/cffi, running the following command > from the root directory of the repository will generate the module for the > version of python you are running: > > $ python utils/isam_build.py > > You will then find a _isam_cffi*.so file in pyisam/backend/cffi, this > should depend on libifisam.so and libifisamx.so, which should have been > copied into pyisam/backend/lib. You can check the dependencies using 'ldd' > or 'patchelf --print-needed' if installed. > > Richard > > — > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[rpmoseley]

There was a problem in the 'utils/isam_build.py' where the definition of di_nrecords was long, which is 64 bits, CISAM expects an int (32 bits). I have pushed a change to fix the problem, alternately you can change the line 79 from: long di_nrecords to: {longsz} di_nrecords. (Note the brackets are required, as the type of field is changed if your running on a 32-bit architecture).

[qippur]

Hi Richard, I pulled the repo and tried to build using the script, but I get a bunch of error lines, originating with:

$ python3 utils/isam_build.py /usr/bin/ld: cannot find -lifisam: No such file or directory /usr/bin/ld: cannot find -lifisamx: No such file or directory [...]

Is there any variable I should set or anything else to be done before running the script?

BTW: from what I've seen you removed IBM isam libraries. Is it intended?

roberto

Il giorno sab 30 dic 2023 alle ore 19:01 Richard Moseley < @.***> ha scritto:

There was a problem in the 'utils/isam_build.py' where the definition of di_nrecords was long, which is 64 bits, CISAM expects an int (32 bits). I have pushed a change to fix the problem, alternately you can change the line 79 from: long di_nrecords to: {longsz} di_nrecords. (Note the brackets are required, as the type of field is changed if your running on a 32-bit architecture).

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1872575003, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUNCWY5X4H2GPRZZCO3YMBJGTAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGU3TKMBQGM . You are receiving this because you authored the thread.Message ID: @.***>

[rpmoseley]

Oops!! I managed to loose them, I have now re-added them, I left a .gitignore file that meant that they were not sent. Now fixed.

[qippur]

Hi richard, no problem.

I could compile the library.

Now I get a different error:

$ python3 tables/lstisam.py Traceback (most recent call last): File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 37, in

lstisam('cgtotst1', 'tables') File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 31, in lstisam ti, ii = fetch_isam_info(tabname, tabpath) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 22, in fetch_isam_info idxinfo.append(tabobj.iskeyinfo(idxnum+1)) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/roberto/devel/python/pyisam/pyisam/backend/cffi/__init__.py", line 323, in iskeyinfo self._chkerror(lib.iskeyinfo(self._isfd_, keyinfo, keynum+1), 'iskeyinfo') File "/home/roberto/devel/python/pyisam/pyisam/backend/cffi/__init__.py", line 199, in _chkerror raise IsamFunctionFailed(ISAM_str(args), self.iserrno, self.strerror(self.iserrno)) pyisam.error.IsamFunctionFailed: iskeyinfo: Illegal argument (102) roberto Il giorno sab 30 dic 2023 alle ore 19:45 Richard Moseley < ***@***.***> ha scritto: > Oops!! I managed to loose them, I have now re-added them, I left a > .gitignore file that meant that they were not sent. Now fixed. > > — > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[rpmoseley]

Could you send me the tar file that you mentioned, as it has been removed from the comments, my email address is:

richard.moseley4@gmail.com

I can then take a look.

First impressions would imply that you have no keys defined on the table, that is zero. Thus asking for the first key will fail with the error that you gave.

[qippur]

Hi Richard,

here you are!

Thank you.

roberto

Il giorno sab 30 dic 2023 alle ore 20:27 Richard Moseley < @.***> ha scritto:

Could you send me the tar file that you mentioned, as it has been removed from the comments, my email address is:

@.***

I can then take a look.

First impressions would imply that you have no keys defined on the table, that is zero. Thus asking for the first key will fail with the error that you gave.

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1872588690, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXULWGVMVMT5FVZVP73TYMBTJBAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGU4DQNRZGA . You are receiving this because you authored the thread.Message ID: @.***>

[rpmoseley]

Your problem is that your adding one to the index number, then the pyisam also adds one to that. Thus when using the pyisam library, the first key is 0, which maps to key 1 to the ISAM iskeyinfo() function. I will send you the source code to a program which performs an also identical action as your lstisam.py. I will send you a tarball with my version of the script, which should be placed into the root directory of the environment, the script expects the data to be under exdata, but change that to suit.

[qippur]

Thank you Richard.

Il Sab 30 Dic 2023, 21:44 Richard Moseley @.***> ha scritto:

Your problem is that your adding one to the index number, then the pyisam also adds one to that. Thus when using the pyisam library, the first key is 0, which maps to key 1 to the ISAM iskeyinfo() function. I will send you the source code to a program which performs an also identical action as your lstisam.py. I will send you a tarball with my version of the script, which should be placed into the root directory of the environment, the script expects the data to be under exdata, but change that to suit.

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1872603124, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUNT2GGB27CVXVMXOADYMB4MFAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGYYDGMJSGQ . You are receiving this because you authored the thread.Message ID: @.***>

[rpmoseley]

I have now pushed my changes which will now pull a version of pyisam which can be used within a virtualenv using the script 'mkvenv' and support both CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared libraries in the repository, so that you no longer need to compile the extension modules before use. The README.md file has been updated with more useful information. According to the README that is part of the CISAM distribution, the use of isindexinfo() is deprecated due to the fact that the second argument is a void *, and there are now two separate functions, isdictinfo() which is the same as calling isindexinfo() with a key of zero, and iskeyinfo() which is the same as calling if isindexinfo() is called with a key of non-zero. I trust this will help you continue your work.

[qippur]

Thank you Richard, I am not at home right now. I will check it as soon as I get home.

Roberto

Il Mer 3 Gen 2024, 12:48 Richard Moseley @.***> ha scritto:

I have now pushed my changes which will now pull a version of pyisam which can be used within a virtualenv using the script 'mkvenv' and support both CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared libraries in the repository, so that you no longer need to compile the extension modules before use. The README.md file has been updated with more useful information. According to the README that is part of the CISAM distribution, the use of isindexinfo() is deprecated due to the fact that the second argument is a void *, and there are now two separate functions, isdictinfo() which is the same as calling isindexinfo() with a key of zero, and iskeyinfo() which is the same as calling if isindexinfo() is called with a key of non-zero. I trust this will help you continue your work.

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1875249686, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUJ6AMHLD7I65J77XBDYMVARJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGI2DSNRYGY . You are receiving this because you authored the thread.Message ID: @.***>

[qippur]

Hi Richard,

I am trying to read the records from my test table cgtotst1. As I do not know ISAM very well I am trying to reverse engineer your python examples. It would be wonderful if I had an example of creating a new table, reading the records of an existing table and updating it (update, delete, add). I will anyway try to do it by myself.

In the meanwhile I noticed one thing: it seems that vbisam does not report the correct number of records.

I built the library twice, using ifisam and vbisam libraries. Here are the results:

python3 tables/lstisam.py tables/cgtotst1 Using cffi and ifisam library Library: IBM Informix C-ISAM Version 7.26.F Num Keys : 3 Rec Size : 70 Idx Size : 1023 Num Rows : 807

$ python3 tables/lstisam.py tables/cgtotst1 Using cffi and vbisam library Library: VBISAM 2.1.1 Num Keys : 3 Rec Size : 70 Idx Size : 1024 Num Rows : 0

As you can see, using the vbisam library 0 records are reported, while using the ifisam library the correct number of 807 records is shown. Also, the size of the index reported by vbisam is bigger then the one reported by ifisam

roberto

Il giorno mer 3 gen 2024 alle ore 12:48 Richard Moseley < @.***> ha scritto:

I have now pushed my changes which will now pull a version of pyisam which can be used within a virtualenv using the script 'mkvenv' and support both CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared libraries in the repository, so that you no longer need to compile the extension modules before use. The README.md file has been updated with more useful information. According to the README that is part of the CISAM distribution, the use of isindexinfo() is deprecated due to the fact that the second argument is a void *, and there are now two separate functions, isdictinfo() which is the same as calling isindexinfo() with a key of zero, and iskeyinfo() which is the same as calling if isindexinfo() is called with a key of non-zero. I trust this will help you continue your work.

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1875249686, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUJ6AMHLD7I65J77XBDYMVARJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGI2DSNRYGY . You are receiving this because you authored the thread.Message ID: @.***>

[rpmoseley]

Hi, I can do that, if you could let me know how the .fld file stores the field information.

I noticed that the first index on the example table has 0 key parts, this indicates that the table can be referenced using record numbers. The program 'isql' (which was a front-end for Informix, stored information about tables in tables named 'sys*', the field information was stored in 'syscolumns'.

vbisam seems to always return zero rows when you request the dictinfo, seems that there may be a speed issue, as the original code called a function, tcountrows. Looking at the comments, it would appear that the tnrows is set to 0 or 1 depending on whether or not there are any records in the table, I assume that this is because, the program using the library is expected to perform the count itself. This will need some work to correct, as I have compiled the support in and now it gives a count of 65166 against 807 that CISAM gives, there must something else going there.

As for an example, I assume that you want an example written in python, I will work on it and get back to you once finished. At the sametime I will look into what the isdictino() function of CISAM actually calls.

Richard

[rpmoseley]

Please find attached a XZ compressed version of the CIMAM reference manual original produced by Informix, it gives an overview of the various functions within the library, but no the newer isdictinfo() and iskeyinfo() functions which are simply the isindexinfo() function split into two, if you passed a key number of 0 to isindexinfo() it calls isdictinfo(), otherwise iskeyinfo(). The pyisam package provides convenience methods that do this for you.

Richard

On Thu, 4 Jan 2024 at 12:25, qippur @.***> wrote:

Hi Richard,

I am trying to read the records from my test table cgtotst1. As I do not know ISAM very well I am trying to reverse engineer your python examples. It would be wonderful if I had an example of creating a new table, reading the records of an existing table and updating it (update, delete, add). I will anyway try to do it by myself.

In the meanwhile I noticed one thing: it seems that vbisam does not report the correct number of records.

I built the library twice, using ifisam and vbisam libraries. Here are the results:

python3 tables/lstisam.py tables/cgtotst1 Using cffi and ifisam library Library: IBM Informix C-ISAM Version 7.26.F Num Keys : 3 Rec Size : 70 Idx Size : 1023 Num Rows : 807

$ python3 tables/lstisam.py tables/cgtotst1 Using cffi and vbisam library Library: VBISAM 2.1.1 Num Keys : 3 Rec Size : 70 Idx Size : 1024 Num Rows : 0

As you can see, using the vbisam library 0 records are reported, while using the ifisam library the correct number of 807 records is shown. Also, the size of the index reported by vbisam is bigger then the one reported by ifisam

roberto

Il giorno mer 3 gen 2024 alle ore 12:48 Richard Moseley < @.***> ha scritto:

I have now pushed my changes which will now pull a version of pyisam which can be used within a virtualenv using the script 'mkvenv' and support both CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared libraries in the repository, so that you no longer need to compile the extension modules before use. The README.md file has been updated with more useful information. According to the README that is part of the CISAM distribution, the use of isindexinfo() is deprecated due to the fact that the second argument is a void *, and there are now two separate functions, isdictinfo() which is the same as calling isindexinfo() with a key of zero, and iskeyinfo() which is the same as calling if isindexinfo() is called with a key of non-zero. I trust this will help you continue your work.

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1875249686, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACXMXUJ6AMHLD7I65J77XBDYMVARJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGI2DSNRYGY>

. You are receiving this because you authored the thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1877014877, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6UJZFNFOHM6QUKZXPX5SLYM2NTZAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGAYTIOBXG4 . You are receiving this because you commented.Message ID: @.***>

[qippur]

Hi Richard,

the .fld file is a proprietary file format from a niche and quite old programming language. Its current name is verastream host integrator from MicroFocus. Its original name is SuperNova. It is very powerful, and ahead of its time, but as it comes with a license fee it is not very broadly used. Therefore the format of the .fld is unknown to me, but I have received a screenshot of the fields,

[image: immagine.png]

I would like to help my good friend to have python as an alternative to that language. All his data are in C-ISAM, which is blazing fast for his use case, faster than any DBMS he tested.

roberto resecco

Il giorno ven 5 gen 2024 alle ore 13:56 Richard Moseley < @.***> ha scritto:

Hi, I can do that, if you could let me know how the .fld file stores the field information.

I noticed that the first index on the example table has 0 key parts, this indicates that the table can be referenced using record numbers. The program 'isql' (which was a front-end for Informix, stored information about tables in tables named 'sys*', the field information was stored in 'syscolumns'.

vbisam seems to always return zero rows when you request the dictinfo, seems that there may be a speed issue, as the original code called a function, tcountrows. Looking at the comments, it would appear that the tnrows is set to 0 or 1 depending on whether or not there are any records in the table, I assume that this is because, the program using the library is expected to perform the count itself. This will need some work to correct, as I have compiled the support in and now it gives a count of 65166 against 807 that CISAM gives, there must something else going there.

As for an example, I assume that you want an example written in python, I will work on it and get back to you once finished. At the sametime I will look into what the isdictino() function of CISAM actually calls.

Richard

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1878622600, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUMEFNJ5EPLC3ISAM4TYM72AXAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZYGYZDENRQGA . You are receiving this because you authored the thread.Message ID: @.***>

[qippur]

Hi Richard,

I believe the attachment has been stripped from the email.

roberto resecco

Il giorno ven 5 gen 2024 alle ore 14:06 Richard Moseley < @.***> ha scritto:

Please find attached a XZ compressed version of the CIMAM reference manual original produced by Informix, it gives an overview of the various functions within the library, but no the newer isdictinfo() and iskeyinfo() functions which are simply the isindexinfo() function split into two, if you passed a key number of 0 to isindexinfo() it calls isdictinfo(), otherwise iskeyinfo(). The pyisam package provides convenience methods that do this for you.

Richard

On Thu, 4 Jan 2024 at 12:25, qippur @.***> wrote:

Hi Richard,

I am trying to read the records from my test table cgtotst1. As I do not know ISAM very well I am trying to reverse engineer your python examples. It would be wonderful if I had an example of creating a new table, reading the records of an existing table and updating it (update, delete, add). I will anyway try to do it by myself.

In the meanwhile I noticed one thing: it seems that vbisam does not report the correct number of records.

I built the library twice, using ifisam and vbisam libraries. Here are the results:

python3 tables/lstisam.py tables/cgtotst1 Using cffi and ifisam library Library: IBM Informix C-ISAM Version 7.26.F Num Keys : 3 Rec Size : 70 Idx Size : 1023 Num Rows : 807

$ python3 tables/lstisam.py tables/cgtotst1 Using cffi and vbisam library Library: VBISAM 2.1.1 Num Keys : 3 Rec Size : 70 Idx Size : 1024 Num Rows : 0

As you can see, using the vbisam library 0 records are reported, while using the ifisam library the correct number of 807 records is shown. Also, the size of the index reported by vbisam is bigger then the one reported by ifisam

roberto

Il giorno mer 3 gen 2024 alle ore 12:48 Richard Moseley < @.***> ha scritto:

I have now pushed my changes which will now pull a version of pyisam which can be used within a virtualenv using the script 'mkvenv' and support both CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared libraries in the repository, so that you no longer need to compile the extension modules before use. The README.md file has been updated with more useful information. According to the README that is part of the CISAM distribution, the use of isindexinfo() is deprecated due to the fact that the second argument is a void *, and there are now two separate functions, isdictinfo() which is the same as calling isindexinfo() with a key of zero, and iskeyinfo() which is the same as calling if isindexinfo() is called with a key of non-zero. I trust this will help you continue your work.

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1875249686,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ACXMXUJ6AMHLD7I65J77XBDYMVARJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGI2DSNRYGY>

. You are receiving this because you authored the thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1877014877, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AB6UJZFNFOHM6QUKZXPX5SLYM2NTZAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGAYTIOBXG4>

. You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1878633537, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUJHIMAF5NNZTSHTCQLYM73DTAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZYGYZTGNJTG4 . You are receiving this because you authored the thread.Message ID: @.***>

[rpmoseley]

Roberto,

Yes the image has been removed from the comments, could you send it directly to me at richard.moseley4@gmail.com, in the meantime, I've updated my additional isdictinfo() now uses a local copy of countrows() which now returns the correct row count, which I have now pushed. I am in the process of checking the difference between 32-bit and 64-bit version of the underlying ISAM formats.

Richard

[rpmoseley]

I've now written a module called 'pyisam.tabdefns.fldfile' which will take one of the .fld files and generate the table definition from it, using a DynamicTableDefn. Since there is no field names or index information within the file, the module generates fields of the format 'fldNNNN' where the first field is numbered 1. The pyisam package will attempt to match the indexes when the first read is performed. I'm still working on a complete program that creates a new table, inserts data, updates data and deletes data, and finally removes the table again.

[qippur]

Hi Richard, did you reverse-engineer the .fld file? Wow!

roberto resecco

Il giorno gio 11 gen 2024 alle ore 17:08 Richard Moseley < @.***> ha scritto:

I've now written a module called 'pyisam.tabdefns.fldfile' which will take one of the .fld files and generate the table definition from it, using a DynamicTableDefn. Since there is no field names or index information within the file, the module generates fields of the format 'fldNNNN' where the first field is numbered 1. The pyisam package will attempt to match the indexes when the first read is performed. I'm still working on a complete program that creates a new table, inserts data, updates data and deletes data, and finally removes the table again.

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1887491428, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUJK67TAM7KQFXUW2FLYOAFARAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBXGQ4TCNBSHA . You are receiving this because you authored the thread.Message ID: @.***>

[rpmoseley]

Yes, its quite simple, each element is 4 bytes stored LSB (for example 0x08 0x00 0x00 0x00), the first such element is the number of fields that follow, then for each field the first element is the type of data, and the the second seems to be screen size, which in combination with the image of the table you gave me, allowed me guess suitable types of field. If you look at _fldtype_map in the fldfile.py module this indicates that they are not all types are mapped. It is then a simple case of adding suitable TableColumn instances to the definition. There must be somewhere else that the field and index information is stored, as there is no field name or indexes within the .fld file.

[rpmoseley]

I do apologise for not getting back sooner, but as I was writing the missing support for inserting, deleting and updating records, I've encountered an issue over the creation of the underlying record buffer, which is present in the layer above the ISAMobject which is causing me a headache.

[qippur]

Hi Richard,

No need to apologize. You are already doing so much.

Thank you

roberto resecco

Il giorno ven 9 feb 2024 alle 18:29 Richard Moseley < @.***> ha scritto:

I do apologise for not getting back sooner, but as I was writing the missing support for inserting, deleting and updating records, I've encountered an issue over the creation of the underlying record buffer, which is present in the layer above the ISAMobject which is causing me a headache.

— Reply to this email directly, view it on GitHub https://github.com/rpmoseley/pyisam/issues/1#issuecomment-1936314491, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXMXUMGOTZVCQFMURYACFDYSZMG7AVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZWGMYTINBZGE . You are receiving this because you authored the thread.Message ID: @.***>

[rpmoseley]

I have just made a push of code that ensures that the libvbisam library gets initialised correctly. Previously it was using data as left by the operating system, and happened to leave the file descriptor for the logging to zero, which when the first (or only) table was opened, the index file was opened and given the same file descriptor and hence all the logging information gets appended to the end of the index file incorrectly. The fix is to make the vbisam backend call the appropriate routine to establish the file descriptors as -1 to disable the logging by default.