regexp works on all platforms except Termux

[eliranwong]

May you please help? I have codes like:

def regexp(expr, item): reg = re.compile(expr, flags=0 if config.enableCaseSensitiveSearch else re.IGNORECASE) return reg.search(item) is not None

connection = apsw.Connection(self.database) connection.createscalarfunction("REGEXP", regexp)

With the codes above in place I can run the following query on Windows, macOS, Linux, but no Termux

SELECT * FROM table WHERE text REGEXP ?

It returns nothing on Termux.

Can you give me some idea how it may also work on Termux too?

Did I need to install any packages to get it working?

[rogerbinns]

I know of no reason APSW or SQLite will behave differently under termux. You should run the APSW test suite to check everything is ok:

python3 -m apsw.tests

If a regexp function is not defined you get an error (even for an empty table), so your regexp function will definitely be used. You'll likely need to look deeper into that.

[eliranwong]

If a regexp function is not defined you get an error (even for an empty table), so your regexp function will definitely be used. You'll likely need to look deeper into that.

I do define it. I use the exactly same codes and same databases on the same project on Termux, macOS, Linux, Windows. I got empty result only on Termux. Apsw works great on platforms non-Termux platforms.

Thanks for reply anyway.

[rogerbinns]

I do define it.

Sorry for being unclear. That was to show your function is definitely being called, so you do need to verify it is giving the results you expect.

What did running the apsw test suite show?

[eliranwong]

I appreciate your help. Running ' python3 -m apsw.tests' gives the following results:

Python /data/data/com.termux/files/usr/bin/python3 sys.version_info(major=3, minor=10, micro=7, releaselevel='final', serial=0) Testing with APSW file /data/data/com.termux/files/usr/lib/python3.10/site-packages/apsw/init.cpython-310.so APSW version 3.39.3.0 SQLite lib version 3.39.3 SQLite headers version 3039003 Using amalgamation True Not doing LoadExtension test. You need to compile the extension first

python3 setup.py build_test_extension ................................................................................................

Ran 96 tests in 66.878s

OK

[rogerbinns]

That shows APSW and the SQLite embedded inside it are working fine. (They do a lot of testing of functions too.)

Usually this kind of issue shows up when the database files actually being operated on aren't the files the developer thinks are being operated on, especially because database file names are relative to the current working directory, which can get murky in environments like termux.

You should get the SQLite diagnostics when running your code as that may give more clues - see the documentation

[rogerbinns]

@eliranwong do you have any further updates? Specifically any indication of where the problem is, or a reproducer so any problem in APSW can be tracked down?

[eliranwong]

@rogerbinns thanks for asking, but I have no idea yet. I tried many different things. I tried with same codes, database, query statement, etc. All platforms work, except Termux. I am sure the right database is connected, because when I search without regular expression on Termux, it works as expected, the exception is only when I use regular expression.

[rogerbinns]

@eliranwong CPython uses its own C code for the regex engine, so that is unlikely to be an issue. However it uses case functions that are long since deprecated with one example of problems.

My guess would be problems with case insensitive stuff not working well in termux. (The python build process should also include a whole bunch of unicode data tables with take up a lot of space, which may not be happening.)

I'm going to close this issue since I don't see anything APSW itself is doing - the same code is used for all platforms for function calls. Please don't hesitate to reopen if you have something reproducible I can work with.

[eliranwong]

Hi @rogerbinns may I bring to your attention that searching regular expression with apsw doesn't work on some GNU/Linux, please read https://github.com/termux/termux-packages/issues/12340#issuecomment-1280002917

[xtkoba]

Some remarks. This happens for me on Gentoo GNU/Linux when installed through pip from source (without using any binary packages; with option --no-binary :all:). Does NOT happen when installed through Portage (emerge dev-python/apsw) which uses system SQLite.

[rogerbinns]

@xtoba I just installed gentoo (AMD64) and tried. I got identical behaviour (lots of listed items):

• pip no-binary
• pip binary (uses manylinux_2_17 image)
• emerge dev-python/apsw

[rogerbinns]

@eliranwong have you tried doing a case sensitive search? If you make the "j" in your regex upper case and leave out the case insensitive flag then it should find matches. (I did try to get termux going yesterday but was ultimately thwarted.)

[eliranwong]

@rogerbinns I tried both case sensitive and insensitive searches, results are the same. Working on Arch Linux, Debian and CentOS but not on Termux.

Bty, I also had hard time getting Termux installed. Finally, I managed to get it running and put some notes at https://github.com/eliranwong/UniqueBible/wiki/Install-Termux-on-Android#install-termux-app

[eliranwong]

@rogerbinns an update. I installed the Termux package python-apsw @xtkoba created. It works now. Thank you very much.

[rogerbinns]

@eliranwong glad to hear things are working for you now, although I would have liked to find the root cause. It wasn't unicode case conversion tables then.

My termux issues were because I was using the Play Store version on ChromeOS. That is unsupported, and I almost got it working, since all I needed was Python. (Installing an APK not from the Play Store requires wiping the device to put it into developer mode which I did not do.)