VTModule VTTable can't import

[fzakaria]

Why can't I import these types? It's a bit confusing from the docstrings.

I would like to use them as actual type things.

>>> from apsw import VTModule
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'VTModule' from 'apsw' (/nix/store/a06jcqgr0bc0358w7b26kisrg05sg5rw-python3-3.10.10-env/lib/python3.10/site-packages/apsw/__init__.cpython-310-x86_64-linux-gnu.so)

[rogerbinns]

That is the expected behaviour. Can you clarify what you are trying to do with the types? There may be a better way of meeting your goals.

The documentation does say:

Note There is no actual VTModule class - it is shown this way for documentation convenience and is present as a typing protocol.

The nitty gritty details are that this typing information is provided by type stubs - separate files from the extension. All of the type checking tools, IDEs etc understand them and work with them. You can read more about stubs and see the APSW stubs here.

It would also never make sense to have an actual VTModule type - ie something you could call isinstance on. Duck typing is used - ie APSW does the equivalent of a getattr for the 3 methods (eg "Create" or "Connect") and calls those.

[fzakaria]

I see -- I was reading the "protocol" link in the docs and it said you can also inherit from the types if you wanted. I misunderstood how to use this and thought they should be importable since it's a Python file.

Sorry for the churn here.

[rogerbinns]

I always welcome a chance to improve the docs. The important distinction here is that APSW is not a Python file! It is C code. There are a lot of restrictions on C code that are not present for regular Python code - one of the reasons type stubs exist.

Have you tried using make_virtual_module to make an initial virtual table implementation?

[fzakaria]

Appreciate the feedback!

I am doing a bit of an advanced usecase which you can see here https://github.com/fzakaria/sqlelf where that method is not powerful enough. In fact, I find wanting to make multiple tables a bit cumbersome since they each need a Cursor etc..

I found I had to add: from __future__ import annotations Otherwise Python was complaining about the use of some of the types in the stub such as SQLValue

[fzakaria]

❯ sqlelf /usr/bin/ruby /bin/ls
Traceback (most recent call last):
  File "/usr/local/google/home/fmzakari/code/github.com/fzakaria/sqlelf/.devenv/state/venv/bin/sqlelf", line 5, in <module>
    from sqlelf.cli import start
  File "/usr/local/google/home/fmzakari/code/github.com/fzakaria/sqlelf/sqlelf/cli.py", line 9, in <module>
    from .elf import header
  File "/usr/local/google/home/fmzakari/code/github.com/fzakaria/sqlelf/sqlelf/elf/header.py", line 55, in <module>
    class Cursor(object):
  File "/usr/local/google/home/fmzakari/code/github.com/fzakaria/sqlelf/sqlelf/elf/header.py", line 77, in Cursor
    def Column(self, number: int) -> apsw.SQLiteValue:
AttributeError: Unknown apsw attribute 'SQLiteValue'

interestingly my static checker passes though.

[rogerbinns]

You do indeed need the future annotations thing, otherwise you fall foul of the type stubs not being known to CPython and its eager evaluation. This stuff is slowly but surely improving with each Python version.

What you currently have in sqlelf is a perfect fit for make_virtual_module! The example shows doing roughly the same thing, but the inner loop calls os.stat instead of returning elf information as in your case. However if you are going to allow updates then you will have to do the whole virtual table thing yourself. Do file issues or the post to the mailing list if there is anything that can make it easier.

I did check to see if any existing tools could make sqlelf easier, but sadly they require parsing of their arbitrary output (eg readelf), or are incomplete (eg pyelftools).

[fzakaria]

Thank you so much for the clarification and edification. I will give that method a shot.

[fzakaria]

@rogerbinns here is my much smaller implementation:

def elf_headers(binaries):
    def generator():
        for binary in binaries:
            yield {
                "path": binary.name,
                "type": binary.header.file_type.value,
                "machine": binary.header.machine_type.value,
                "version": binary.header.identity_version.value,
                "entry": binary.header.entrypoint,
            }

    return generator

def register(connection, binaries):
    generator = elf_headers(binaries)
    # setup columns and access by providing an example of the first entry returned
    generator.columns, generator.column_access = apsw.ext.get_column_names(
        next(generator())
    )
    apsw.ext.make_virtual_module(connection, "elf_header2", generator)

[fzakaria]

One downside to this implementation is that they are eponymous and do not show up when i do .tables in the CLI

[rogerbinns]

Note that they are not eponymous only - ie you can still create a virtual table USING them. If you are going to be doing joins and other fun stuff that results in the tables being rescanned, be aware that you can get SQLite to cache the data for you.

CREATE TABLE temp.headers AS SELECT * FROM elf_header2;

SELECT headers.foo FROM .... WHERE headers.field = ...

There will now be what appears as a regular table named headers that has the content.