Best practices for read only QHOME

[cmccarthy1]

Questionnaire removed as query unrelated to specific version of PyQ or a specific bug in the code.

Context: The following issue was raised via internal channels and is a high priority item for the client.

Query: When we install the package to a conda environment, it creates a q/ directory within that environment. Afterwards, we need to copy files from our QHOME to that q/ directory for pyq to work. Can the conda package add all the necessary files during installation, so we won’t need to copy additional files afterwards?

Related question, I’ve generally had one QHOME, a read-only central location, and I’ve not found a good way to install pyq without building in pyq to QHOME. Is there a clean way to install pyq that doesn’t involve adding files to QHOME?

In my case, the setup.py wants to put files into QHOME, but some other user owns it, and it is read only to me.

We have a central QHOME installation, and if I put pyq in there, it would mean that all users have to settle on using the same pyq (and the same python too, as pyq-config.q hardcodes the libpython path). Is this avoidable?

I just today experimented with this as my pyq-config.q and it seems to work… but I’m not sure if it’s a good idea in the long term. This does away with the hardcoded python path at least…

Some of this I lifted from what setup.py does to create the hard-coded pyq-config.q.

$ cat pyq-config.q

\d .pyq

python_dll:raze[system "python -c 'import sys;from distutils import sysconfig; print(\"{}/libpython{}.{}m.so\".format(sysconfig.get_config_var(\"LIBDIR\"), *sys.version_info[:2]))'"],"\000";

pyq_executable:raze[system "which pyq"];

In order to run pyq in a way that was decoupled from the q QHOME installatio, I have had to make code changes to python.q (\l p.k from an absolute path instead of relying on the limited flexibility of how to influence where \l looks in absence of an absolute path).

I also had to make some changes to pyq.c to influence where it finds python.q. This line is what I had to change https://github.com/KxSystems/pyq/blob/master/src/pyq.c#L357.

Doing it this way (making some code changes) allows me to have different versions of pyq but only 1 QHOME (with no built-in pyq.so, python.q, pyq-*.q, etc.). However, it makes upgrading somewhat painful as I would have to go and make all these changes to each version (I have a makefile doing some kludgy things with sed to change some paths when building pyq).

I’d love to know if there is an accepted and cleaner way to handle the situation where users don’t have access to change QHOME, but still want to be able to install pyq into their custom conda/venv environments (with the assumption that they can’t change the Q/kdb QHOME deployment).

[sashkab]

We have a central QHOME installation, and if I put pyq in there, it would mean that all users have to settle on using the same pyq (and the same python too, as pyq-config.q hardcodes the libpython path). Is this avoidable?

Did you consider using symlinks from $QHOME into $VIRTUAL_ENV/q to avoid need to copying anything around? We considered that $QHOME in $VIRTUAL_ENV/q is the best option to avoid conflicts between python and kdb versions. i.e each virtual environment is kind of hard-coded combination of Python and kdb+ versions.

Will leave to @abalkin to answer other questions.

[github-actions[bot]]

Stale issue message