Suggestion: separate the package to two

[co-dh]

Hello: We are in a situation that the q is installed in a place that we don't have write access. I'm wondering if we can split the pyq package into 2 parts: those installed to q, and those into virtual environment.

Regards

[sashkab]

You don't need to. You can specify QHOME environmental variable, which will point to the q installation path. This will allow to install all necessary files into QHOME, and Python module will be installed in the virtual environment. Afterwards, you need to ensure that QHOME variable is properly set in order to use PyQ.

Please note, the drawback of this method — you won't able to share your q installation with different version of Python.

linux, macOS:

export QHOME=path/to/q
pip install pyq

windows

set QHOME=path\to\q
pip install pyq

We are in a situation that the q is installed in a place that we don't have write access.

You will need to get a write access for this to work.

[co-dh]

@sashkab : Thank you for your fast response!

As I said, we don't have write access to the q installation path directory. I got error: could not create '..../pyq-config.q' : Permission denied.

We might be able to copy the q/ into virtualenv, but that's not idea.

[sashkab]

Copying q into your virtualenv would be the best and simplest solution.

Meanwhile, here what you can try to do — this will require someone with the write permissions for your QHOME:

1. Create new virtualenv:

   virtualenv -p path/to/python pyq
   source pyq/bin/activate

2. Create q directory:

   mkdir ${VIRTUAL_ENV}/q

3. Copy q.k:

   cp path/to/q.k ${VIRTUAL_ENV}/q/q.k

4. Install PyQ:

   pip install pyq

5. Afterwards, you can copy all content from the ${VIRTUAL_ENV}/q into your read-only location (you should skip copying q.k) -- (ask your admins, or whoever has write permissions).

[co-dh]

@sashkab Thank you! May I ask what's special about q.k that it need to be copied first in step 3 and ignored in 5? We are talking about deploying this to many machines, and new machines in future. Asking sys admin to do manual process might not be an option.

As for copy q into virtualenv, I'm preparing a package for many other programmers, and also for integration test. Coping q around is not a good idea.

It would be better to separate the install_qlib and install_qext commands out of the setup.py to another package.

[sashkab]

May I ask what's special about q.k that it need to be copied first in step 3 and ignored in 5?

We are checking which version of the kdb+ you have: kdb+ 2.x or 3.x.

It would be better to separate the install_qlib and install_qext commands out of the setup.py to another package.

You can try running these tasks separately:

python setup.py install_qlib
python setup.py install_qext

But there is no guaranty you will end up with working PyQ installation afterwards, sorry.

[abalkin]

May I ask what's special about q.k that it need to be copied first in step 3 and ignored in 5?

The installer looks inside q.k to figure out the version of kdb+. Some compiler flags are different for older kdb+ versions. You can supress this behavior by specifying the version in your own setup.cfg file. Run python setup.py config in the git clone for details.

It would be better to separate the install_qlib and install_qext commands out of the setup.py to another package.

If you are installing from source, you should be able to run any combination of install_* commands to install the components separately.

If you plan to deploy to different machines, I would recommend that you create custom wheels.

[abalkin]

Coping q around is not a good idea.

To the contrary, copying the q directory to $VIRTUAL_ENV/q is the best and the recommended way to install pyq. This gives you an isolated environment where you control the versions of python and kdb+. Core kdb+ installation is tiny: a few hundreds KB.

We've experimented with different installation schemas in the past including some similar you what you propose, but concluded that having a dedicated kdb+ installed in a python virtual environment is the best solution.

[co-dh]

@abalkin @sashkab Thank you both very much!