Can we have separate virtualenv for some package?

[NyaMisty]

I'm going to write a plugin combining which internally uses angr for some analysis, because the angr is dropping Python 2 support, so I decided to use a package system for easier installation. However the angr is recommended to be installed in a separate virtualenv. But after going through the doc of idapkg, it seems that the whole idapkg is sharing one single virtualenv. Can we add options to somewhere like info.json in order to make some packages use their own env?

[Jinmo]

Wow, this looks challenging. Seems like there's two issues here:

On interpreter version

In fact I'm not very confident about if we can run both of python 2 & 3 in one process. Which version of the interpreter do you prefer for the analysis?

On having multiple virtualenvs

In summary, to switch virtualenvs, sys.modules and sys.path must be cleared. Normally, angr won't have z3/vex problems on idapkg.

I've read the documentation from angr after reading the issue.

Several of angr's dependencies (z3, pyvex) require libraries of native code that are forked from their originals, and if you already have libz3 or libVEX installed, you definitely don't want to overwrite the official shared objects with ours. In general, don't expect support for problems arising from installing angr outside of a virtualenv.

Sharing virtualenvs won't cause idapkg-specific problems if they do not install other z3/vex builds inside idapkg environment (angr will install their z3 build in the environment if z3-solver package which they manages is not present). But I'm interested in this issue too, so let me write some more. 🤓

First, I thought of nesting virtualenvs (inserting paths) but the packages will still collide in a interpreter because sys.modules is shared; The module itself will be same if the name is same. This can be solved by clearing z3-related(or all) items from sys.modules before importing angr, but it seems a bit hacky.

To completely solve this, separate namespaces for modules will be required, so non-angr and angr dependencies cannot interfere each other, but I think that it cannot be done in a package level, since all packages run in a same interpreter.

p.s. I saw idapkg environment as one pipenv project-like, but in this point of view, maybe idb-specific envs can be provided (but it seems hard because of the same problem).

Btw I'm sincerely hoping IDA to support running headless workers for one immutable IDB, which will solve all of these issues. If it's supported, then having separated virtualenv would be easier...

[NyaMisty]

Well I forgot to mention that I decided to use the angr's python2 branch, as I don't need the new functionalities in angr. What I want is something like https://github.com/deactivated/idaenv, which create an separate virtualenv for every plugin.

[Jinmo]

Ah, I got it. (I'm glad that idaenv is updated recently) You can configure virtualenv path in ~/idapkg/config.json. But it doesn't seem very intuitive; editing the file every time would be a little bit annoying. I'll consider changing this, like CLI or in other ways, like, switching idapkg base path(or config.json) before launching IDA.

I'll close issue after a solution is made.

[Jinmo]

In addition, can I ask if idaenv separate vitualenv for every plugins? Maybe I'm understanding it incorrectly, but I thought that it creates an environment folder and an user can install multiple plugins, and launch ida in that context.

[NyaMisty]

Seems that the idaenv update command will wrap the plugin into an env