Handling python tests dependencies the same way as our generated code

[dlehavi-resonai]

Our python tests import a lot of modules, which we typically do not need in either the run or the build environment. Our current solution is to have a dedicated docket image (python test base) which we manually update, which is horrible.

After a discussion with @aviresonai we suggest to adapt the solution which we use for generated code. Namely exposing certain directories to the build environment docker where the need libraries.

This means we will need to use (see [https://stackoverflow.com/questions/2915471/install-a-python-package-into-a-different-directory-using-pip] and [https://stackoverflow.com/questions/5226311/installing-specific-package-versions-with-pip] ) pip install --install-option="--prefix=$PREFIX_PATH" package_name==

This also means that python test will have to declare which libraries they need.

[dlehavi-resonai]

from an implementation POV, the closest function to what I suggest here is -- I think link_cpp_artifacts . Only that instead of placing soft links, we have two options

• creating some libraries bucket, pip install there, and soft linking there.
• actually pip installing everything into the designated place

[itamaro]

I think the issue is more basic to any kind of builder that has a "testenv_image" / "buildenv_image" property - there's no easy way to distinguish between deps that go into the artifact of that target, or deps that need to be added to the "[build/test]env_image" in order to build/test the target.

if we solve this issue in a more robust way, I think the reported behavior will also be solved, and the suggested fix will not be necessary.

WDYT?

[dlehavi-resonai]

Here is the way I see it:

• A build-envelope is a machine, which runs a program (the build) script, to manipulate data (whatever it builds). This data is not stored on the machine, but is accessible to it.
• A run envelope is a machine which runs a program which is - in effect - the data which some build envelope built before. This program already comes installed on the machine.
• Test envelope is just a type of run envelope. There are nice concepts but now we have to hit the real world - we need a "debug environment": We need some machine(s) with debug tools to enable development. For a given program, this means that the machine will have the following
• all the libraries the program needs
• debugging tools
• The program itself (and for a compiled language, this program should be compiled with the debug symbols). Due to the way cpp linking works - even for dynamic links - our cpp solution was the following: For a given cpp gadget we build on the build environment we ,must already have all the libraries we need to link against; hence, the build environment can execute whatever it compiled, so we can use the build environment itself to run the program; all we have to do now is throw in some extra tools and we are done. When coming to interpreted languages, you don't need the libraries to copy a piece of code (and maybe for some non cpp/c/fortran dynamic links you don't need them either, I don't know), so you cannot run the program. OTOH the run environment has all the libraries because, well, we put them there. What I suggest is in effect two things:
• the bad part (which I don't think is that bad), and which I'm willing to give up on: continue the trend of unifying the build and debug environments.
• The good part: a debug environment should have access to data it needs, but it should not be part of the machine itself (this builds on the cpp trend). WDYT ?

[itamaro]

discussed, agreed on a solution that should work as long as we don't have version conflicts (pip install stuff under yabtwork, bind-mount, append to PYTHONPATH)