Have separate python environments for hark-related python code and external python code

[krrome]

Is your feature request related to a problem? Please describe. In order to generalise it will be necessary for users to bring their own requirements, python versions etc.

An example - at the moment when you setup a project and you do:

poetry new --src hark-rle
cd hark-rle
poetry add hark-lang

Then you try to install your custom package which requires e.g.: "pyyaml==5.1.1" for whatever reason then poetry add ./src/ext/my_other_package will fail with:

[SolverProblemError]
Because my_other_package (0.1) depends on PyYAML (5.1.1)
 and hark-lang (0.5.1) depends on pyyaml (>=5.3.1,<6.0.0), my_other_package (0.1) is incompatible with hark-lang (0.5.1).
And because no versions of hark-lang match >0.5.1,<0.6.0, my_other_package (0.1) is incompatible with hark-lang (>=0.5.1,<0.6.0).
So, because rtbx-auto depends on both hark-lang (^0.5.1) and my_other_package (0.1 src/ext/my_other_package), version solving failed.

Describe the solution you'd like Lose the restrictions by hark-lang requirements

Describe alternatives you've considered To make the hark-lang runtime environment independent from the user's runtime environment it will be required to call user defined python functions "outside" of the hark-lang python process. Some options that come to mind:

• Put the hark-lang in a different environment than the user's code. Then execute python functions in the scope of user's code using the python executable that was defined in the user's environment.
• Alternatively, move hark-lang to a different interpreted language which is not considered a possible "user's language", or have hark as a compiled package

[rmhsilva]

This is a great request. There are a couple of ways to do it, as you say. I'll have a think. One (maybe easy) way that comes to mind is to change the PYTHONPATH when user Python modules are imported - _load_module in machine/foreign.py. I think this is what your first option is suggesting.

[krrome]

I think the practical problem is actually bigger than how to load user libraries in machine... If there are version conflicts then poetry add hark-lang during project setup will already stop the user from progressing. I'm putting down what I think the options are when keeping hark implemented in python:

1. Using separate environments

   • The poetry project with user code does not require hark to be installed
   • Have a separate virtualenv / conda env with hark-lang installed which is used to administrate everything hark-related
   • Instead of running user's python code inside the hark python process, have a completely independent python process executing the user code (maybe also different python executable e.g. python 3.5).
   • To integrate user function calls seamlessly hark-lang could offer "adapters" (only with dependencies in the standard lib!) for a set of python versions which just deserialise arguments to the function call and serialise return values in e.g. JSON - assuming that hark-lang only support primitive function arguments and return values. This of course would require to have e.g. a python 3.8 (for hark) and a python x.y (for the user function available)
   • To avoid having to have 2 versions of python available on one Lambda the aforementioned "adapter" + user code could be a Lambda function itself and the adapter would have to have the capability of returning results and triggering the parent process on return. All this still only by using standard lib functionality for that python version - otherwise there are restrictions again on which versions the user can use.

2. Using the same environment for hark and for the user code and executing user code inside the hark python process

   • hark would have to be available in all "hark-supported" python versions
   • hark would have to lose as many non-standard library packages a possible - every single non-standard library package has potentially conflicting versions with the user's code. Even though there are tricks to reload a different version of a package during runtime and then switching back, I think this approach is very tricky to keep clean and is ultimately less general than approach 1
   • This approach does not solve the problem of package and python version conflicts when setting up the project in e.g. poetry, which from a user's perspective is the biggest hurdle.