python and system packages defined multiple times in roles

[lerwys]

There are some python and system packages that do not really belong to any specific role, but it's needed by lots of repositories (e.g., numpy, scipy, etc).

This seems a symptom of big and generic roles. In my opinion, they should be small and self-contained. So, the dependencies (either python or system ones) should be in the same role as the application that needs them. In that regard, roles installing multiple repositories should be broken into multiple ones, and only the dependencies for that application should be installed.

Also, I don't think it's a problem to repeat the dependency in multiple roles, as long as it's needed by that role.

[carneirofc]

Personally I think we are commiting a big mistake when messing with the system python environment.

If for some reason a python application need some specific version of numpy or any other package, that concern should be set inside that application requirements.txt, or any other suitable place such as the setup.py file or a conda recipe, in an exclusive virtual environment (using conda, virtualenv or even a container).

The way It is beeing done, we are trying to micromanage too much and coupling many things without a good reason. It is as if we are build a gigantic monolith application with the OS.

[lerwys]

Yes. I agree. Managing python packages inside the applications would be better, I think.

Maybe we can start by adding a requeriments.txt to all python applications and packaging them with conda or any other package system, for instance?

What do you think @xresende , @fernandohds564 , @anacso17 ?

[fernandohds564]

I don't see a problem with this idea of managing python packages locally @lerwys and @carneirofc. I agree with it, as long as we keep it relatively simple to run our applications from the terminal and open ipython interpretrs and jupyter notebooks to use all the packages we need to make the machine studies. Maybe this will require some teaching (and pacience :smile: ) from your part.

[lerwys]

Yes. @carneirofc might know better than me, but I think it all boils down on separating the requirements of our python apps (with a requirements.txt, for instance) from how we run it (containers, virtualenv, conda, etc).

I've used virtualenv before and it's simple, but requires you to manually enter some commands prior to run your application:

virtualenv env --python python3 
source env/bin/activate
< install python packages >
< do stuff >
deactivate

Conda might be a bit more work to set it up, but it's simple to use, as well.

To me it seems that virtualenv is more appropriate for testing and developing applications and conda more suitable for deploying applications. Is that generally the case @carneirofc?

[xresende]

I guess using conda for FAC python packages as well, as we had envisaged, will take take of all these issues, right?

[carneirofc]

Yes. @carneirofc might know better than me, but I think it all boils down on separating the requirements of our python apps (with a requirements.txt, for instance) from how we run it (containers, virtualenv, conda, etc).

I've used virtualenv before and it's simple, but requires you to manually enter some commands prior to run your application:

virtualenv env --python python3 
source env/bin/activate
< install python packages >
< do stuff >
deactivate

Conda might be a bit more work to set it up, but it's simple to use, as well.

To me it seems that virtualenv is more appropriate for testing and developing applications and conda more suitable for deploying applications. Is that generally the case @carneirofc?

Both have their pros and cons. virtualenv is quicker to setup but we are still tied to the base python that is being used. We get more control over the environment using conda and I think it provides a smoother experience.