search

PawseySC / maali

A build system by the Pawsey Supercomputing Centre
BSD 3-Clause "New" or "Revised" License
1 stars 7 forks source link

Updates Python module loading process #116

Closed dipietrantonio closed 4 years ago

dipietrantonio commented 4 years ago

Modifies maali such that it does not load a Python interpreter when one is not already loaded, but throws an error instead. This is avoid users relying on a default setting that may change in the future. The user must explicitly load a proper Python interpreter before loading a Python module.

dipietrantonio commented 4 years ago

As per last supercomputing meetings, we would like this behavior to persist even after the transition to Python 3, as if ever we ended up transitioning to a Python4, we would not have the same problems. I have implemented the switch as you suggested, but we would like it to be enabled on all Pawsey systems. I have thought about this, and the reason behind this conclusion is ambiguity. Lazy loading works well when prerequisites are well defined and unambiguous. This is the case with compiled software. During compilation, we specify what are the prerequisites, the program links against them and they are the only options to be used later during runtime. But with interpreted languages, it is like, or we can see as, compilation and execution happening at the same time. Before interpretation starts, for example, both numpy/1.13.0-python2 and numpy/1.13.0-python3 are possible valid options. Because they are indeed two different numpy, that you select not through its version number, but through the python interpreter. It is indeed a particular case but I think it is more correct to consider it in this way. Lazy loading should automate an unambiguous task and save users some time and some likes of code in a script, but not to lead to possible different behavior if used. As an example, assume casacore/3.2.1 was built with boost/1.66.0. Regardless of the currently loaded version of boost is loaded, casacore/3.2.1 will load boost/1.66.0 because it can only work with that one. But every time a user wants to use numpy, it may be able to decide which interpreter to run with (but not always, for example, numpy/1.19 requires python3, and this translates into a conflict clause with python2). It is like when you write a cygnet file and you want to specify version numbers for every prerequisite. For interpreted languages, the execution script is the same as the build script.

I am happy to continue this discussion with you

cheers Cristian

marcodelapierre commented 4 years ago

I completely agree with this being enabled across all systems on a continuining basis, as I 100% agree with your points (I remembered them from our previous chat but...was "lazy" in citing them ;) ).

I would just like the switch to be made available in the maali source...we may even have it turned ON by default, rather than off. ;-)

...so, my updated, simplified request:

In MAALI around line 2179:

MAALI_PYTHON_DEFAULT_LOAD=${MAALI_PYTHON_DEFAULT_LOAD:-1}

Then, condition those 2 modulefile prints to when the variable value is 1.

dipietrantonio commented 4 years ago

I have left this new behaviour off by default but enabled it on all our systems in the installation scripts. Please check if I did it correctly. Glad to see we see it the same way :)

marcodelapierre commented 4 years ago

Let's go @Halolegend94 !

I have created maali version 1.8.2. Please go ahead with installation an all systems in conjunction with the python module updates (maybe make a note in the corresponding CR).