aws / aws-parallelcluster

AWS ParallelCluster is an AWS supported Open Source cluster management tool to deploy and manage HPC clusters in the AWS cloud.
https://github.com/aws/aws-parallelcluster
Apache License 2.0
828 stars 312 forks source link

Necessity of CFN virtualenv in `/etc/profile.d/pcluster.sh` #5390

Open nyetsche opened 1 year ago

nyetsche commented 1 year ago

This was noticed in pcluster 3.6.0 and affects earlier versions as well.

The /etc/profile.d/pcluster.sh script is created by https://github.com/aws/aws-parallelcluster-cookbook/blob/v3.6.0/cookbooks/aws-parallelcluster-install/templates/default/base/pcluster.sh.erb#L6 ; it points to a virtualenv with various cfn- scripts used for updates, install, etc. All fine.

But, everything in /etc/profile.d/* is sourced by interactive shells; regular users get this added to their $PATH, ex:

[ec2-user@pcluster ~]$ whoami
ec2-user
[ec2-user@pcluster ~]$ echo $PATH
/opt/amazon/openmpi/bin/:/opt/amazon/efa/bin/:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/bin:/opt/aws/bin:/opt/parallelcluster/pyenv/versions/3.7.16/envs/cfn_bootstrap_virtualenv/bin:/opt/slurm/bin

/usr/bin, /usr/local/bin and so on are all before /opt/parallelcluster/pyenv/versions/3.7.16/envs/cfn_bootstrap_virtualenv/bin; so for the most part this is no big deal. But at least on RHEL8, /usr/bin/pip no longer exists (choose between /usr/bin/pip3 or /usr/bin/pip2). Some of my users are typing pip and getting sent to the copy in the virtualenv.

Is it required that the CFN virtualenv be set in $PATH for all users ? The user-data startup script sources it:

https://github.com/aws/aws-parallelcluster/blob/v3.6.0/cli/src/pcluster/resources/head_node/user_data.sh#L104

and the CFN templates:

https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/templates/cluster_stack.py#L1141 https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/templates/cluster_stack.py#L1232

But a source'd file anywhere on disk would be sufficient, it doesn't have to be in /etc/profile.d/.

davprat commented 1 year ago

Thank you for the report. I have added your request to our feature backlog.

enrico-usai commented 4 months ago

Patch for this: https://github.com/aws/aws-parallelcluster-cookbook/pull/2729

enrico-usai commented 4 months ago

Another issue related to this: https://github.com/aws/aws-parallelcluster-cookbook/pull/2499