search

E3SM-Project / E3SM

Energy Exascale Earth System Model source code. NOTE: use "maint" branches for your work. Head of master is not validated.
https://docs.e3sm.org/E3SM
Other
346 stars 353 forks source link

Required python not found with csh on titan with CIME5 #1214

Open jonbob opened 7 years ago

jonbob commented 7 years ago

The case.test and case.run files on titan use

!/usr/bin/env python

for the shebang line to set the interpreter. However, on titan when using csh/tcsh that points to Python 2.6.9 and the scripts all fail to run because the required version of Python has to be at least 2.7.

jayeshkrishna commented 7 years ago

Do you have the latest version of python loaded in .cshrc and .bashrc ?

jonbob commented 7 years ago

yes, but the tests still failed

jayeshkrishna commented 7 years ago

I just verified that with master (v1.0.0-beta-165-g32129cf) ./create_newcase works for me (does not complain about python version) on titan. I have the following in .bashrc and .tcshrc ,

module load python/2.7.9

@jgfouca : Any ideas?

jayeshkrishna commented 7 years ago

Note that I had to load the latest python on BOTH .bashrc and .tcshrc previously (even though my login shell is tcsh).

jonbob commented 7 years ago

Ah, maybe that's the issue -- I also have tcsh and was only adding to .tcshrc. I'll check on modifying .bashrc as well. But we still should request a newer version of python on titan, available under /usr/bin?

jgfouca commented 7 years ago

@jonbob did it fail immediately or when running a subprocess?

jonbob commented 7 years ago

In a subprocess -- I think then it actually launched the test with qsub

jgfouca commented 7 years ago

OK, the root of the problem is that the subprocess and/or batch jobs should inherit the environment from the parent process which does have the correct python, but apparently that's not happening.

jonbob commented 7 years ago

That's exactly right -- I printed out the version of python at different points and it was OK until the case.test script

jgfouca commented 7 years ago

@jonbob ok then this is probably a titan problem. Not enough of the interactive node's env is making it to the compute node.

jgfouca commented 7 years ago

@jonbob i'm not seeing this problem on titan. Is it resolved?

jonbob commented 7 years ago

It's still a problem for me, but could be my environment?

jgfouca commented 7 years ago

@jonbob That must be the reason but I don't see how that's possible. My login shell is bash but I don't have anything in my .bash_profile or .bashrc.

It's possible that this is all due to the fast that you're using tcsh as your login shell. Did you try the fix suggested by @jayeshkrishna ?

jgfouca commented 7 years ago

@jonbob status?

jonbob commented 7 years ago

spread thin and not having time to follow up yet....

jonbob commented 7 years ago

@jfouca - I don't even have .bash files, since I'm not using that shell. I don't think it's a reasonable solution to require that?

rljacob commented 7 years ago

You don't have to switch shells. Just add a .bashrc with "module load python/2.7.9" and put that in your .tcshrc or .cshrc too.

jonbob commented 7 years ago

That does work. Do you think that's a reasonable solution for the people who might run the model? Is there a good place to document it?

rljacob commented 7 years ago

I think this is still a bug and that's just a workaround. Try putting it on this page: https://acme-climate.atlassian.net/wiki/pages/viewpage.action?pageId=110788680

jonbob commented 7 years ago

OK, it's on that page. Do you want to close it, or leave it open since it may still be a bug? I'm fine either way.

rljacob commented 7 years ago

Leave it open. I'll change the title.

rljacob commented 7 years ago

I'd be curious to know if people who have bash as their login shell see this problem.

whannah1 commented 7 years ago

I use bash on titan, and I had to add a “module load python” statement in my bashrc to fix this.

Walter

From: Robert Jacob notifications@github.com Reply-To: ACME-Climate/ACME reply@reply.github.com Date: Friday, February 17, 2017 at 12:20 PM To: ACME-Climate/ACME ACME@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [ACME-Climate/ACME] Required python not found with csh on titan with CIME5 (#1214)

I'd be curious to know if people who have bash as their login shell see this problem.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/ACME-Climate/ACME/issues/1214#issuecomment-280756035, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJOw09Fts-RjYsodxPEfStPOnhTI9IMNks5rdgDzgaJpZM4Lgw9F.

mrnorman commented 7 years ago

Yeah, just source the module init file and module load python.

Matthew R. Norman Computational Climate Scientist Scientific Computing Group National Center for Computational Sciences Oak Ridge National Laboratory

From: Walter Hannah notifications@github.com Sent: Friday, February 17, 2017 3:22 PM To: ACME-Climate/ACME Cc: Norman, Matthew R.; Assign Subject: Re: [ACME-Climate/ACME] Required python not found with csh on titan with CIME5 (#1214)

I use bash on titan, and I had to add a "module load python" statement in my bashrc to fix this.

Walter

From: Robert Jacob notifications@github.com Reply-To: ACME-Climate/ACME reply@reply.github.com Date: Friday, February 17, 2017 at 12:20 PM To: ACME-Climate/ACME ACME@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [ACME-Climate/ACME] Required python not found with csh on titan with CIME5 (#1214)

I'd be curious to know if people who have bash as their login shell see this problem.

- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/ACME-Climate/ACME/issues/1214#issuecomment-280756035, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJOw09Fts-RjYsodxPEfStPOnhTI9IMNks5rdgDzgaJpZM4Lgw9F.

- You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/ACME-Climate/ACME/issues/1214#issuecomment-280756594, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHr58N6yuv17CyaRlpK43Hfj6uCM3ffQks5rdgGWgaJpZM4Lgw9F.

rljacob commented 7 years ago

@whannah1 but you didn't need to also create a .cshrc file right?

sarich commented 7 years ago

I use bash on titan. I have to load module python/2.7.9 from my working bash shell, but I don't have a .bashrc or .tcshrc and the tests run fine for me.

Jason

On Feb 17, 2017 4:01 PM, "Robert Jacob" notifications@github.com wrote:

@whannah1 https://github.com/whannah1 but you didn't need to also create a .cshrc file right?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ACME-Climate/ACME/issues/1214#issuecomment-280778610, or mute the thread https://github.com/notifications/unsubscribe-auth/AEs_umDjBMcYlH77ma18tdrxXrebOoHYks5rdhiugaJpZM4Lgw9F .

maltrud commented 7 years ago

i guess i'm a little surprised that tests aren't done to make things work with both bash and tcsh (and of course sh and ksh).

minxu74 commented 6 years ago

@jonbob @rljacob Shall we keep the issue open? The problem should be still there as the default python (/usr/bin/python) on Titan is very old and still python 2.6. I can ask Titan admins to see if they can make python 2.7 as default or load it automatically.