Strange interactions with subprocess.open on tcsh shell

[skelow]

Hey Ryan,

I've been trying to get automacs up and running, came back to it today. I'm getting some strange behavior on the cluster I'm running it on.

This cluster uses the tcsh shell in an executable /usr/bin/local/tcsh For a while in log-pdb2gmx after failure, I would see '/bin/bash gmx_mpi: command not found' (this version of gromacs uses the _mpi suffix). I changes all calls to subprocess.Popen to include executable = '/usr/bin/local/tcsh' and shell=False, then the program proceeded. However, now it hangs up in exec.py and in the log-pdb2gmx file, I see this strange output:

^[]0;kelows@tshpcmgtprd01:s01-protein^G[kelows@tshpcmgtprd01 s01-protein]

It seems to be passing my username and the s01-protein and other strange terminal commands, I think this is because of the interaction between subprocess but I'm not sure, any ideas?

Thanks

[skelow]

also, it hangs in line 21 of exec.py:

try: exec(compile(code_ready,filename='',mode='exec'))

It might be useful to include a setting in the configuration file that lets users specify their shell, e.g. in the diction:

shell: '/usr/bin/bash'

[bradleyrp]

Most people will recommend against shell=True because it spawns a mysterious child process, but in our case we are really doing some shell-like calls from python in several places, particularly in the supervised execution of the main script and make prep. I just switched all of the executable flags to tsch with find/sed and tested it without setting shell=True and it worked fine on a test. Could you try that and let me know if it works? I think that using the shell this way is still worth it, despite any possibly security risks. Anytime subprocess hangs inside automacs is probably from shell=False. We could remove most of the shell=False but I figured that's not necessary.

[skelow]

Alright I went ahead and made that change with sed, now when I run it I return a gmx_mpi command not found in log-pdb2gmx. This is different that the one I used to get as it used to say /usr/bin//bash gmx_mpi: command not found.

On Thu, Nov 16, 2017 at 5:49 PM, Ryan Bradley notifications@github.com wrote:

Most people will recommend against shell=True because it spawns a mysterious child process, but in our case we are really doing some shell-like calls from python in several places, particularly in the supervised execution of the main script and make prep. I just switched all of the executable flags to tsch with find/sed and tested it without setting shell=True and it worked fine on a test. Could you try that and let me know if it works? I think that using the shell this way is still worth it, despite any possibly security risks. Anytime subprocess hangs inside automacs is probably from shell=False. We could remove most of the shell=False but I figured that's not necessary.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/biophyscode/automacs/issues/3#issuecomment-345088275, or mute the thread https://github.com/notifications/unsubscribe-auth/AMNePooyjkxRsg2uH_AVIeMxwoWXbP_tks5s3LvygaJpZM4QhK9B .

-- Simon Kelow Ph.D Student - Biochemistry and Molecular Biophysics University of Pennsylvania Research Assistant - Roland Dunbrack's Lab - Fox Chase Cancer Center

[bradleyrp]

I reproduced the error by switching from bash to tcsh. You should always get a "gromacs is absent" warning from automacs before it even tries to run pdb2gmx or any other utility. The function that checks for gromacs failed because tcsh says "Command not found" while bash says "command not found". I just fixed that problem, but I think the root problem is that your path is not propagating to the child shell or gmx_mpi really isn't in the path. I'm not familiar with tcsh but in my test case it's still picking up my path correctly. Python should get the path from your current ENV regardless of the executable in subprocess. I recommend (1) making sure gmx_mpi is in the path in the same terminal before you run automacs. If you have to source it from a module then you can add it to the modules key in ~/.automacs.py or ./gromacs_config.py. (2) Otherwise I recommend using pdb.set_trace() to make sure that os.system('gmx_mpi') doesn't fail somewhere in the code. Failing that, we'll have to do some more careful debugging.

[skelow]

Awesome I'll Try to fix this problem tomorrow, thanks Ryan!

On Nov 16, 2017 8:46 PM, "Ryan Bradley" notifications@github.com wrote:

I reproduced the error by switching from bash to tcsh. You should always get a "gromacs is absent" warning from automacs before it even tries to run pdb2gmx or any other utility. The function that checks for gromacs failed because tcsh says "Command not found" while bash says "command not found". I just fixed that problem, but I think the root problem is that your path is not propagating to the child shell or gmx_mpi really isn't in the path. I'm not familiar with tcsh but in my test case it's still picking up my path correctly. Python should get the path from your current ENV regardless of the executable in subprocess. I recommend (1) making sure gmx_mpi is in the path in the same terminal before you run automacs. If you have to source it from a module then you can add it to the modules key in ~/.automacs.py or ./gromacs_config.py. (2) Otherwise I recommend using pdb.set_trace() to make sure that os.system('gmx_mpi') doesn't fail somewhere in the code. Failing that, we'll have to do some more careful debugging.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/biophyscode/automacs/issues/3#issuecomment-345120397, or mute the thread https://github.com/notifications/unsubscribe-auth/AMNePof1t7n-OFSpIo1MOSNZwozHAcCLks5s3OWBgaJpZM4QhK9B .