Transport should ensure that command are executed by /bin/sh under the "C" locale

[GoogleCodeExporter]

Use any of the `Transport` classes when the default shell is set to
come C-shell variant and the locale is a non-English one, may result
in:

* non-English messages being printed by the remote commands, which messes up 
our parsing routines.
* command syntax being wrongly interpreted.

The `Transport` class and its descendants should ensure that:

1) remote commands are executed by `/bin/sh`
2) the remote environment has the locale set to "POSIX" (`export LC_ALL=POSIX` 
in sh syntax)

Original issue reported on code.google.com by riccardo.murri@gmail.com on 22 Aug 2012 at 1:54

[GoogleCodeExporter]

I think the safest way to do this is by creating a file on the remote end 
containing the command to be executed and run `/bin/sh scriptfile`

Parsing a command line and escaping what has to be escaped in order to run "sh 
-c ..." could be error prone, we probably need to escape every special char 
like $ \ " and '

However, in order to do that we would probably need to run a "mktemp" in the 
remote end, put in it the command and remember to cleanup everything, either 
the command succeeded or not...

Original comment by arcimbo...@gmail.com on 23 Aug 2012 at 11:38

[GoogleCodeExporter]

| I think the safest way to do this is by creating a file on the remote end
| containing the command to be executed and run `/bin/sh scriptfile`
|
| Parsing a command line and escaping what has to be escaped in order to run
| "sh -c ..." could be error prone, we probably need to escape every special
| char like $ \ " and '

how about transport issues a 'exec /bin/sh' as first command, just
after the connection is opened?

That would ensure that all subsequent commands are executed by a
Bourne-type shell.

Original comment by riccardo.murri@gmail.com on 23 Aug 2012 at 12:15

[GoogleCodeExporter]

The following is took from the discussion in the gc3pie-dev mailing list.

Antonio Messina:

It does not seems to work, every time you call exec_command() a new
channel is opened, and a new command is executed by the ssh server.
An alternative could be to open an interactive shell on the server,
and use stdin to feed the commands, but we may have problems in
getting the output and the exit code: we can only communicate reading
standard output and errors, and in order to know if the program
finished writing in its standard output we have to set a timeout
otherwise the read() will hang forever, but this will break if the
program is slower than the timeout we have set.

=================================================
Riccardo Murri riccardo.murri@gmail.com Aug 23

On Thu, Aug 23, 2012 at 3:11 PM, Antonio Messina <arcimboldo@gmail.com> wrote:
| On Thu, Aug 23, 2012 at 2:15 PM,  <gc3pie@googlecode.com> wrote:
|| how about transport issues a 'exec /bin/sh' as first command, just
|| after the connection is opened?
|
| It does not seems to work, every time you call exec_command() a new
| channel is opened, and a new command is executed by the ssh server.

Is there an option/environment variable to teel Paramiko what shell to
use on the remote side?

| An alternative could be to open an interactive shell on the server,
| and use stdin to feed the commands, but we may have problems in
| getting the output and the exit code: we can only communicate reading
| standard output and errors, and in order to know if the program
| finished writing in its standard output we have to set a timeout
| otherwise the read() will hang forever, but this will break if the
| program is slower than the timeout we have set.

No, this definitely should be avoided; better run risks connected with
/bin/sh not being the default than having to use a timeout with every
command...

Ciao,
R

=================================================
Antonio Messina Aug 23

On Thu, Aug 23, 2012 at 3:49 PM, Riccardo Murri
<riccardo.murri@gmail.com> wrote:
> On Thu, Aug 23, 2012 at 3:11 PM, Antonio Messina <arcimboldo@gmail.com> wrote:
> | On Thu, Aug 23, 2012 at 2:15 PM,  <gc3pie@googlecode.com> wrote:
> || how about transport issues a 'exec /bin/sh' as first command, just
> || after the connection is opened?
> |
> | It does not seems to work, every time you call exec_command() a new
> | channel is opened, and a new command is executed by the ssh server.
>
> Is there an option/environment variable to teel Paramiko what shell to
> use on the remote side?

As far as I can tell, no.

Paramiko call a paramiko.transport.open_session() which will open a
channel. A channel has a exec_command() method which only accept an
argument: `command`, and then send to the ssh server the command
"exec" followed by the command. I think that this is not possible per
RFC[1], you can only ask the server to "exec" a command or run an
interactive shell.

[1] http://tools.ietf.org/html/rfc4254

My proposal is to leave it as it is now. It is always easier to modify
the application to run "/bin/sh -c "... instead of having the
transport doing this.

=================================================
Riccardo Murri riccardo.murri@gmail.com Aug 23

On Thu, Aug 23, 2012 at 4:19 PM, Antonio Messina <arcimboldo@gmail.com> wrote:
| My proposal is to leave it as it is now. It is always easier to modify
| the application to run "/bin/sh -c "... instead of having the
| transport doing this.

That's only half of the problem.  The other one is that we are issuing
a number of batch system commands through Transport, and sh-syntax
might get intermixed with that.  Example: we do 'mkdir -p ... && cd
... && qsub ...".  It's likely that the syntax we use is simple enough
to work with both Bourne-type shells and C-type shells, but nobody has
ever checked and ambiguity is really calling for subtle bugs to happen
here.

I understand that issuing a new "exec()" call for every command also
rules out setting LC_ALL="POSIX" with a single command.

Apparently we cannot fix this issue.  Let's resume the discussion in
the ticket body for completeness, but leave it open for now...

Original comment by arcimbo...@gmail.com on 29 Aug 2012 at 5:21

[GoogleCodeExporter]

Still, the only option we have is to *wrap* the command into a shell script and 
execute "sh -c <shellscript>".

Arguments against this solution (please add more):

1) we need to execute 2 or 3 command in the remote end for each 
`Transport.exececute_command()` call

Arguments in favor of this solution (please add more):

1) it may solve the issue
2) we can have different scripts to use with different resources, configurable 
via `gc3pie.conf` (use cases for this?)

Original comment by arcimbo...@gmail.com on 29 Aug 2012 at 5:28