Difficulty to put destination filename (redirection) in the $ITEM

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Generate this data file
./try.pl -prec=6 -threshold=0.3 f1 > f1.res
./try.pl -prec=6 -threshold=0.3 f2 > f2.res

2. Generate this perl script try.pl
use strict;

die unless @ARGV>=1;

print "try: parameters are ".join(" ", @ARGV)."\n";

3. Run ppss
ppss -f todo  -c '$ITEM'

What is the expected output? What do you see instead?
I'd like that each line of the todo file would be executing, the shell
sending each parameter to the perl script and the output being redirected
to the given filename.

Instead, I get, for one of the files:
===== PPSS Item Log File =====
Host:           tangerine
Process:        7020
Item:           ./try.pl -prec=6 -threshold=0.3 f1 > f1.res
Start date:     mai 10 18:28:17

try: parameters are f1/>/f1.res

Status:         SUCCESS
mai 10 18:28:17:  Total processing time: 00:00:00

What version of the product are you using? On what operating system?
Distributed Parallel Processing Shell Script version 2.62
Mandriva Linux 2010

Please provide any additional information below.
I have (due to dependencies I cannot change) to forge the input and output
filenames before runnning the perl script and to put them in the commands
list

Original issue reported on code.google.com by kle...@gmail.com on 10 May 2010 at 4:33

[GoogleCodeExporter]

Hi, interesting issue.

Would this solve your problem? Data file contains: 

f1
f2 
f3

Then run this command: 

./ppss -f todo -c './try.pl -prec=6 -treshold=0.3 "$ITEM" > "$ITEM".res'

This works exactly as you described under the 'expected output' section.

{{{
===== PPSS Item Log File =====
Host:       MacBoek.local
Process:    53209
Item:       f3
Start date: mei 10 22:43:48

Status:     SUCCESS
mei 10 22:43:48:  Total processing time (hh:mm:ss): 00:00:00
}}}

bash-3.2$ cat f1.res 
try: parameters are -prec=6 -treshold=0.3 f1

Original comment by Louwrentius on 10 May 2010 at 8:48

[GoogleCodeExporter]

Yes, it works for this case, but I have a second case where the parameters 
would be:
try.pl  -prec=6 -threshold=0.3 file1 file2 > file1.file2.res
I thus have to put several values used to forge the parameters and the 
destination
file name in the items file.

I don't understand why putting (as I tried) "./try.pl -prec=6 -treshold=0.3 f1 >
f1.res" in the items file and calling 
ppss -f todo -c 'bash "$ITEM"
does not work ?

Original comment by kle...@gmail.com on 11 May 2010 at 9:11

[GoogleCodeExporter]

OK. I found how to do. I put:
/path/to/script.pl -prec=6 -treshold=0.3 f1 > f1.res
in the items list file and I run ppss with:
ppss -f todo -c 'bash -c "$ITEM"'

So, with the "-c" parameter passed to bash.

Thanks for your help.

Original comment by kle...@gmail.com on 11 May 2010 at 9:25

[GoogleCodeExporter]

There is still a problem: it puts along my result file (f1.res) two empty files
named " f1.res" and "_f1.res".

Also, if I pass files with long paths, the log file name generated by ppss is 
too
long for bash. To avoid the problem I changed the following in ppss (around line
1709):
    #
    # Create the log file containing the output of the command.
    #
    LOG_FILE_NAME=`echo "$ITEM" | sed s/^\\\.//g | sed s/^\\\.\\\.//g | sed
s/\\\///g | sed s/\\ /_/g`
    if [[ ${#LOG_FILE_NAME} -gt 255 ]];
    then LOG_FILE_NAME=`uuidgen` ;
    fi
    ITEM_LOG_FILE="$JOB_LOG_DIR/$LOG_FILE_NAME"

But it's probably not very portable.

Original comment by kle...@gmail.com on 11 May 2010 at 12:15

[GoogleCodeExporter]

Thanks, I look into this. Portable as in portable to other shells, no. But PPSS 
requires Bash in the first place. Only 
the uuidgen is maybe an issue. The solution with the uuidgen may require 
alteration, since the current 
mechanism uses this file to determine if an item already has been processed.

Original comment by Louwrentius on 11 May 2010 at 11:38

• Changed state: Accepted

[GoogleCodeExporter]

I don't know what the output of your program should be. PPSS is designed to 
catch this output in it's log files as 
found in ppss_dir/job_log so you should not need to echo stuff in separate 
files. However, if you need clean 
output files, thus without the PPSS header and footer, there is an issue. If I 
would make an option that allows you 
to disable the header/footer, this issue would be solved.

Original comment by Louwrentius on 12 May 2010 at 11:59

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I second kleagg latter comment. I have been happily using ppss for a while now, 
and
one of my codes needs to send the <-c command> output to a file which name has
nothing to do with the $ITEM in the items list file, and it has to be saved in a
specific folder (not ppss_dir/job_log). 

His suggestion of using <ppss -f itemsfile -c 'bash -c $ITEM'> worked like a 
charm
for me (thanks for the hint). I do get those empty files as he does, so I just 
added
a <rm -rf emptyfile*> in my code to get rid of those.

As for the long log file name problem, I used `echo $RANDOM` as opposed 
`uuidgen` -
would that have less potential for problems?

Estephan

Original comment by estephan...@gmail.com on 12 May 2010 at 4:01

[GoogleCodeExporter]

Thanks for the input. I'm going to use that to improve PPSS.

Original comment by Louwrentius on 13 May 2010 at 1:46

[GoogleCodeExporter]

Hello,

I have released PPSS version 2.65 which now uses MD5 sums for file names. The 
MD5 sums are based on the 
items. This should fix the long filename issues. 

Also, I was able to track down the strange " f1.res" files that somehow 
appeared (insufficient escaping).

Please download this new version and report any issues you may have.

Original comment by Louwrentius on 14 May 2010 at 4:24

[GoogleCodeExporter]

Original comment by Louwrentius on 14 May 2010 at 10:16

[GoogleCodeExporter]

Original comment by Louwrentius on 15 May 2010 at 9:20

• Changed state: Fixed