dakusui / bredxbred

mapreduce in bash
Other
4 stars 1 forks source link

Example not working (need more documentation) #14

Closed dlazesz closed 7 years ago

dlazesz commented 7 years ago

Could you provide a complete example for wordcount?

I'm using Ubuntu 16.04. I've followed the steps to setup the program. Kept the default settings. I'm running the following command:

cat input.txt | wordcount/main.xbred > output.txt

It says:

xbred: starting pipeline
xbred: done

But output.txt is empty and there is some processing after the main program exited.

Where supposed the output to go?

dlazesz commented 7 years ago

Ok. Problem solved. It was my fault.

The correct invocation is: cat input.txt | wordcount/main.xbred > output.txt

The following problems occured:

1) Wrong netcat version. Should be a check for it! (I know I should RTFM more carefully.) 2) Need to enable debuging to catch the errors (not documented): export XBRED_DEBUG="on" 3) Because the asynchronous behaviour there is a delay between "xbred: done" and the actual writing of the result. 4) Some cleanup script would be useful in the case of failure: killall awk; killall nc; rm -rf work/ && mkdir -p work/{fs,jm,sort} 5) bredfs write uses non-interactive ssh session, so ~/.bashrc is not executed and bred is not found when the PATH variable is updated in it to contain bred. Should be documented...

dlazesz commented 7 years ago

I created PR #15 to resolve the issue. Please consider merging it.

dlazesz commented 7 years ago

Merged. Thank you!