Open jdimatteo opened 10 years ago
jdimatteo
commented
10 years ago I updated the permissions to /ark/temp so that any user can add files/folders (and remove/modify files that they own).
jdm@tod:~$ sudo chmod a+rwxt /ark/temp/This fixed a problem where the jenkins user couldn't add the temp folders needed by GPL16043.sh.
jdimatteo
commented
10 years ago @bradnerComputation / Charles, this is all set up now, let me know if you have any problems or if I should make any changes:
http://tod.dfci.harvard.edu/jenkins/job/Normalize_GPL16043/build?delay=0sec
charlesylin
commented
10 years ago Awesome. Will test tonight. Thanks for your help as always, especially this late.
-Charles
On Oct 17, 2013, at 11:47 PM, jdimatteo notifications@github.com wrote:
@bradnerComputation / Charles, this is all set up now, let me know if you have any problems or if I should make any changes:
http://tod.dfci.harvard.edu/jenkins/job/Normalize_GPL16043/build?delay=0sec
— Reply to this email directly or view it on GitHub.
charlesylin
commented
10 years ago John,
It seems like the uploaded zip isn't getting cp'd to the right tmp folder
Started by user anonymous http://tod.dfci.harvard.edu/jenkins/user/null Building in workspace /ark/home/jenkins/jobs/Normalize_GPL16043/workspace Checkout:workspace / /ark/home/jenkins/jobs/Normalize_GPL16043/workspace - hudson.remoting.LocalChannel@e1935d6 Using strategy: Default Last Built Revision: Revision 79e34215db5a50d8df3a41c9081673d2741d578f (origin/HEAD, origin/master) Fetching changes from 1 remote Git repository Fetching upstream changes from origin Seen branch in repository origin/HEAD Seen branch in repository origin/master Seen 2 remote branches Commencing build of Revision 79e34215db5a50d8df3a41c9081673d2741d578f (origin/HEAD, origin/master) Checking out Revision 79e34215db5a50d8df3a41c9081673d2741d578f (origin/HEAD, origin/master) Warning : There are multiple branch changesets here No emails were triggered. Copying file to input.zip [workspace] $ /bin/sh -xe /tmp/hudson3165583539119894779.sh
Charles Y. Lin, Ph.D. Dana-Farber Cancer Institute Department of Medical Oncology charles_lin@dfci.harvard.edumailto:charles_lin@dfci.harvard.edu http://bradnerlab.com
On Thu, Oct 17, 2013 at 11:52 PM, Charles Lin charles.y.lin@gmail.comwrote:
Awesome. Will test tonight. Thanks for your help as always, especially this late.
-Charles
On Oct 17, 2013, at 11:47 PM, jdimatteo notifications@github.com wrote:
@bradnerComputation https://github.com/bradnerComputation / Charles, this is all set up now, let me know if you have any problems or if I should make any changes:
http://tod.dfci.harvard.edu/jenkins/job/Normalize_GPL16043/build?delay=0sec
— Reply to this email directly or view it on GitHubhttps://github.com/BradnerLab/JenkinsJobs/issues/6#issuecomment-26570579 .
jdimatteo
commented
10 years ago @bradnerComputation / Charles,
I fixed the input.zip problem, sorry about that.
I still see errors in the stdout though, e.g.
> selibrary(affy)
Error: could not find function "selibrary"
Execution haltedI see this selibrary on the first line of /ark/home/cl512/pipeline/GPL16043.r
Charles, is there a file missing with the definition of selibrary?
Also, on line 28 of GPL16043.sh it is calling "/ark/home/cl512/pipeline/GPL16043.r" instead of just "GPL16043.r" -- was that deliberate? I setup the Jenkins job to check out the latest files from github, so if you just leave off the path GPL16043.sh will use the latest version of GPL16043.r in the git repo instead of whatever is in your home directory.
charlesylin
commented
10 years ago That's a typo that I must have inserted today when I was trying to muck with the code. I didn't know we could pull right from the github for the GPL16043.r code. That would definitely be easier.
-Charles
Charles Y. Lin, Ph.D. Dana-Farber Cancer Institute Department of Medical Oncology charles_lin@dfci.harvard.edumailto:charles_lin@dfci.harvard.edu http://bradnerlab.com
On Fri, Oct 18, 2013 at 9:10 PM, jdimatteo notifications@github.com wrote:
@bradnerComputation https://github.com/bradnerComputation / Charles,
I fixed the input.zip problem, sorry about that.
I still see errors in the stdout though, e.g.
selibrary(affy) Error: could not find function "selibrary" Execution halted
I see this selibrary on the first line of GPL16043.r .
Charles, is there a file missing with the definition of selibrary?
Also, on line 28 of GPL16043.sh it is calling "/ark/home/cl512/pipeline/GPL16043.r" instead of just "GPL16043.r" -- was that deliberate? I setup the Jenkins job to check out the latest files from github, so if you just leave off the path GPL16043.sh will use the latest version of GPL16043.r instead of whatever is in your home directory.
— Reply to this email directly or view it on GitHubhttps://github.com/BradnerLab/JenkinsJobs/issues/6#issuecomment-26640170 .
jdimatteo
commented
10 years ago It seems to be working now that I've made the changes shown below. Let me know if you'd like me to checkin these changes to the repo:
jdm@tod:~/pipeline$ git diff
diff --git a/GPL16043.sh b/GPL16043.sh
index 4058854..83cf604 100755
--- a/GPL16043.sh
+++ b/GPL16043.sh
@@ -12,6 +12,7 @@ ID=$RANDOM
#ID=1234
TEMP_DIR_ROOT=/ark/temp/
TEMP_DIR=$TEMP_DIR_ROOT$NAME\_tmp\_$ID
+INITIAL_DIR=`pwd`
#making the temp directory
mkdir $TEMP_DIR
@@ -25,14 +26,14 @@ unzip $TEMP_DIR/$NAME\_tmp_$ID.zip
mkdir $TEMP_DIR/output
#run the spikey normy
-R --no-save $TEMP_DIR/ $NAME < /ark/home/cl512/pipeline/GPL16043.r
+R --no-save $TEMP_DIR/ $NAME < $INITIAL_DIR/GPL16043.r
#run the GPL gene level script
-python /ark/home/cl512/pipeline/GPL16043.py -i $TEMP_DIR/output/$NAME\_all_mas
-python /ark/home/cl512/pipeline/GPL16043.py -i $TEMP_DIR/output/$NAME\_all_mas
+python $INITIAL_DIR/GPL16043.py -i $TEMP_DIR/output/$NAME\_all_mas5_probe_expr
+python $INITIAL_DIR/GPL16043.py -i $TEMP_DIR/output/$NAME\_all_mas5_probe_expr
#zip up the output
cd $TEMP_DIR
zip -r $NAME\_output.zip output
-echo "WROTE OUTPUT TO: " $TEMP_DIR/$NAME\_output.zip
\ No newline at end of file
+echo "WROTE OUTPUT TO: " $TEMP_DIR/$NAME\_output.zip
jdm@tod:~/pipeline$ I updated the job to use the fork at https://github.com/jdimatteo/pipeline, which currently just has the changes above. The Jenkins job seems to be working correctly without any errors now
Charles explained the following in an email:
the user uploads a zip file and provides a name
jenkins can then call the following job
/ark/home/cl512/pipeline/GPL16043.sh ARG1 ARG2
e.g. bash /ark/home/cl512/pipeline/GPL16043.sh /ark/home/cl512/ressrv19/raw/expression/JB20130926st.zip affy2
where ARG1 is the path to the zip file
and ARG2 is the name that the user provided (the name can't have any spaces)
In the last line of the std out of GPL16043.sh, the location of the output file is provided as follows (example):
"WROTE OUTPUT TO: /ark/temp/affy2_tmp_15578/affy2_output.zip"
this zip should then be sent by email to the user.