Question about job submissions on PBS using biomake

[samwalrus]

This is more a usability question, I am just trying to learn how to use biomake with job submissions:

I have the following Makeprog file:

'$(Base).noheader.csv' <-- '../../withdrawls/$(Base).csv',
    'tail -n +2 ../../withdrawls/$(Base).csv > $(Base).noheader.csv'.

'$(Base).plain' <-- '$(Base).noheader.csv',
    'sed "s/,//g" $(Base).noheader.csv > $(Base).plain'.

'both.cat' <-- ['trip_quad.plain',
    'consent_withdrawn.plain'],
    'cat consent_withdrawn.plain trip_quad.plain > both.cat'.  %line 38 cat needs new line

'wocs.txt' <-- ['both.cat'],
    'uniq -u both.cat > wocs.txt'.

Which works fine to concatenate two csv files (consent_withdrawn.csv and trip_quad.csv), and turn them into a plain list file called wocs.txt. I can run this with :

biomake wocs.txt

I have tried to submit the job to our queuing system with :

biomake -Q pbs wocs.txt

This makes a set of jobs:

`newblue4:/var/spool/mail$ qstat -u myusername master.cm.cluster: Reqd Reqd Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

8867488.master.c alspacdata veryshor trip_quad.nohead -- -- -- -- 01:00:00 Q -- 8867489.master.c alspacdata veryshor trip_quad.plain -- -- -- -- 01:00:00 H 01:00:00 8867490.master.c alspacdata veryshor consent_withdraw -- -- -- -- 01:00:00 Q -- 8867491.master.c alspacdata veryshor consent_withdraw -- -- -- -- 01:00:00 H 01:00:00 8867492.master.c alspacdata veryshor both.cat -- -- -- -- 01:00:00 H 01:00:00 8867493.master.c alspacdata veryshor wocs.txt -- -- -- -- 01:00:00 H 01:00:00 `

But the jobs all get deleted and I do not have any successful builds (for example a message):

` From hpc-admin@bristol.ac.uk Tue Nov 5 17:31:41 2019 Return-Path: hpc-admin@bristol.ac.uk X-Original-To: alspacdata@newblue1 Delivered-To: alspacdata@newmaster2.acrc.bris.ac.uk Received: by newmaster2.acrc.bris.ac.uk (Postfix, from userid 0) id CBD6932436E; Tue, 5 Nov 2019 17:31:41 +0000 (GMT) To: alspacdata@newmaster2.acrc.bris.ac.uk Subject: PBS JOB 8867489.master.cm.cluster Precedence: bulk Message-Id: 20191105173141.CBD6932436E@newmaster2.acrc.bris.ac.uk Date: Tue, 5 Nov 2019 17:31:41 +0000 (GMT) From: hpc-admin@bristol.ac.uk (root)

PBS Job Id: 8867489.master.cm.cluster Job Name: trip_quad.plain Aborted by PBS Server Job deleted as result of dependency on job 8867488.master.cm.cluster `

What am I doing wrong in the job submissions for the builds, how is this meant to work?

[cmungall]

Hi Sam! @ihh is the expert on the qsub part hopefully he can answer

[samwalrus]

@cmungall @inh I worked this out :).
I saw that a .biomake dir is created which had useful files with the errors. There I could see that it could not find the files, which I worked out is because when connecting to our compute cluster the login nodes have some storage mounted which is where I was launching the work from. But these drives are not mounted to the compute nodes, so the files were not visible in the running job and hence the failure of the jobs. I will continue to explore biomake, so far it seems like a really cool tool, so thanks for the writing it !