[Code Refactor]: Shebang lines

[sof202]

Problem

In a large quantity of scripts (examples):

general/jobSubmission/copyFiles.sh
integrative/chromHMM/jobSubmission/4_chmmCompareModels.sh
array/SNPArray/jobSubmission/formatforQTLs.sh

The shebang is given as:

#!/usr/bin/sh

or

#!/bin/sh

Why this is a problem

Bash has many built-ins that POSIX shell does not. This includes (but is not limited to) source, arrays and mapfile. In the vast majority of cases source is being used in these scripts that use #!/bin/sh in the shebang.

On many systems (including the HPC at UoE), there is a symbolic link between /bin/sh and /bin/bash. This means that if any issues arise during runtime of these scripts (related to POSIX shell not interpreting these commands), bash will be called to interpret the line instead.

The problem comes when the script is ran on a different machine/system where this is not the case. Such a machine/system would not be able to interpret these commands correctly which can result in unwanted behaviour. It is best to state your intentions in the shebang of an executable file.

If you are not using any bash built-ins in your script, then this is not an issue.

Fix

Run these commands to convert all instances of #!/bin/sh (or #!/usr/bin/sh) to #!/bin/bash in your project:

list_of_shell_scripts=$(find . -type f -name *.sh)

for file in "${list_of_shell_scripts}"; do
  # Using '@' delimiter to avoid excessive escaping of '/' characters
  sed -i 's@#!/bin/sh@#!/bin/bash@' $file
  sed -i 's@#!/usr/bin/sh@#!/bin/bash@' $file
done

[sof202]

A comprehensive list of affected files: (Note: these are just files that have the shebang #!/bin/sh, the issue might not actually relate to all of these files if they do indeed only use POSIX shell commands).

general/jobSubmission/downloadENA.sh general/jobSubmission/copyFiles.sh general/jobSubmission/findError.sh general/jobSubmission/batchDownload.sh general/jobSubmission/downloadEGA.sh general/jobSubmission/downloadPsychEncode.sh general/jobSubmission/filePermissions.sh general/jobSubmission/downloadCommonMind.sh general/jobSubmission/replaceName.sh integrative/chromTools/jobSubmission/completeSample.sh integrative/chromTools/jobSubmission/complete.sh integrative/segway/jobSubmission/segwayBin.sh integrative/segway/jobSubmission/2_learnSegway.sh integrative/segway/jobSubmission/bamtoBig.sh integrative/segway/jobSubmission/optimalStates.sh integrative/segway/jobSubmission/1_genomeDataGroup.sh integrative/segway/jobSubmission/1_genomeData.sh integrative/segway/jobSubmission/bootstrapPeakCalling.sh integrative/segway/jobSubmission/bootstrapLearnSegway.sh integrative/segway/jobSubmission/bootstrapGenomeData.sh integrative/chromHMM/jobSubmission/calcStateQCMetrics.sh integrative/chromHMM/jobSubmission/subsampleByReads.sh integrative/chromHMM/jobSubmission/subsampleBinarise.sh integrative/chromHMM/jobSubmission/subsampleBed.sh integrative/chromHMM/jobSubmission/subsampleBySample.sh integrative/chromHMM/jobSubmission/runLDSCNeuralPeaks.sh integrative/chromHMM/jobSubmission/2_chmmMergeBinary.sh integrative/chromHMM/jobSubmission/runChromatinStateEnrichments.sh integrative/chromHMM/jobSubmission/1_chmmBinarise.sh integrative/chromHMM/jobSubmission/3_chmmLearnModel.sh integrative/chromHMM/jobSubmission/bootstrapBinarise.sh integrative/chromHMM/jobSubmission/bootstrapLearnModel.sh integrative/chromHMM/jobSubmission/4_chmmCompareModels.sh integrative/chromHMM/jobSubmission/calcBootstrapStateQCMetrics.sh integrative/chromHMM/jobSubmission/bedToBam.sh array/DNAm/preprocessing/jobSubmission/1_runDNAmQC.sh array/DNAm/analysis/neuralCellComposition/jobSubmission/testCTModels.sh array/DNAm/analysis/EWAS/jobSubmission/runEWAS.sh array/DNAm/analysis/QTLs/jobSubmission/runQTLs.sh array/DNAm/analysis/QTLs/jobSubmission/prepQTLData.sh array/DNAm/analysis/methodsDevelopment/jobSubmission/simulateCellSpecificEWAS.sh array/SNPArray/jobSubmission/runSNPImputationQC.sh array/SNPArray/jobSubmission/runSNPQC.sh array/SNPArray/jobSubmission/runFormatting1000G.sh array/SNPArray/jobSubmission/formatForQTLs.sh array/SNPArray/jobSubmission/liftoverImputation.sh array/SNPArray/jobSubmission/runCNVCalling.sh sequencing/BSSeq/preprocessing/_bsmrkGenomeConversion.sh sequencing/BSSeq/jobSubmission/segwayBin.sh sequencing/BSSeq/jobSubmission/3_collateS1QCMetrics.sh sequencing/BSSeq/jobSubmission/1_batchRunAlignment.sh sequencing/BSSeq/jobSubmission/2_batchComputeCorr.sh sequencing/longReadseq/jobSubmission/09c_talonTest_P0027_01db.sh sequencing/longReadseq/jobSubmission/squanti3BatchArray.sh sequencing/longReadseq/jobSubmission/06_talonLabelReadsBatchArray.sh sequencing/longReadseq/jobSubmission/09c_talon_P0027_01.sh sequencing/longReadseq/jobSubmission/05_transcriptCleanBatchArray.sh sequencing/longReadseq/jobSubmission/01_mergeSampleBarcodeFastqFiles.sh sequencing/longReadseq/jobSubmission/08_createTalonDB.sh sequencing/longReadseq/jobSubmission/04_minimap2AlignmentBatchArray.sh sequencing/longReadseq/jobSubmission/03_pychopperBatchArray.sh sequencing/longReadseq/jobSubmission/TESTsquanti3.sh sequencing/longReadseq/jobSubmission/02_nanoCompOnAllBarcodes.sh sequencing/longReadseq/jobSubmission/09_talonAllSamples.sh sequencing/longReadseq/jobSubmission/09c_talonTest_AllSamplesDB_P0027_01.sh sequencing/longReadseq/jobSubmission/09b_talon_LondonCallingSamples.sh sequencing/ChIPSeq/jobSubmission/3_batchCalcQCMetrics.sh sequencing/ChIPSeq/jobSubmission/bamtoBigwig.sh sequencing/ChIPSeq/jobSubmission/1_batchRunAlignment.sh sequencing/ChIPSeq/jobSubmission/2_batchRunPeakCalling.sh sequencing/ChIPSeq/jobSubmission/4_collateStage1QCMetrics.sh sequencing/ChIPSeq/jobSubmission/peakCallingByGroup.sh sequencing/DNAhydroxy/jobSubmission/peakCalling5hmCIncreasingSampleSizes.sh sequencing/DNAhydroxy/jobSubmission/runHydroxyPeakCalling.sh sequencing/DNAhydroxy/jobSubmission/batchPeakCallingByGroup.sh sequencing/DNAhydroxy/jobSubmission/calcSampleQCMetrics.sh sequencing/DNAhydroxy/jobSubmission/batchQCMetrics.sh sequencing/DNAhydroxy/jobSubmission/runHydroxyQC.sh sequencing/ATACSeq/preprocessing/jobSubmission/1_batchRunAlignment.sh sequencing/ATACSeq/preprocessing/jobSubmission/7_batchRunGenotypeConcordance.sh sequencing/ATACSeq/preprocessing/jobSubmission/6_sexCheck.sh sequencing/ATACSeq/preprocessing/jobSubmission/5_batchFormatSexChrs.sh sequencing/ATACSeq/preprocessing/jobSubmission/2_batchCalcQCMetrics.sh sequencing/ATACSeq/preprocessing/jobSubmission/4_collateStage1QCMetrics.sh sequencing/ATACSeq/preprocessing/jobSubmission/3_batchRunPeakCalling.sh sequencing/ATACSeq/preprocessing/jobSubmission/8_batchRunGenotypeSearch.sh sequencing/ATACSeq/jobSubmission/9_idrAnalysis.sh sequencing/ATACSeq/jobSubmission/groupPeakCallingHMMR.sh sequencing/ATACSeq/jobSubmission/subsetNmix.sh sequencing/ATACSeq/jobSubmission/samplePeakCallingHMMR.sh sequencing/ATACSeq/jobSubmission/macsTools.sh sequencing/ATACSeq/jobSubmission/1_batchRunAlignment.sh sequencing/ATACSeq/jobSubmission/7_batchRunGenotypeConcordance.sh sequencing/ATACSeq/jobSubmission/7_2_batchRunGenotypeSearch.sh sequencing/ATACSeq/jobSubmission/correctFripBAMPE.sh sequencing/ATACSeq/jobSubmission/8_batchPeakCallingByGroup.sh sequencing/ATACSeq/jobSubmission/6_sexCheck.sh sequencing/ATACSeq/jobSubmission/5_batchFormatSexChrs.sh sequencing/ATACSeq/jobSubmission/scoping.sh sequencing/ATACSeq/jobSubmission/2_batchCalcQCMetrics.sh sequencing/ATACSeq/jobSubmission/lifthg37to38.sh sequencing/ATACSeq/jobSubmission/4_collateStage1QCMetrics.sh sequencing/ATACSeq/jobSubmission/testPeakCallingbyGroup.sh sequencing/ATACSeq/jobSubmission/3_batchRunPeakCalling.sh sequencing/ATACSeq/jobSubmission/batchFormatSexChrs.sh sequencing/ATACSeq/jobSubmission/10_diffAnalysis.sh sequencing/SingleCell/scRNASeq-ann-twins/scripts/scripts-ann/isca-transfers/230905-transfer-backup.sh