search

BCCDC-DSI / RADD

Consult at the BCCDC for Mass Spectrometry
MIT License
1 stars 0 forks source link

Sockeye: urgent request to finish 3 more batches of part1 #36

Closed lisatwyw closed 6 days ago

lisatwyw commented 1 week ago

SLURM job arrays

Batch 2:

cd /scratch/st-ashapi01-1/RADD/part1_sbatch_n_err/
mkdir exp2023_nps
cd exp2023_nps
sbatch ../go_loop.sh  /arc/project/st-cfjell-1/ms_data/expedited_2023/mzML/ /arc/project/st-ashapi01-1/RADD_libraries/NPS_DB-240705.csv /scratch/st-ashapi01-1/expedited_2023 50 1

Batch 3:

cd /scratch/st-ashapi01-1/RADD/part1_sbatch_n_err/
mkdir exp2023_highres
cd exp2024_highres
sbatch ../go_loop.sh  /arc/project/st-cfjell-1/ms_data/expedited_2024/ /arc/project/st-ashapi01-1/RADD_libraries/HRN_2023-10-01_v4_v5.csv /scratch/st-ashapi01-1/expedited_2024 50 1

Batch 4:

cd /scratch/st-ashapi01-1/RADD/part1_sbatch_n_err/
mkdir exp2024_nps
cd exp2024_nps
sbatch ../go_loop.sh  /arc/project/st-cfjell-1/ms_data/expedited_2024/ /arc/project/st-ashapi01-1/RADD_libraries/NPS_DB-240705.csv /scratch/st-ashapi01-1/expedited_2024 50 1

Check job completion statuses

Simply copy-paste

bash /scratch/st-ashapi01-1/RADD/part1_sbatch_n_err/chk.sh

To-do

  1. Check on above
  2. Download to O drive to inspect why databases gave different responses (+ why as.numeric() need applied twice)
  3. User docs: add notes about using consistent naming conventions; a. Input folders:
    • expedited_2023/mzML/*mzML
    • expedited_2024/*mzML b. Database: remove spaces and use consistent naming conventions as the prefix will be the folder name of the output files (*rds); e.g.
    • name_v6_2024-07-06
    • name_v7_2024.07.09
lisatwyw commented 1 week ago

Upload done; n = 827:

66152 -rw-r----- 1 ashapi01 st-cfjell-1-rw 53907371 Jul  5 07:29 /arc/project/st-cfjell-1/ms_data/expedited_2024/2024-2337BG01.mzML
66096 -rw-r----- 1 ashapi01 st-cfjell-1-rw 53823958 Jul  5 07:29 /arc/project/st-cfjell-1/ms_data/expedited_2024/2024-2338BG01.mzML
65936 -rw-r----- 1 ashapi01 st-cfjell-1-rw 53639762 Jul  5 07:29 /arc/project/st-cfjell-1/ms_data/expedited_2024/2024-2339BG01.mzML
65832 -rw-r----- 1 ashapi01 st-cfjell-1-rw 53590506 Jul  5 07:29 /arc/project/st-cfjell-1/ms_data/expedited_2024/2024-2340BG01.mzML
lisatwyw commented 1 week ago

Check how many not finished

2390 completed in [2023] folder that got compared to database [HRN_2023-10-01_v4_v5] at time 08:50:15 
2390 completed in [2023] folder that got compared to database [NPS_DB-240705] at time 08:50:16 
781 completed in [2024] folder that got compared to database [HRN_2023-10-01_v4_v5] at time 08:50:16 
757 completed in [2024] folder that got compared to database [NPS_DB-240705] at time 08:50:16

Complete the remaining for the 2024 data pool

cd /scratch/st-ashapi01-1/RADD/part1_sbatch_n_err/exp2024_nps/
sbatch ../go_loop.sh  /arc/project/st-cfjell-1/ms_data/expedited_2024/ /arc/project/st-ashapi01-1/RADD_libraries/NPS_DB-240705.csv /scratch/st-ashapi01-1/expedited_2024 5 1 
cd /scratch/st-ashapi01-1/RADD/part1_sbatch_n_err/exp2024_highres/
sbatch ../go_loop.sh  /arc/project/st-cfjell-1/ms_data/expedited_2024/ /arc/project/st-ashapi01-1/RADD_libraries/HRN_2023-10-01_v4_v5.csv /scratch/st-ashapi01-1/expedited_2024 5 1