cleanup homes directory

[ala-ebi]

make sure all the scheduled slurm jobs in scrontab have an error and output dir instead of using sportbot home dir by default. among those jobs:

• [x] FTP SYNC (prod, dev)

/hps/software/users/parkinso/spot/gwas/prod/scripts/cron/ftp_sync_wrapper_slurm.sh /hps/software/users/parkinso/spot/gwas/dev/scripts/cron/ftp_sync_wrapper_slurm.sh Although the sbatch inside scrontab has defined --output and --error, there logs in the homes dir indicating another job is created, probably because the original sbatch starts another sbatch (although I cant see it in the file, maybe from the .py script that's called)

• [x] GENERATE YAML (DEV)

/hps/software/users/parkinso/spot/gwas/dev/scripts/cron/sumstats_service/submit_yaml_generation.sh

• [x] GENERATE YAML (PROD)

/hps/software/users/parkinso/spot/gwas/prod/scripts/cron/sumstats_service/submit_yaml_generation.sh

• [ ] Stop RabbitMQ consuming messages for publication added in Depo curation

/hps/software/users/parkinso/spot/gwas/prod/sw/gwas-rabbitmq-listener/cancel-slurm-publication-import.sh

• [ ] Start RabbitMQ to consume messages for publication added in Depo curation

/hps/software/users/parkinso/spot/gwas/prod/sw/gwas-rabbitmq-listener/run-slurm-publication-import.sh

• [x] SEND YAML GEN FAILURES (DEV)

/hps/software/users/parkinso/spot/gwas/dev/scripts/cron/sumstats_service/submit_agg_email.sh

• [x] SEND YAML GEN FAILURES (PROD)

/hps/software/users/parkinso/spot/gwas/prod/scripts/cron/sumstats_service/submit_agg_email.sh

• [ ] /hps/software/users/parkinso/spot/gwas/prod/sw/deposition_sync/run-slurm-sync-wrapper.sh

• [ ] /hps/software/users/parkinso/spot/gwas/prod/sw/deposition_sync/run-slurm-sync-wrapper.sh unpublished

• [ ] /hps/software/users/parkinso/spot/gwas/prod/sw/mapping-pipeline/run-slurm-scheduled-mapping.sh

[karatugo]

Updated:

• FTP SYNC

  • /hps/software/users/parkinso/spot/gwas/dev/scripts/cron/ftp_sync_wrapper_slurm.sh
  • /hps/software/users/parkinso/spot/gwas/prod/scripts/cron/ftp_sync_wrapper_slurm.sh

• GENERATE YAML

  • /hps/software/users/parkinso/spot/gwas/prod/scripts/cron/sumstats_service/submit_yaml_generation.sh
  • /hps/software/users/parkinso/spot/gwas/dev/scripts/cron/sumstats_service/submit_yaml_generation.sh

• SEND YAML GEN FAILURES

  • /hps/software/users/parkinso/spot/gwas/dev/scripts/cron/sumstats_service/submit_agg_email.sh
  • /hps/software/users/parkinso/spot/gwas/prod/scripts/cron/sumstats_service/submit_agg_email.sh

• DEPO SYNC

  • /hps/software/users/parkinso/spot/gwas/dev/scripts/cron/sync_from_depo_slurm_dev.sh
  • /hps/software/users/parkinso/spot/gwas/prod/scripts/cron/sync_from_depo_slurm.sh

[karatugo]

@karatugo

• [x] Create empty log files

[karatugo]

• [x] Add log folders for dev and prod at /hps/nobackup/parkinso/spot/gwas/logs/sbatch and move sbatch logs there with sbatch script.sh > /hps/nobackup/parkinso/spot/gwas/logs/sbatch/prod/script.log

[karatugo]

Moved logs of rabbitmq start and stop scripts.

#SBATCH --output=/hps/nobackup/parkinso/spot/gwas/logs/sbatch/slurm-%j.txt
#SBATCH --error=/hps/nobackup/parkinso/spot/gwas/logs/sbatch/slurm-%j.txt

[karatugo]

Added 2>&1 to all scrontab entries to redirect both stderr and stdout.

[sprintell]

@sajo-ebi complete his part in the next sprint

[karatugo]

-rw-r--r--  1 spotbot spot         0 Nov 14 00:00 slurm-1343173.out - FTP-SYNC (PROD, MIDNIGHT)
-rw-r--r--  1 spotbot spot         0 Nov 14 00:15 slurm-1343176.out - SEND YAML GEN FAILURES (PROD)
-rw-r--r--  1 spotbot spot         0 Nov 14 00:15 slurm-1343177.out - SEND YAML GEN FAILURES (DEV)
-rw-rw-r--  1 spotbot spot   3417769 Nov 14 00:25 slurm-2072286.out - mapping
-rw-r--r--  1 spotbot spot         0 Nov 14 01:00 slurm-1343182.out - GENERATE YAML (PROD)
-rw-r--r--  1 spotbot spot         0 Nov 14 01:00 slurm-1343181.out - GENERATE YAML (DEV)
-rw-rw-r--  1 spotbot spot   3430881 Nov 14 04:25 slurm-2255120.out - mapping
-rw-r--r--  1 spotbot spot         0 Nov 14 06:50 slurm-1343179.out - rabbitmq
-rw-r--r--  1 spotbot spot         0 Nov 14 07:00 slurm-1343178.out - rabbitmq
-rw-rw-r--  1 spotbot spot   3406096 Nov 14 08:25 slurm-2394279.out - mapping
-rw-r--r--  1 spotbot spot         0 Nov 14 12:00 slurm-1343174.out - DEPO-SYNC (DEV)
-rw-r--r--  1 spotbot spot        28 Nov 14 12:25 slurm-1343168.out
-rw-rw-r--  1 spotbot spot   3173499 Nov 14 12:25 slurm-2426477.out - mapping
-rw-r--r--  1 spotbot spot         0 Nov 14 13:00 slurm-1343171.out - DEPO-SYNC (PROD, MORNING BATCH)
-rw-r--r--  1 spotbot spot         0 Nov 14 13:00 slurm-1343175.out - FTP-SYNC (DEV)
-rw-rw-r--  1 spotbot spot   1943308 Nov 14 14:59 slurm-2485397.out

[karatugo]

Okay I see that after a few days empty log files get deleted (probably automatically). So the following should be done now.

FTP SYNC (prod, dev)
GENERATE YAML (DEV)
GENERATE YAML (PROD)
SEND YAML GEN FAILURES (DEV)
SEND YAML GEN FAILURES (PROD)