search

hammerlab / biokepi

Bioinformatics Ketrew Pipelines
Apache License 2.0
27 stars 4 forks source link

Nodes don't appear to wait on gsutil cp to be done #457

Closed tavinathanson closed 7 years ago

tavinathanson commented 7 years ago

The missing files are there when I check them later on:

### Job e652c8ce-0a5c-5859-9223-a8fd1f4135dc
### Freshness: Archived because of error: Shell-command failed:
Command:

kubectl logs e652c8ce-0a5c-5859-9223-a8fd1f4135dc

Status: Exited with 1

Standard-output: empty.

Standard-error:

Error from server (NotFound): pods "e652c8ce-0a5c-5859-9223-a8fd1f4135dc" not found

### Output:

## Biokepi machine: 'umask 000'
## Biokepi machine: 'whoami'
biokepi
## Biokepi machine: 'groups'
biokepi
## Biokepi machine: 'hostname'
e652c8ce-0a5c-5859-9223-a8fd1f4135dc
## Biokepi machine: 'uname -a'
Linux e652c8ce-0a5c-5859-9223-a8fd1f4135dc 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 x86_64 x86_64 GNU/Linux
## Biokepi machine: 'export PATH=/home/biokepi/google-cloud-sdk/bin:/opt/google-cloud-sdk/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
## Biokepi machine: 'ls /home/biokepi/google-cloud-sdk/bin'
ls: cannot access '/home/biokepi/google-cloud-sdk/bin': No such file or directory
Command failed
[E::hts_open_format] fail to open file '/nfs-pool-27/biokepi/work/work-hg19-tcga_realign_3773/122551207f56f2e2452b149523de4dadC347.TCGA-67-3773-01A-01D-1040-01.1_gdc_realn.bam'
[bam_sort_core] fail to open '/nfs-pool-27/biokepi/work/work-hg19-tcga_realign_3773/122551207f56f2e2452b149523de4dadC347.TCGA-67-3773-01A-01D-1040-01.1_gdc_realn.bam': No such file or directory
smondet commented 7 years ago

This was an NFS overload→sync problem; not biokepi itself.