search

DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.
Apache License 2.0
265 stars 44 forks source link

Missing timestamps in .log files #147

Closed pbilling closed 5 years ago

pbilling commented 5 years ago

I noticed that some time between dsub v1 & 2 the timestamps have disappeared from inside the .log files. These were very useful in determining runtimes; is there any way to get them back? I'm currently using dsub v0.2.5 with the v2 provider.

Thanks, Paul

mbookman commented 5 years ago

Thanks Paul. Can you be more specific about what timestamps you would like to see?

pbilling commented 5 years ago

Hi Matt, for sure.

Here is an excerpt from a v1 log file where date/time stamps precede each status update:

2018/06/29 22:24:18 I: Pulled image "gcr.io/gbsc-gcp-project-mvp/samtools" successfully.
2018/06/29 22:24:18 I: Switching to status: localizing-files
2018/06/29 22:24:18 I: Calling SetOperationStatus(localizing-files)
2018/06/29 22:24:18 I: SetOperationStatus(localizing-files) succeeded
2018/06/29 22:24:18 I: Docker file /mnt/data/input/gs/my-bucket/SHIP4447072/Alignments/SHIP4447072_chromosome_1.recal.bam maps to host location /mnt/datadisk/input/gs/my-bucket/SHIP4447072/Alignments/SHIP4447072_chromosome_1.recal.bam.
2018/06/29 22:24:18 I: Running command: sudo gsutil -q -m cp gs://my-bucket/SHIP4447072/Alignments/SHIP4447072_chromosome_1.recal.bam /mnt/datadisk/input/gs/my-bucket/SHIP4447072/Alignments/SHIP4447072_chromosome_1.recal.bam
2018/06/29 22:26:52 I: Done copying files.

And a similar v2 log file:

gsutil -h Content-Type:text/plain  -mq cp /tmp/continuous_logging_action/output gs://my-bucket/dsub/flagstat/samtools/logs/20190329/samtools--root--190329-195304-57.log
mkdir -m 777 -p /mnt/data/input/gs/my-bucket/SHIP4731843/Alignments
mkdir -m 777 -p /mnt/data/output/gs/my-bucket/dsub/flagstat/samtools/objects
Localizing INPUT
gsutil   -mq cp gs://my-bucket/SHIP4731843/Alignments/SHIP4731843_chromosome_4.recal.bam /mnt/data/input/gs/my-bucket/SHIP4731843/Alignments/SHIP4731843_chromosome_4.recal.bam
gsutil -h Content-Type:text/plain  -mq cp /tmp/continuous_logging_action/output gs://my-bucket/dsub/flagstat/samtools/logs/20190329/samtools--root--190329-195304-57.log

It would be great to have date/time stamps in front of each line again.

mbookman commented 5 years ago

Agreed. Thanks for the report.

mbookman commented 5 years ago

This is fixed in release 0.3.1 by d3269e53a3dd68063a94b474fc1efd4825130259.