Open SilasK opened 1 year ago
which wrapper are you using?
samtools / stats
I think that is because that is not a wrapper error, but rather a snakemake
error.
The wrapper requires the input file to be names as bam
. Can you include the rule you are trying to run?
I see. Then this is more of a feature request. If there is such an error in a normal rule. It gets detected by the main snakemake instance at the very beginning. Here the rule gets submitted and only there is it caught.
Also if there is anything else in the wrapper that doesn't work the error will just be printed to the stderr or stdout. Which will likely be missed.
But those errors should be caught if you do a dry run of your workflow, no?
Snakemake version: 7.32.3 Snakemake wrapper: 2.6.0
I am getting a similar issue where logs for rules that use snakemake wrappers aren't getting written to logs if they error out. Running the jobs locally, correctly writes the errors. Will snakemake still upload the logs to remote storage if that rule errors out?
Can you provide an example of the logs you expect to be written?
The rule:
rule STAR_multi:
input:
fq1 = input_reads,
fq2 = input_reads_2,
idx = config["star_index"],
output:
aln = f"{bp}STAR/{{sample}}/{{sample}}.bam",
log = f"{bp}STAR/{{sample}}/{{sample}}.log",
log_final = f"{bp}STAR/{{sample}}/Log.final.out",
sj = f"{bp}STAR/{{sample}}/SJ.out.tab"
log:
f"{bp}logs/STAR/{{sample}}.log"
resources:
mem_mb = 54000
params:
extra = "--outSAMtype BAM SortedByCoordinate",
threads:
8
wrapper:
"v2.9.0/bio/star/align"
And the written log on local: /home/jeff/mambaforge/envs/8c6cfff4a79e638bd42875ab09a0331d_/bin/STAR: line 8: 353419 Killed "${cmd}" "$@"
If you run locally, you say that it "correctly writes the errors"; what do you mean? Does it write them to the screen? To the rule log
file? To the snakemake log?
And when is the message not printed? When you run it in a cluster? Are you using slurm?
The error is written to the log file specified in the snakemake rule. The cluster is a google kubernetes cluster. No log file is created. Completed rules do write the log file.
And do you see the same behavior with rules that do not use wrappers? Or is it only when using wrappers?
@johanneskoester I would like to add logging to each wrapper. I think we can do this more or less automated.
E.g. I would like to add something like the following code at the beginning of each wrapper. Before making a PR I would discuss if this is ok, or needs adaption.
### loging all exceptiuon to log file if available
import sys
import logging, traceback
import warnings
if len(snakemake.log)>0:
logfile = snakemake.log[0]
else:
logfile =None
logging.basicConfig(
filename=logfile,
level=logging.INFO,
format="%(asctime)s %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(__file__) # Or simply "wrapper"
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logger.error(
"".join(
[
"Uncaught exception: ",
*traceback.format_exception(exc_type, exc_value, exc_traceback),
]
)
)
# Install exception handler
sys.excepthook = handle_exception
###################################### Beginning of wrapper ######################################
Eventually, the logfile
should be determined by _infer_stdout_and_stderr
https://github.com/snakemake/snakemake/pull/2474 to allow logging to stderr file if there are multiple log files.
And do you see the same behavior with rules that do not use wrappers? Or is it only when using wrappers?
Yeah, my python scripts without the sys.excethook = handle_exception that @SilasK posted have the same problem.
Snakemake version Note the Snakemake and snakemake-wrappers version for which you experience the bug. Always make sure to try out the latest version of both before creating a new bug report.
7.15
Describe the bug
Error in snakemake wrapper goes not to log file
Logs
Minimal example see log
Additional context cluster execution