TimeTracker database output option sometimes causes exception

knoepfel commented 3 years ago

This issue has been migrated from https://cdcvs.fnal.gov/redmine/issues/18305 (FNAL account required) Originally created by @brownd1978 on 2017-11-16 14:07:17

I am using the dbOutput (SQLite) output of TimeTracker to study module timing distributions. This works when I create a new file for each run. If instead I use the 'overwrite' option (overwrite : true) to overwrite an existing file, art sometimes issues an exception, noted below. The exception seems to be issued randomly (rerunning the same job multiple times produces different results), but is fairly consistently produced when processing more events the 2nd time than the first.

%MSG-s ArtException: Early 30-Oct-2017 16:04:12 CDT JobSetup
cet::exception caught in art
---- OtherArt BEGIN
  ServiceCreation
  ---- SQLExecutionError BEGIN
    database disk image is malformed
  ---- SQLExecutionError END
  cet::exception caught during construction of service type
art::TimeTracker:
---- OtherArt END
%MSG
Art has completed and will exit with status 1

As a more minor but annoying issue, I note that, without the overwrite option, if I try to write to the same database file, the exception message is obscure and doesn't help the user understand what the problem is.

Transaction d'tor called before commit was called.
terminate called after throwing an instance of
'cet::coded_exception'
  what():  ---- SQLExecutionError BEGIN
  Failed to bind int64.
  Return code: 21
---- SQLExecutionError END

knoepfel commented 3 years ago

Comment by @chissg on 2017-11-20 17:43:48

We will attempt to reproduce and resolve the primary issue, and produce a more informative message in the second case.

knoepfel commented 3 years ago

Comment by @knoepfel on 2017-11-27 14:16:43

Dave, can you provide a sample job that demonstrates this (sporadic) failure? I need to know:

the machine you're running on (e.g. mu2egpvm01)
the version of mu2e Offline you're using if the job depends on anything from there
ideally, the version of art--the "Occurs In" field says you observe this is cetlib:version:1.03.09, which does not correspond to a version of art.

knoepfel commented 3 years ago

Comment by @brownd1978 on 2017-12-06 15:49:22

I have not been able to reproduce this error. I suggest we close this issue for now.

knoepfel commented 3 years ago

Comment by @knoepfel on 2017-12-06 15:59:39

Thank you, Dave, for the extra information. I will put the issue in "Remission" state to reflect that it may still be an issue later on.

art-framework-suite / art

TimeTracker database output option sometimes causes exception #78