search

Sage-Bionetworks / SynapseWorkflowHook

Code for linking a workflow engine to a Synapse evaluation queue
Apache License 2.0
4 stars 1 forks source link

Adding restrictions for size of log file returned #20

Open thomasyu888 opened 6 years ago

thomasyu888 commented 6 years ago

Should the hook have this feature?

brucehoff commented 6 years ago

I don't know if the requirements for limiting the log size are generic across all applications. E.g., in the DM challenge we sought to limit the size of the total data returned (not just logs). Depending on the requirements it might be better to provide a mechanism for an outside process to 'turn off' log uploads based on its own criteria, much as today you can determine that a time quota has been exceeded and signal that a submission should be terminated. Bottom line: It just depends on the application's requirements.

thomasyu888 commented 6 years ago

@brucehoff

Ahh I see, so I need to have two external processes:

a. Looks at log file size b. Looks at submission run time

If either go over the limit, then terminate the submission

brucehoff commented 6 years ago

@thomasyu888 An alternative to terminating the submission when the log size is exceeded is to continue running the submission but not to upload any more logs.

We need to make a small change in the 'hook to check for the log-size-exceeded signal. (Currently we only check for the time-limit-exceeded signal.)

thomasyu888 commented 6 years ago

@brucehoff

Actually there are TOIL commands for max log size! We could just use these instead of making submissions invalid?

Well I guess the issue would be if the logs are rotating, then users can look at older versions to piece things together...


--rotatingLogging
--
  | Turn on rotating logging, which prevents log files from getting too big (set using --maxLogFileSize BYTESIZE).
--maxLogFileSize BYTESIZE
  | Sets the maximum log file size in bytes (--rotatingLogging must be active).