feat: custom log file behaviour

cmeesters commented 1 month ago

This PR should allow to

set a custom log directory for SLURM jobs
enable auto-deletion of older SLURM log files after 10 days
allow changing the that behaviour or disabling auto deletion all together
auto delete log files of successful SLURM jobs (and ignore this default behaviour)

The idea behind this PR is to

limit the number of SLURM log files to keep, as a workflow can easily produce thousands of log files. In case of a successful job, the information is redundant anyway, hence the proposed auto-deletion of successful jobs.
users may select a custom directory for SLURM log files - this way different workflows can point to one prefix. Together with the auto-deletion of older log files, this futher limits the number of present log files.

It addresses issues #94 and #123.

Summary by CodeRabbit

New Features
- Added a method to clean old log files based on age, improving log maintenance.
- Introduced new command line flags for enhanced log management, including options for log directory and retention settings.
Improvements
- Enhanced documentation for clearer guidance on configuring SLURM resources and managing logs.
- Updated job status handling to respect new log retention settings, improving configurability.

coderabbitai[bot] commented 3 weeks ago

Walkthrough

The changes enhance the Executor class in the SLURM executor plugin for Snakemake by introducing new log management features and configuration options. A method named clean_old_logs is added to delete outdated log files, and the ExecutorSettings dataclass is updated with new fields for configuring the log directory and retention settings. The run_job method is modified to utilize the new log configuration, and the check_active_jobs method now respects the setting for retaining successful logs.

Changes

File	Change Summary
`snakemake_executor_plugin_slurm/__init__.py`	- Added method `clean_old_logs(self)` in `Executor`. - Added fields `logdir`, `keep_successful_logs`, and `delete_logfiles_older_than` in `ExecutorSettings`. - Updated `run_job` to use `logdir` for SLURM log file path. - Modified `check_active_jobs` to consider `keep_successful_logs`.
`snakemake_executor_plugin_slurm/utils.py`	- Imported `logging`, created a logger instance. - No changes to `delete_slurm_environment`.
`docs/further.md`	- Added command line flags: `--slurm-logdir`, `--slurm-keep-successful-logs`, `--slurm-delete-logfiles-older-than`. - Clarified existing flags: `--slurm-init-seconds-before-status-checks`, `--slurm-requeue`. - Expanded section on "Advanced Resource Specifications" and updated "Retries" section. - Introduced section on "Dynamic Parameterization".

Possibly related PRs

#137: Modifies the warn_on_jobcontext method in the ExecutorSettings class, which is related to the Executor class changes in the main PR, as both involve enhancements to job management within the SLURM context.
#140: Changes to the run_job method in this PR directly relate to the modifications made in the main PR's run_job method, specifically regarding log file handling and job execution.
#161: This PR also modifies the run_job method, enhancing error handling during SLURM job submission, which is relevant to the changes made in the main PR regarding log management and job execution.
#153: The documentation updates in this PR include new command line flags for log management, directly related to the new fields added in the ExecutorSettings dataclass in the main PR.

Suggested reviewers

johanneskoester
dlaehnemann

🐰 In the log dir, where bunnies hop,
New methods sprout, they never stop!
With jobs to run and logs to clear,
SLURM's now tidy, bring on the cheer!
Hopping through tasks, all is bright,
In the world of jobs, everything's right! 🌟

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

🪧 Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit , please review it.` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (Invoked using PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. ### Other keywords and placeholders - Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. - Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description. - Add `@coderabbitai` anywhere in the PR title to generate the title automatically. ### Documentation and Community - Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit. - Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

cmeesters commented 2 days ago

@johanneskoester , @dlaehnemann , @fgvieira - I would be happy if at least two of you agreed on the changes in this PR. There are a number of design decisions and coding decisions to consider.

I want all SLURM logs of successful jobs to be auto-deleted. These are just too many files. Users can turn this behaviour off.
Ultimately, all SLURM logs shall be deleted. After 10 days is the default. No reason to keep old SLURM logs: The job logs are, after all, partial Snakemake logs as Snakemake is the job script. And the Snakemake logs are unaffected by this.
Per default, all SLURM logs will be written to ~/.snakemake/slurm_logs. This is a constant which allows the previous feature to work. Users may overwrite this behaviour.
The clean-up method is triggered at exit. For this, the atexit module is just. Triggering by __del__ has its own issues. For atexit.register() to work, the clean-up methods needs to proceed the __post_init__ method. Not nice, but stable.

snakemake / snakemake-executor-plugin-slurm