tphakala / birdnet-go

Realtime BirdNET soundscape analyzer
Other
135 stars 13 forks source link

feat: New "priority" cleanup policy #180

Closed tphakala closed 1 month ago

tphakala commented 1 month ago

The priority disk cleanup feature purges recordings based on several policy rules. This helps maintain disk usage within user-defined limits while retaining valuable recordings.

This retention policy will be enabled by default.

  1. Configurable Threshold:

    • Cleanup starts when disk usage exceeds a user-defined threshold (e.g., 80%).
  2. Sorting Policies:

    • Oldest Files First: The primary criterion is to delete the oldest recordings first.
    • Species Occurrences: Files are prioritized for deletion based on the number of occurrences of each species, with species having the most recordings prioritized to be deleted first as these are often least interesting to user
    • Confidence Level: Files with lower confidence levels are prioritized for deletion over those with higher confidence levels.
    • Default to Oldest Timestamp: If other criteria are equal, the oldest files are prioritized.
  3. Minimum Clips Per Species:

    • Ensures that a minimum number of recordings per species are retained for each month.
    • This attempts to spare rare recordings from deletion
  4. Controlled Deletion Process:

    • Stops deleting files once disk usage falls below the threshold, each cleanup cycle is limited 1000 deletions
    • Cleanup cycle runs at 5 minute intervals
    • QuitChannel is monitored and cleanup loop exits cleanly if application exit is requested
coderabbitai[bot] commented 1 month ago

Walkthrough

The recent changes enhance the audio clip retention system by introducing new cleanup modes based on age and priority. These modifications involve updating configuration settings, adding utility functions for error handling, and implementing age and priority-based cleanup capabilities in the diskmanager package. The ClipCleanupMonitor function has been adjusted to delegate cleanup operations according to the selected mode.

Changes

File Path Change Summary
internal/analysis/realtime.go Modified ClipCleanupMonitor to delegate cleanup to diskmanager based on mode (age or priority).
internal/conf/config.go Updated Settings struct to include Mode and DiskUsageLimit fields.
internal/conf/config.yaml Altered configuration for clip retention, including new eviction modes and disk usage limits.
internal/conf/defaults.go Updated default settings for audio export retention, adding new fields for retention policies.
internal/conf/utils.go Added ParsePercentage function and included errors and strconv packages for utility purposes.
internal/diskmanager/age.go Introduced functionality for age-based cleanup of clips.
internal/diskmanager/priority.go Added priority-based cleanup, including policy loading, disk usage calculation, and file sorting.
internal/diskmanager/util.go Added WriteSortedFilesToFile function to write sorted files to a text file for investigation.

Sequence Diagram(s) (Beta)

sequenceDiagram
    participant User
    participant Config
    participant ClipCleanupMonitor
    participant DiskManager

    User->>Config: Load Settings
    Config->>ClipCleanupMonitor: Provide Settings
    ClipCleanupMonitor->>DiskManager: Delegate Cleanup (based on mode)
    DiskManager->>DiskManager: Perform Age-Based Cleanup
    DiskManager->>DiskManager: Perform Priority-Based Cleanup

In fields of code, where bytes do roam, We clean with age and priority's tome. Disk space saved, our data bright, Retention policies set just right. A rabbit's work, both day and night, To keep the system running light.

[!TIP]

New Features and Improvements ## Review Settings Introduced new personality profiles for code reviews. Users can now select between "Chill" and "Assertive" review tones to tailor feedback styles according to their preferences. The "Assertive" profile posts more comments and nitpicks the code more aggressively, while the "Chill" profile is more relaxed and posts fewer comments. ## AST-based Instructions CodeRabbit offers customizing reviews based on the Abstract Syntax Tree (AST) pattern matching. Read more about AST-based instructions in the [documentation](https://docs.coderabbit.ai/guides/review-instructions#ast-based). ## Community-driven AST-based Rules We are kicking off a community-driven initiative to create and share AST-based rules. Users can now contribute their AST-based rules to detect security vulnerabilities, code smells, and anti-patterns. Please see the [ast-grep-essentials](https://github.com/coderabbitai/ast-grep-essentials) repository for more information. ## New Static Analysis Tools We are continually expanding our support for static analysis tools. We have added support for `biome`, `hadolint`, and `ast-grep`. Update the settings in your `.coderabbit.yaml` file or head over to the settings page to enable or disable the tools you want to use. ## Tone Settings Users can now customize CodeRabbit to review code in the style of their favorite characters or personalities. Here are some of our favorite examples: - Mr. T: "You must talk like Mr. T in all your code reviews. I pity the fool who doesn't!" - Pirate: "Arr, matey! Ye must talk like a pirate in all yer code reviews. Yarrr!" - Snarky: "You must be snarky in all your code reviews. Snark, snark, snark!" ## Revamped Settings Page We have redesigned the settings page for a more intuitive layout, enabling users to find and adjust settings quickly. This change was long overdue; it not only improves the user experience but also allows our development team to add more settings in the future with ease. Going forward, the changes to `.coderabbit.yaml` will be reflected in the settings page, and vice versa. ## Miscellaneous - Turn off free summarization: You can switch off free summarization of PRs opened by users not on a paid plan using the `enable_free_tier` setting. - Knowledge-base scope: You can now set the scope of the knowledge base to either the repository (`local`) or the organization (`global`) level using the `knowledge_base` setting. In addition, you can specify Jira project keys and Linear team keys to limit the knowledge base scope for those integrations. - High-level summary placement: You can now customize the location of the high-level summary in the PR description using the `high_level_summary_placeholder` setting (default `@coderabbitai summary`). - Revamped request changes workflow: You can now configure CodeRabbit to auto-approve or request changes on PRs based on the review feedback using the `request_changes_workflow` setting.

[!TIP]

Early Access Features - `gpt-4o` model for chat

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share - [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)
Tips ### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to full the review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.