Implement log caching and optimize log retrieval

[sweep-ai[bot]]

This pull request was created by Sweep to resolve the following request by @KristopherKubicki:

The logs are only just that one file and it can get pretty big. Should the log endpoint maybe cache the logs (in the background) and then make them searchable from memory? No need to have them in th db. Some rows are very long and can be truncated for sure

Continue chatting at https://sweep-chat-demo.vercel.app/c/a4f27424-2b11-41b1-8768-15425664109e.

Purpose

This pull request aims to improve the performance and reliability of the log retrieval functionality in the Glimpser application.

Description

The key changes in this pull request are:

1. Implemented a log caching mechanism to store the last 10,000 log entries in memory, reducing the need to read from the log file on every request.
2. Optimized the read_logs_from_memory function to filter the logs directly from the in-memory cache, rather than reading from the file.
3. Added a background thread to continuously monitor the log file and update the in-memory cache with new log entries.
4. Refactored the read_logs_from_file function to read_logs_from_memory to reflect the new caching implementation.

Summary

• Implemented log_cache and log_cache_lock to store and manage the in-memory log cache
• Added cache_logs function to continuously update the log cache from the log file
• Refactored read_logs_from_file to read_logs_from_memory to use the in-memory cache
• Optimized log retrieval by filtering logs directly from the in-memory cache
• Added start_log_caching function to start the log caching background thread

Summary by CodeRabbit

• New Features

  • Introduced an in-memory logging system for faster access to log data.
  • Added a log caching mechanism to efficiently manage recent logs.
  • Implemented a scheduled job for continuous log caching.

• Bug Fixes

  • Enhanced error handling for malformed log entries to prevent interruptions.

• Refactor

  • Removed outdated scheduling routes, streamlining the scheduling interface.

[coderabbitai[bot]]

Walkthrough

The changes involve a transition from file-based logging to an in-memory caching system for log data in the application. The read_logs_from_file function is replaced by read_logs_from_memory, which utilizes a global cache and provides filtering capabilities. Additionally, a new log caching mechanism is introduced in the scheduling module, allowing for efficient management of recent logs. The previous scheduler management routes have been removed.

Changes

File	Change Summary
app/routes.py	Replaced read_logs_from_file with read_logs_from_memory, altering log data source and filtering logic. Removed toggle_scheduler and scheduler_status routes.
app/utils/scheduling.py	Introduced a log caching system using deque for recent logs, added cache_logs and start_log_caching functions for concurrent log processing.

Poem

🐇
In the meadow where logs once lay,
Now in memory, they dance and play.
With threads that spin and caches bright,
Our logs are swift, a joyful sight!
Goodbye to files, hello to speed,
A hopping change, indeed a need!
🌼

---

[!TIP]

Early access features: enabled

We are currently testing the following features in early access: - **Anthropic `claude-3-5-sonnet` for code reviews**: The new Claude model has stronger code understanding and code generation capabilities than the previous models. We're excited to hear your feedback as we evaluate its performance over the next few days. Note: - You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file. - Please join our [Discord Community](https://discord.com/invite/GsXnASn26c) to provide feedback and report issues on the [discussion post](https://discordapp.com/channels/1134356397673414807/1279579842131787838).

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (Invoked using PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. ### Other keywords and placeholders - Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. - Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description. - Add `@coderabbitai` anywhere in the PR title to generate the title automatically. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

[github-actions[bot]]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details

Scanned Manifest Files