Remove python docstrings and Single-line comments

thecurz commented 2 months ago

Summary

Solves #55
Fix for incorrectly detected single-line comments inside string literals
Implements docstring removal fixing errors with previous implementation
Adds various testcases to validate implementation
Altered files

src/core/file/fileManipulator.ts: I added the methods removeDocStrings() and removeHashComments() to be used in removeComments() so that it's easier to implement an option to only delete certain types of comments later on. tests/core/file/fileManipulator.test.ts: There are now 16 tests for Python comment removal.

Observations

Certain edge cases were not considered to avoid complexity. For example in the code
```
""" docstring1""" """ docstring 2
"""
```
only the first docstring is removed. But this is not standard practice in Python

Time Complexity
removeDocStrings Function: *O(m n)** where 'm' is the number of lines and 'n' is the average length of each line
removeHashComments Function: *O((m n) + m log m)**
Benchmark

For a better idea of the time complexity, I ran a quick test where the algorithm was applied to a repeated string of Python code containing several docstrings and comments. For 3 million lines, the algorithm ran in around one second on my computer. This is more likely than not a decent time since LLMs won't take this many tokens anyway.

Next steps
It could be useful to add an option to only remove docstrings or only single-line comments since docstrings could add useful context to the prompt but not the single-line comments.
The algorithm can be further optimized if the current time complexity isn't adequate enough. I'll be happy to help with the implementation of any of these!

Summary by CodeRabbit

New Features
- Enhanced comment removal capabilities in Python code with new methods to remove docstrings and hash comments.
Bug Fixes
- Improved handling of various edge cases in comment removal, including nested quotes and inline comments.
Tests
- Expanded test cases for comment and docstring removal, covering a wider range of scenarios to ensure robustness and readability.

coderabbitai[bot] commented 2 months ago

Walkthrough

The changes introduced enhance the PythonManipulator class in the fileManipulator.ts file by adding methods for removing Python comments and docstrings. Specifically, two new methods, removeDocStrings and removeHashComments, are implemented to handle multi-line docstrings and single-line comments, respectively. The existing removeComments method is modified to call these new methods. Additionally, the test cases in fileManipulator.test.ts are expanded to cover various edge cases related to comment and string handling.

Changes

Files	Change Summary
`src/core/file/fileManipulator.ts`	- Added: `removeDocStrings(content: string): string` - Added: `removeHashComments(content: string): string` - Modified: `removeComments(content: string): string` to include calls to the new methods.
`tests/core/file/fileManipulator.test.ts`	- Updated test case names for clarity. - Expanded test cases for comment and docstring removal, covering various edge cases including nested quotes and mixed comment types.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: -- `I pushed a fix in commit , please review it.` -- `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: -- `@coderabbitai generate unit testing code for this file.` -- `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: -- `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.` -- `@coderabbitai read src/utils.ts and generate unit testing code.` -- `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` -- `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (Invoked using PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. ### Other keywords and placeholders - Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. - Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description. - Add `@coderabbitai` anywhere in the PR title to generate the title automatically. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

[!TIP]

Early access features: enabled
We are currently testing new code review model(s) that may lead to higher noise levels in the review comments. Please disable the early access features if the noise level causes any inconvenience. Note: - You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file.

yamadashy commented 2 months ago

Thank you for this impressive work on improving our Python comment removal functionality. I really appreciate the effort you've put into this.

I'll take some time to review the changes in detail. Your work looks promising, and I'm looking forward to diving into it.

I've made a few commits to address some lint errors that came up. Please take a look at these changes when you have a moment.

yamadashy commented 2 months ago

HI, @thecurz ! Thank you for your excellent work on improving our Python comment removal functionality. I've reviewed the code, and it looks great.

Your attention to detail and the comprehensive test cases you've added are much appreciated.

I'm going to go ahead and merge this PR. Thanks again for your valuable contribution!

On a related note, I've noticed that our fileManipulator is growing quite large. We might consider splitting it into separate files in the future for better maintainability.

yamadashy / repomix