Devops

[NxPKG]

User description

Notes for Reviewers

This PR fixes #

Signed commits

• [*] Yes, I signed my commits.

---

PR Type

enhancement, tests, documentation

---

Description

• Refactored HTML and JavaScript in the Wordlist addition page for improved readability and consistency.
• Added a new data column for h1_team_handle in the target list and conditional display for HackerOne handle.
• Converted static methods to instance methods in serializers to enhance code maintainability.
• Updated task functions in tasks.py with the Reconpoint keyword for better clarity.
• Enhanced test cases with logging and added placeholders for future tests.
• Improved JavaScript functions in bountyhub.js for better clarity and functionality.
• Added a new GitHub Actions workflow for deploying the README to GitHub Pages.
• Updated the SECURITY.md file with new vulnerability reports and improved formatting.

---

Changes walkthrough 📝

	Relevant files
Enhancement	add.html Refactor HTML and JavaScript for Wordlist Addition Page    web/scanEngine/templates/scanEngine/wordlist/add.html Reformatted HTML structure for better readability. Improved consistency in HTML tag usage. Simplified JavaScript event listeners. +48/-53  list.html Add HackerOne Handle Display in Target List                            web/targetApp/templates/target/list.html Added new data column for h1_team_handle. Included conditional display for HackerOne handle in bug bounty mode. +9/-1      serializers.py Refactor Serializer Methods to Instance Methods                    web/api/serializers.py Converted static methods to instance methods in serializers. Improved code readability and maintainability. +73/-128 tasks.py Update Task Functions with Reconpoint Keyword                        web/reconPoint/tasks.py Added Reconpoint keyword in multiple task functions. Improved code comments for clarity. +15/-16  bountyhub.js Improve JavaScript Functions for Bounty Hub                            web/static/custom/bountyhub.js Renamed function for clarity. Enhanced asset accordion creation with additional parameters. +11/-6
Tests	test_scan.py Enhance Test Cases with Logging and Future Tests                  web/tests/test_scan.py Added logging import for test cases. Included commented-out test cases for future use. +15/-1
Configuration changes	github-pages-deploy.yml Add GitHub Pages Deployment Workflow                                          .github/workflows/github-pages-deploy.yml Added new GitHub Actions workflow for deploying README to GitHub Pages. +58/-0
Documentation	SECURITY.md Update Security Policy and Vulnerability Reports                  .github/SECURITY.md Updated formatting for consistency. Added new security vulnerability reports. +23/-22

---

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

Summary by Sourcery

Refactor the API serializers to use instance methods, improve issue templates and HTML templates for better user experience, and enhance JavaScript alert handling. Sanitize domain inputs in API views and refactor task functions for better readability. Update security documentation and improve Docker Compose and Celery configurations. Add a GitHub Actions workflow for deploying documentation to GitHub Pages. Remove deprecated workflows and configurations.

Enhancements:

• Refactor serializers in the API to use instance methods instead of static methods for better encapsulation and potential future extensibility.
• Improve the bug report issue template by adding required validations and enhancing the structure for better user guidance.
• Enhance the wordlist addition HTML template for better readability and maintainability by restructuring the HTML blocks.
• Refactor JavaScript functions in custom_domain.js to use Swal.fire for better alert handling and user experience.
• Improve the handling of domain names in the API views by sanitizing input to remove wildcards and leading dots.
• Refactor the task functions in tasks.py to remove unnecessary static method usage and improve code readability.
• Enhance the security documentation by formatting and aligning the text for better readability.
• Improve the CONTRIBUTING.md file by adding spacing and formatting for better readability and guidance.
• Refactor the Docker Compose files to use a consistent image naming convention for better clarity.
• Enhance the Celery entrypoint script by correcting the app and directory paths for better reliability.
• Improve the handling of HackerOne API keys in the dashboard views to prevent potential null reference errors.
• Refactor the database utility functions to include return types for better code clarity and maintainability.
• Enhance the InstalledExternalTool model by updating the default regex pattern for version matching.
• Improve the scan engine management commands by converting static methods to instance methods for better encapsulation.
• Add a GitHub Actions workflow to deploy the README to GitHub Pages for better documentation accessibility.

CI:

• Add a GitHub Actions workflow to deploy the README to GitHub Pages, enhancing documentation accessibility.

Documentation:

• Update the SECURITY.md file to improve formatting and readability.
• Enhance the CONTRIBUTORS.md file by updating the formatting for better readability.

Tests:

• Add new test cases in test_scan.py to cover additional functionalities and improve test coverage.

Chores:

• Remove deprecated GitHub Actions workflows and configuration files to clean up the repository.

[gitguardian[bot]]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

| GitGuardian id | GitGuardian status | Secret | Commit | Filename | | | -------------- | ------------------ | ------------------------------ | ---------------- | --------------- | -------------------- | | [13037857](https://dashboard.gitguardian.com/workspace/515913/incidents/13037857?occurrence=166866739) | Triggered | Username Password | 9d09c55e1e4d77806de036297451d4b25d308f42 | .env | [View secret](https://github.com/khulnasoft/reconpoint/commit/9d09c55e1e4d77806de036297451d4b25d308f42#diff-e9cbb0224c4a3d23a6019ba557e0cd568c1ad5e1582ff1e335fb7d99b7a1055dR37) |

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code. 2. Replace and store your secret safely. [Learn here](https://blog.gitguardian.com/secrets-api-management?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) the best practices. 3. Revoke and [rotate this secret](https://docs.gitguardian.com/secrets-detection/secrets-detection-engine/detectors/generics/username_password#revoke-the-secret?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). 4. If possible, [rewrite git history](https://blog.gitguardian.com/rewriting-git-history-cheatsheet?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data. To avoid such incidents in the future consider - following these [best practices](https://blog.gitguardian.com/secrets-api-management/?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) for managing and storing secrets including API keys and other credentials - install [secret detection on pre-commit](https://docs.gitguardian.com/ggshield-docs/integrations/git-hooks/pre-commit?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[sourcery-ai[bot]]

Reviewer's Guide by Sourcery

This pull request implements various DevOps-related changes across the reconPoint project. The changes include updates to configuration files, code refactoring, dependency updates, and modifications to CI/CD workflows. Key areas affected include YAML configurations, API views, models, templates, and GitHub Actions workflows.

Architecture diagram for updated CI/CD workflows

graph TD;
    A[GitHub Repository] -->|Push to master| B[Build and Push Docker Image]
    A -->|Pull Request| C[Build Docker Image for PR]
    A -->|Push to master| D[Deploy README to GitHub Pages]
    B --> E[Docker Hub]
    C --> E
    D --> F[GitHub Pages]

Updated class diagram for serializers in web/api

classDiagram
    class HackerOneProgramAttributesSerializer {
        +BooleanField fast_payments
        +BooleanField gold_standard_safe_harbor
        +to_representation(self, instance)
    }
    class Meta {
        +get_project_name(self, obj)
        +get_vuln_count(self, obj)
        +get_organization(self, obj)
        +get_most_recent_scan(self, obj)
        +get_insert_date(self, obj)
        +get_insert_date_humanized(self, obj)
        +get_start_scan_date(self, obj)
        +get_start_scan_date_humanized(self, obj)
    }
    class TodoNote {
        +get_domain_name(self, note)
        +get_subdomain_name(self, note)
        +get_scan_started_time(self, note)
    }
    class SubScan {
        +get_subdomain_name(self, subscan)
        +get_total_time_taken(self, subscan)
        +get_elapsed_time(self, subscan)
        +get_completed_ago(self, subscan)
        +get_engine_name(self, subscan)
    }
    class ScanHistory {
        +get_subdomain_count(self, scan_history)
        +get_endpoint_count(self, scan_history)
        +get_vulnerability_count(self, scan_history)
        +get_progress(self, scan_history)
        +get_total_scan_time_in_sec(self, scan_history)
        +get_elapsed_time(self, scan_history)
        +get_completed_ago(self, scan_history)
        +get_organizations(self, scan_history)
    }
    class EngineSerializer {
        +get_tasks(self, instance)
    }
    class VulnerabilitySerializer {
        +get_discovered_date(self, Vulnerability)
        +get_severity(self, Vulnerability)
    }

File-Level Changes

Change	Details	Files
Refactored YAML configuration structure and formatting	Updated indentation and formatting for better readability Converted some dictionary structures to use JSON-like formatting Added new configuration options for various tools and scans	default_yaml_config.yaml
Refactored API views and serializers	Removed @staticmethod decorators from several methods Updated method signatures to include 'self' parameter Modified some API endpoints and their logic	web/api/serializers.py web/api/views.py
Updated GitHub Actions workflows	Modified build and deployment workflows Added new workflow for deploying README to GitHub Pages Removed some outdated workflows	.github/workflows/build.yml .github/workflows/build-pr.yml .github/workflows/github-pages-deploy.yml
Refactored models and database-related code	Updated model fields and relationships Modified database migration files Refactored database utility functions	web/startScan/models.py web/targetApp/models.py web/reconPoint/database_utils.py
Updated Docker-related configurations	Modified Docker image references Updated environment variables and configurations in docker-compose files	docker-compose.dev.yml docker-compose.yml
Refactored and updated various utility functions and tasks	Modified Celery task implementations Updated common utility functions Refactored scan engine and tool-specific code	web/reconPoint/celery_custom_task.py web/reconPoint/common_func.py web/scanEngine/views.py
Updated templates and frontend-related code	Modified HTML templates for better structure and conditional rendering Updated JavaScript files for improved functionality Refactored CSS and styling-related code	web/targetApp/templates/target/list.html web/targetApp/templates/target/add.html web/static/custom/update.js

---

Tips and commands

#### Interacting with Sourcery - **Trigger a new review:** Comment `@sourcery-ai review` on the pull request. - **Continue discussions:** Reply directly to Sourcery's review comments. - **Generate a GitHub issue from a review comment:** Ask Sourcery to create an issue from a review comment by replying to it. #### Customizing Your Experience Access your [dashboard](https://app.sourcery.ai) to: - Enable or disable review features such as the Sourcery-generated pull request summary, the reviewer's guide, and others. - Change the review language. - Add, remove or edit custom review instructions. - Adjust other review settings. #### Getting Help - [Contact our support team](mailto:support@sourcery.ai) for questions or feedback. - Visit our [documentation](https://docs.sourcery.ai) for detailed guides and information. - Keep in touch with the Sourcery team by following us on [X/Twitter](https://x.com/SourceryAI), [LinkedIn](https://www.linkedin.com/company/sourcery-ai/) or [GitHub](https://github.com/sourcery-ai).

[github-actions[bot]]

Woohoo @NxPKG! 🎉 You've just dropped some hot new code! 🔥

Hang tight while we review this! You rock! 🤘

[github-actions[bot]]

Woohoo @NxPKG! 🎉 You've just dropped some hot new code! 🔥

Hang tight while we review this! You rock! 🤘

[deepsource-io[bot]]

Here's the code health analysis summary for commits b9ed7dd..abef68b. View details on DeepSource ↗.

Analysis Summary

Analyzer	Status	Summary	Link
Python	❌ Failure	❗ 169 occurences introduced 🎯 7 occurences resolved	View Check ↗
Docker	❌ Failure	❗ 8 occurences introduced 🎯 6 occurences resolved	View Check ↗

---

💡 If you’re a repository administrator, you can configure the quality gates from the settings.

[codiumai-pr-agent-free[bot]]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Security Issue The bulk_import_targets function is called without proper input validation, which could potentially lead to security vulnerabilities if malicious input is provided. Potential Bug The delete_targets function may not be properly deleting all selected targets due to incorrect handling of request.POST items. Performance Concern The tool_specific_settings function is using glob to list files, which may be inefficient for large directories. Consider using os.scandir for better performance.

[codiumai-pr-agent-free[bot]]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Score
Enhancement	Use a dictionary mapping for severity levels instead of multiple if-elif statements ___ **In the VisualiseSubdomainSerializer, consider using a dictionary mapping for severity levels instead of multiple if-elif statements. This would make the code more maintainable and easier to extend if new severity levels are added in the future.** [web/api/serializers.py [959-971]](https://github.com/khulnasoft/reconpoint/pull/77/files#diff-75f649b54bbbc604765490a0eae8c755b7fa5d2db6b1ee968802c6a380e4d699R959-R971) ```diff +SEVERITY_MAP = { + 0: "Info", + 1: "Low", + 2: "Medium", + 3: "High", + 4: "Critical" +} + def get_severity(self, Vulnerability): - if Vulnerability.severity == 0: - return "Info" - elif Vulnerability.severity == 1: - return "Low" - elif Vulnerability.severity == 2: - return "Medium" - elif Vulnerability.severity == 3: - return "High" - elif Vulnerability.severity == 4: - return "Critical" - else: - return "Unknown" + return SEVERITY_MAP.get(Vulnerability.severity, "Unknown") ``` - [ ] **Apply this suggestion** Suggestion importance[1-10]: 9 Why: The suggestion to use a dictionary for mapping severity levels is excellent. It simplifies the code, enhances maintainability, and makes it easier to extend if new severity levels are added in the future.	9
	✅ Remove unnecessary keyword from comment ___ Suggestion Impact: The 'Reconpoint' keyword was removed from the comment, as suggested. code diff: ```diff - # Get URLs to take screenshot ofReconpoint + # Get URLs to take screenshot of ``` ___ **Remove the 'Reconpoint' keyword that was added at the end of the comment. It appears to be unintentionally placed and doesn't provide any additional information to the comment.** [web/reconPoint/tasks.py [1196]](https://github.com/khulnasoft/reconpoint/pull/77/files#diff-6652e50bee8c18604878210d537df3622ff8207eb68b8febb5894b40ccd406adR1196-R1196) ```diff -# Get URLs to take screenshot ofReconpoint +# Get URLs to take screenshot of ``` - [ ] **Apply this suggestion** Suggestion importance[1-10]: 8 Why: The suggestion correctly identifies an unnecessary 'Reconpoint' keyword in a comment, which does not add any value. Removing it improves comment clarity and code readability.	8
Possible issue	✅ Remove unnecessary keyword from function parameter ___ Suggestion Impact: The 'Reconpoint' keyword was removed from the function parameter, aligning with the suggestion. code diff: ```diff - host=None,Reconpoint + host=None, ``` ___ **Remove the 'Reconpoint' keyword from the function parameter as it seems to be unintentionally added and doesn't serve any purpose. This will maintain consistency with the function signature and prevent potential errors.** [web/reconPoint/tasks.py [396-400]](https://github.com/khulnasoft/reconpoint/pull/77/files#diff-6652e50bee8c18604878210d537df3622ff8207eb68b8febb5894b40ccd406adR396-R400) ```diff def subdomain_discovery( self, - host=None,Reconpoint + host=None, ctx=None, description=None): ``` - [ ] **Apply this suggestion** Suggestion importance[1-10]: 9 Why: The suggestion correctly identifies an unnecessary keyword 'Reconpoint' in the function parameter, which could lead to syntax errors. Removing it improves the function's correctness and maintainability.	9
	✅ Remove unnecessary keyword from function docstring ___ Suggestion Impact: The 'Reconpoint' keyword was removed from the function's docstring, as suggested. code diff: ```diff -Reconpoint + Args: ``` ___ **Remove the 'Reconpoint' keyword that was added at the end of the line. It appears to be unintentionally placed and doesn't serve any purpose in the function's docstring.** [web/reconPoint/tasks.py [631-634]](https://github.com/khulnasoft/reconpoint/pull/77/files#diff-6652e50bee8c18604878210d537df3622ff8207eb68b8febb5894b40ccd406adR631-R634) ```diff """Run Open-Source Intelligence tools on selected domain. -Reconpoint Args: host (str): Hostname to scan. ``` - [ ] **Apply this suggestion** Suggestion importance[1-10]: 9 Why: The suggestion accurately points out an extraneous 'Reconpoint' keyword in the docstring, which serves no purpose and could confuse developers. Removing it enhances code clarity.	9
Maintainability	✅ Standardize naming conventions for consistency and clarity ___ Suggestion Impact: The commit addressed the inconsistency in naming conventions by changing "reconPointnPoint" to "reconPoint" and "reconPointint" to "reconPoint" in the script, aligning with the suggestion to standardize naming. code diff: ```diff - local app=${4:-"reconPointint.tasks"} - local directory=${5:-"/usr/src/app/reconPointint/"} + local app=${4:-"reconPoint.tasks"} + local directory=${5:-"/usr/src/app/reconPoint/"} local base_command="celery -A $app worker --pool=gevent --optimization=fair --autoscale=$concurrency,1 --loglevel=$loglevel -Q $queue -n $worker_name" @@ -220,9 +220,9 @@ # Main scan worker if [ "$DEBUG" == "1" ]; then - commands+="watchmedo auto-restart --recursive --pattern=\"*.py\" --directory=\"/usr/src/app/reconPointint/\" -- celery reconPointnPoint.tasks worker --loglevel=$loglevel --optimization=fair --autoscale=$MAX_CONCURRENCY,$MIN_CONCURRENCY -Q main_scan_queue &"$'\n' + commands+="watchmedo auto-restart --recursive --pattern=\"*.py\" --directory=\"/usr/src/app/reconPoint/\" -- celery -A reconPoint.tasks worker --loglevel=$loglevel --optimization=fair --autoscale=$MAX_CONCURRENCY,$MIN_CONCURRENCY -Q main_scan_queue &"$'\n' else - commands+="celery -A reconPointint.tasks worker --loglevel=$loglevel --optimization=fair --autoscale=$MAX_CONCURRENCY,$MIN_CONCURRENCY -Q main_scan_queue &"$'\n' + commands+="celery -A reconPoint.tasks worker --loglevel=$loglevel --optimization=fair --autoscale=$MAX_CONCURRENCY,$MIN_CONCURRENCY -Q main_scan_queue &"$'\n' ``` ___ **Consider using consistent naming conventions for variables. The script uses both reconPointint and reconPointnPoint in different places. Standardizing on one naming convention would improve code consistency and reduce potential confusion.** [web/celery-entrypoint.sh [223]](https://github.com/khulnasoft/reconpoint/pull/77/files#diff-a3dee63a39c856932f86794664d3e827bc94381bd4ff662d97a89b5c2fc26f1aR223-R223) ```diff -commands+="watchmedo auto-restart --recursive --pattern=\"*.py\" --directory=\"/usr/src/app/reconPointint/\" -- celery reconPointnPoint.tasks worker --loglevel=$loglevel --optimization=fair --autoscale=$MAX_CONCURRENCY,$MIN_CONCURRENCY -Q main_scan_queue &"$'\n' +commands+="watchmedo auto-restart --recursive --pattern=\"*.py\" --directory=\"/usr/src/app/reconPointint/\" -- celery reconPointint.tasks worker --loglevel=$loglevel --optimization=fair --autoscale=$MAX_CONCURRENCY,$MIN_CONCURRENCY -Q main_scan_queue &"$'\n' ``` - [ ] **Apply this suggestion** Suggestion importance[1-10]: 8 Why: The suggestion addresses an inconsistency in naming conventions within the script, which could lead to confusion. Correcting this improves maintainability and clarity, making it a valuable enhancement.	8
	Use more descriptive method names to accurately reflect their functionality ___ **Consider using a more descriptive name for the get_description method in cases where it's not actually returning a description. For example, in the VisualiseIpSerializer, it's returning an IP address.** [web/api/serializers.py [374-375]](https://github.com/khulnasoft/reconpoint/pull/77/files#diff-75f649b54bbbc604765490a0eae8c755b7fa5d2db6b1ee968802c6a380e4d699R374-R375) ```diff -def get_description(self, Ip): +def get_ip_address(self, Ip): return Ip.address ``` - [ ] **Apply this suggestion** Suggestion importance[1-10]: 7 Why: The suggestion to rename the method to better reflect its functionality is valid and improves code readability. It accurately identifies that the method name should match its purpose, which is to return an IP address, not a description.	7
	Remove unused commented code ___ **Remove the commented-out line # data = req.data as it's not being used and may cause confusion. If this line is needed for future use, consider adding a more descriptive comment explaining its purpose.** [web/api/views.py [960-963]](https://github.com/khulnasoft/reconpoint/pull/77/files#diff-525e6a8b8c6d97257a0fb87bb0119921b99e6d0b8dd43395c28f425c3c4072bcR960-R963) ```diff def get(self, request): req = self.request - # data = req.data subscan_id = req.query_params.get('subscan_id') ``` - [ ] **Apply this suggestion** Suggestion importance[1-10]: 7 Why: The suggestion proposes removing an unused commented-out line, which can help reduce clutter and potential confusion in the code. This enhances maintainability.	7
	Use more descriptive variable names to improve code readability ___ **Consider using a more descriptive variable name instead of deleteForm. A name like multipleTargetsDeleteForm would be more specific and easier to understand in the context of the function.** [web/targetApp/static/targetApp/js/custom_domain.js [95-97]](https://github.com/khulnasoft/reconpoint/pull/77/files#diff-72ce156d863f684282bc270d8706ecc0ee45130b965e13d2898ac589f9da3f5dR95-R97) ```diff -const deleteForm = document.getElementById("multiple_targets_form"); -deleteForm.action = `/target/${slug}/delete/multiple`; -deleteForm.submit(); +const multipleTargetsDeleteForm = document.getElementById("multiple_targets_form"); +multipleTargetsDeleteForm.action = `/target/${slug}/delete/multiple`; +multipleTargetsDeleteForm.submit(); ``` - [ ] **Apply this suggestion** Suggestion importance[1-10]: 7 Why: The suggestion to rename 'deleteForm' to 'multipleTargetsDeleteForm' enhances code readability by providing a more descriptive variable name, which aligns with best practices for maintainability.	7

💡 Need additional feedback ? start a PR chat

[github-actions[bot]]

Hey, thanks for your contribution! 🙏

We appreciate the time and effort you put into this PR. Sadly this is not the right fit for reconPoint at the moment.

While we couldn't merge it this time, we value your interest in improving reconPoint.

Feel free to reach out if you have any questions. Thanks again!