Hu-Fi / Mr.Market

Mr. Market is the exchange oracle of HuFi, and a CeFi crypto bot on Mixin Messenger
https://mr-market-one.vercel.app
GNU Affero General Public License v3.0
1 stars 6 forks source link

Add updated recording oracle #90

Closed Faouzijedidi1 closed 3 months ago

Faouzijedidi1 commented 3 months ago

Type

enhancement


Description


Changes walkthrough

Relevant files
Enhancement
8 files
records.service.ts
Implement Liquidity Score Calculation and Updates               

recording-oracle/src/records/records.service.ts
  • Added RecordService with methods for fetching trades, open orders,
    order book, and calculating liquidity score.
  • Implemented calculateLiquidityScore method to calculate liquidity
    score based on trade volume, open order volume, average duration, and
    spread.
  • Added a cron job to handle liquidity score updates every minute.
  • +97/-0   
    user.service.ts
    Add User Service for Sign-up and API Key Management           

    recording-oracle/src/user/user.service.ts
  • Added UserService with a signUp method to handle user sign-up
    including API key encryption and campaign association.
  • Implemented encryption for API keys and secrets using
    EncryptionService.
  • +58/-0   
    user.entity.ts
    Define User Entity                                                                             

    recording-oracle/src/common/entities/user.entity.ts
  • Defined User entity with columns for user ID, API key, secret, and
    many-to-many relationship with campaigns.
  • +21/-0   
    campaign.entity.ts
    Define Campaign Entity                                                                     

    recording-oracle/src/common/entities/campaign.entity.ts
  • Defined Campaign entity with columns for campaign address and
    relationships with users and liquidity scores.
  • +19/-0   
    encryption.service.ts
    Implement Encryption Service for API Key Security               

    recording-oracle/src/encryption/encryption.service.ts
  • Implemented EncryptionService with static methods for encrypting and
    decrypting strings.
  • +14/-0   
    liquidity-score.model.ts
    Add Liquidity Score Calculation Model                                       

    recording-oracle/src/records/liquidity-score.model.ts
  • Created LiquidityScoreCalculation model with a method to calculate
    liquidity score.
  • +12/-0   
    user.controller.ts
    Implement User Controller for Sign-up Endpoint                     

    recording-oracle/src/user/user.controller.ts
  • Added UserController with a signUp endpoint to handle user
    registration.
  • +15/-0   
    liquidity-score.entity.ts
    Define Liquidity Score Entity                                                       

    recording-oracle/src/common/entities/liquidity-score.entity.ts
  • Defined LiquidityScore entity with columns for score, calculation
    timestamp, and relationship with campaign.
  • +18/-0   
    Configuration changes
    2 files
    app.module.ts
    Configure AppModule with Necessary Modules and Services   

    recording-oracle/src/app.module.ts
  • Setup AppModule with imports for ConfigModule, TypeOrmModule, and
    RecordsModule.
  • Added EncryptionService and UserService to providers.
  • +28/-0   
    package.json
    Configure Package.json with Dependencies and Scripts         

    recording-oracle/package.json
  • Added project dependencies including NestJS, TypeORM, ccxt, and
    crypto-js.
  • Configured scripts for building, starting, and testing the
    application.
  • +77/-0   
    Documentation
    1 files
    README.md
    Add README for Project Documentation                                         

    recording-oracle/README.md
  • Added README with project description, installation, running, and
    testing instructions.
  • +73/-0   

    PR-Agent usage: Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

    vercel[bot] commented 3 months ago

    The latest updates on your projects. Learn more about Vercel for Git ↗︎

    Name Status Preview Comments Updated (UTC)
    mr-market ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 25, 2024 3:13am
    railway-app[bot] commented 3 months ago

    This PR is being deployed to Railway 🚅

    Mr.Market: ◻️ REMOVED

    github-actions[bot] commented 3 months ago

    PR Description updated to latest commit (https://github.com/Hu-Fi/Mr.Market/commit/e84a1d9e063d36112fe5b3017f4df865ebd71d18)

    github-actions[bot] commented 3 months ago

    PR Review

    ⏱️ Estimated effort to review [1-5] 4, due to the comprehensive nature of the changes across multiple files, including service implementations, entity definitions, and configuration setups. The PR introduces significant functionality, touching on database interactions, encryption, and external API integration, which requires careful review to ensure correctness, security, and performance.
    🧪 Relevant tests Yes
    🔍 Possible issues Possible Security Concern: The `EncryptionService` uses a static method to encrypt and decrypt API keys, relying on an environment variable for the encryption key. If the environment variable is compromised, all encrypted data could be at risk. Consider using a more dynamic approach to key management.
    Performance Concern: The `RecordService` performs multiple asynchronous operations in sequence within `calculateLiquidityScore` and `processOpenOrders`. Depending on the number of users and campaigns, this could lead to performance bottlenecks. Consider optimizing these operations, possibly through parallel processing or caching strategies.
    🔒 Security concerns Sensitive information exposure: The PR includes handling of API keys and secrets, which are sensitive pieces of information. While encryption is used, the security of the encryption depends heavily on the management of the encryption key (`ENCRYPTION_KEY`). Ensure that this key is stored securely and not exposed in the codebase or logs.
    Code feedback:
    relevant filerecording-oracle/src/records/records.service.ts
    suggestion       Consider implementing error handling for the external API calls within `fetchTrades`, `fetchOpenOrders`, and `fetchOrderBook`. This can prevent the entire liquidity score calculation from failing due to temporary issues with a single exchange or network problems. [important]
    relevant linereturn exchange.fetchMyTrades(symbol, since);

    relevant filerecording-oracle/src/encryption/encryption.service.ts
    suggestion       Implement a key rotation mechanism for `ENCRYPTION_KEY` to enhance security. Static keys can become a vulnerability over time. A rotation strategy can mitigate the risk of key compromise. [important]
    relevant linereturn CryptoJS.AES.encrypt(value, ENCRYPTION_KEY).toString();

    relevant filerecording-oracle/src/app.module.ts
    suggestion       The `synchronize: true` option in TypeOrmModule configuration is convenient for development but can be dangerous in production as it can lead to data loss. Consider managing database migrations separately for production environments. [important]
    relevant linesynchronize: true, // Not recommended for production

    relevant filerecording-oracle/src/user/user.service.ts
    suggestion       When checking for the existence of a user or campaign, consider adding explicit error handling or logging. This can help in diagnosing issues related to data integrity or database connectivity. [medium]
    relevant linelet campaign = await this.campaignRepository.findOneBy({ address: campaignAddress });


    ✨ Review tool usage guide:
    **Overview:** The `review` tool scans the PR code changes, and generates a PR review. The tool can be triggered [automatically](https://pr-agent-docs.codium.ai/usage-guide/automations_and_usage/#github-app-automatic-tools-when-a-new-pr-is-opened) every time a new PR is opened, or can be invoked manually by commenting on any PR. When commenting, to edit [configurations](https://github.com/Codium-ai/pr-agent/blob/main/pr_agent/settings/configuration.toml#L19) related to the review tool (`pr_reviewer` section), use the following template: ``` /review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=... ``` With a [configuration file](https://pr-agent-docs.codium.ai/usage-guide/configuration_options/), use the following template: ``` [pr_reviewer] some_config1=... some_config2=... ```
    Utilizing extra instructions
    The `review` tool can be configured with extra instructions, which can be used to guide the model to a feedback tailored to the needs of your project. Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify the relevant sub-tool, and the relevant aspects of the PR that you want to emphasize. Examples for extra instructions: ``` [pr_reviewer] # /review # extra_instructions=""" In the 'possible issues' section, emphasize the following: - Does the code logic cover relevant edge cases? - Is the code logic clear and easy to understand? - Is the code logic efficient? ... """ ``` Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.
    How to enable\disable automation
    - When you first install PR-Agent app, the [default mode](https://pr-agent-docs.codium.ai/usage-guide/automations_and_usage/#github-app-automatic-tools-when-a-new-pr-is-opened) for the `review` tool is: ``` pr_commands = ["/review", ...] ``` meaning the `review` tool will run automatically on every PR, with the default configuration. Edit this field to enable/disable the tool, or to change the used configurations
    Auto-labels
    The `review` tool can auto-generate two specific types of labels for a PR: - a `possible security issue` label, that detects possible [security issues](https://github.com/Codium-ai/pr-agent/blob/tr/user_description/pr_agent/settings/pr_reviewer_prompts.toml#L136) (`enable_review_labels_security` flag) - a `Review effort [1-5]: x` label, where x is the estimated effort to review the PR (`enable_review_labels_effort` flag)
    Extra sub-tools
    The `review` tool provides a collection of possible feedbacks about a PR. It is recommended to review the [possible options](https://pr-agent-docs.codium.ai/tools/review/#enabledisable-features), and choose the ones relevant for your use case. Some of the feature that are disabled by default are quite useful, and should be considered for enabling. For example: `require_score_review`, `require_soc2_ticket`, `require_can_be_split_review`, and more.
    Auto-approve PRs
    By invoking: ``` /review auto_approve ``` The tool will automatically approve the PR, and add a comment with the approval. To ensure safety, the auto-approval feature is disabled by default. To enable auto-approval, you need to actively set in a pre-defined configuration file the following: ``` [pr_reviewer] enable_auto_approval = true ``` (this specific flag cannot be set with a command line argument, only in the configuration file, committed to the repository) You can also enable auto-approval only if the PR meets certain requirements, such as that the `estimated_review_effort` is equal or below a certain threshold, by adjusting the flag: ``` [pr_reviewer] maximal_review_effort = 5 ```
    More PR-Agent commands
    > To invoke the PR-Agent, add a comment using one of the following commands: > - **/review**: Request a review of your Pull Request. > - **/describe**: Update the PR title and description based on the contents of the PR. > - **/improve [--extended]**: Suggest code improvements. Extended mode provides a higher quality feedback. > - **/ask \**: Ask a question about the PR. > - **/update_changelog**: Update the changelog based on the PR's contents. > - **/add_docs** 💎: Generate docstring for new components introduced in the PR. > - **/generate_labels** 💎: Generate labels for the PR based on the PR's contents. > - **/analyze** 💎: Automatically analyzes the PR, and presents changes walkthrough for each component. >See the [tools guide](https://pr-agent-docs.codium.ai/tools/) for more details. >To list the possible configuration parameters, add a **/config** comment.
    See the [review usage](https://pr-agent-docs.codium.ai/tools/review/) page for a comprehensive guide on using this tool.
    github-actions[bot] commented 3 months ago

    PR Code Suggestions

    CategorySuggestions                                                                                                                                                       
    Best practice
    Replace direct synchronize: true with an environment check to prevent data loss in production. ___ **It's recommended to avoid using synchronize: true in your TypeORM configuration for
    production environments as it can lead to data loss by altering the database schema to
    match your entities each time the application runs. Consider using migrations to manage
    your database schema changes more safely.** [recording-oracle/src/app.module.ts [21]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-e11bcc5513b1df2a7ce006f193f7490982af7abf62f211681c773166bd2b734bR21-R21) ```diff -synchronize: true, // Not recommended for production +synchronize: process.env.NODE_ENV !== 'production', ```
    Enable strict mode in TypeScript for improved type-checking. ___ **It's recommended to enable strict mode in TypeScript for a more robust type-checking which
    can catch more errors during development. This can be done by setting "strict": true in
    your compiler options. This change will enforce stricter type-checking rules, which can
    help in identifying potential issues early in the development process.** [recording-oracle/tsconfig.json [15-17]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-0738152eb60a3006c190ca1a10e19844be510384c02b72552f3fa23e75a3e7e6R15-R17) ```diff -"strictNullChecks": false, -"noImplicitAny": false, -"strictBindCallApply": false, +"strict": true, ```
    Enable consistent casing in file names to prevent potential issues. ___ **Consider enabling forceConsistentCasingInFileNames to ensure that the casing of your file
    names is consistent across your project. This can help prevent issues when working on
    case-sensitive file systems or when collaborating with other developers who may be using
    different operating systems.** [recording-oracle/tsconfig.json [18]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-0738152eb60a3006c190ca1a10e19844be510384c02b72552f3fa23e75a3e7e6R18-R18) ```diff -"forceConsistentCasingInFileNames": false, +"forceConsistentCasingInFileNames": true, ```
    Lock dependency versions for consistent builds. ___ **It's a good practice to lock the versions of your dependencies to ensure consistent builds
    and deployments. You can achieve this by removing the caret (^) symbol before the version
    numbers in your dependencies and devDependencies. This prevents automatic updates to newer
    minor versions, which might introduce unexpected changes or bugs.** [recording-oracle/package.json [22-31]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-511fb8afe5be487afda92e97e743b756c7fa3c0983b954842c348398c6f49877R22-R31) ```diff -"@nestjs/common": "^10.0.0", -"@nestjs/config": "^3.2.0", +"@nestjs/common": "10.0.0", +"@nestjs/config": "3.2.0", ```
    Enable source maps for better error tracking in production. ___ **For better debugging and error tracking, consider enabling sourceMap in your production
    builds. This allows you to map the compiled code back to your original source code. If
    you're concerned about exposing your source code, you can deploy source maps only to your
    error reporting tools rather than making them publicly accessible.** [recording-oracle/tsconfig.json [10]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-0738152eb60a3006c190ca1a10e19844be510384c02b72552f3fa23e75a3e7e6R10-R10) ```diff -"sourceMap": true, +"sourceMap": true, // Consider deploying source maps to error reporting tools only ```
    Security
    Securely manage the encryption key using a configuration service with validation. ___ **Storing the encryption key directly in the code or fetching it directly from the
    environment variables without any checks can lead to security vulnerabilities. Consider
    implementing a configuration service to securely manage environment variables and include
    validation to ensure the encryption key exists and meets any required criteria (e.g.,
    length).** [recording-oracle/src/encryption/encryption.service.ts [4]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-73c7beedd5761db4068f99096477d2e2957dc2a39129b0819cad504c7e67c028R4-R4) ```diff -const ENCRYPTION_KEY = process.env.ENCRYPTION_KEY; +const ENCRYPTION_KEY = configService.get('ENCRYPTION_KEY'); ```
    Bug
    Convert order duration from seconds to minutes to match the expected unit. ___ **The method processOpenOrders calculates the average duration in seconds but the comment in
    LiquidityScoreCalculation constructor suggests it expects minutes. Convert the duration
    from seconds to minutes to avoid inconsistencies in calculations.** [recording-oracle/src/records/records.service.ts [45]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-1818e6d68069b8c3703cda2defc212b14561322453eea709094b1e39994281a4R45-R45) ```diff -const duration = (now - orderCreationTime) / (1000); // Convert duration from milliseconds to seconds +const duration = (now - orderCreationTime) / (1000 * 60); // Convert duration from milliseconds to minutes ```
    Enhancement
    Add error handling for API key and secret encryption to improve reliability. ___ **When encrypting the API key and secret, it's important to handle potential errors that
    could arise during the encryption process. Wrap the encryption calls in a try-catch block
    to manage any exceptions and provide a clear error message or handling strategy.** [recording-oracle/src/user/user.service.ts [19-20]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-dff04937eb60173722b8270c06a215f5437ff7866f2fa7452ee843d3539edc83R19-R20) ```diff -const encryptedApiKey = EncryptionService.encrypt(apiKey); -const encryptedSecret = EncryptionService.encrypt(secret); +let encryptedApiKey, encryptedSecret; +try { + encryptedApiKey = EncryptionService.encrypt(apiKey); + encryptedSecret = EncryptionService.encrypt(secret); +} catch (error) { + // Handle encryption error + console.error("Encryption failed", error); + throw new Error("Failed to encrypt API credentials"); +} ```
    Implement error handling in calculateLiquidityScore to manage asynchronous operation failures. ___ **The calculateLiquidityScore method performs several asynchronous operations without
    catching potential errors. Wrap the asynchronous calls within a try-catch block to handle
    errors gracefully, ensuring that the application can respond appropriately to failures in
    external service calls or internal logic errors.** [recording-oracle/src/records/records.service.ts [55-60]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-1818e6d68069b8c3703cda2defc212b14561322453eea709094b1e39994281a4R55-R60) ```diff -const exchange = this.getExchangeInstance(exchangeId, apiKey, secret); -const trades = await this.fetchTrades(exchange, symbol, since); -const { openOrderVolume, averageDuration, spread } = await this.processOpenOrders(exchange, symbol); +let exchange, trades, openOrderVolume, averageDuration, spread; +try { + exchange = this.getExchangeInstance(exchangeId, apiKey, secret); + trades = await this.fetchTrades(exchange, symbol, since); + ({ openOrderVolume, averageDuration, spread } = await this.processOpenOrders(exchange, symbol)); +} catch (error) { + // Handle errors (e.g., log them, throw a custom error, etc.) + console.error("Failed to calculate liquidity score", error); + throw new Error("Liquidity score calculation failed"); +} ```
    Add descriptive description and author fields in package.json. ___ **Consider specifying a more descriptive description and author in your package.json to
    provide more context about the project and its maintainers. This information is useful for
    users and contributors to understand the purpose of the project and who to contact for
    queries or contributions.** [recording-oracle/package.json [4-5]](https://github.com/Hu-Fi/Mr.Market/pull/90/files#diff-511fb8afe5be487afda92e97e743b756c7fa3c0983b954842c348398c6f49877R4-R5) ```diff -"description": "", -"author": "", +"description": "A brief description of the recording oracle project.", +"author": "Your Name ", ```

    ✨ Improve tool usage guide:
    **Overview:** The `improve` tool scans the PR code changes, and automatically generates suggestions for improving the PR code. The tool can be triggered [automatically](https://pr-agent-docs.codium.ai/usage-guide/automations_and_usage/#github-app-automatic-tools-when-a-new-pr-is-opened) every time a new PR is opened, or can be invoked manually by commenting on a PR. When commenting, to edit [configurations](https://github.com/Codium-ai/pr-agent/blob/main/pr_agent/settings/configuration.toml#L69) related to the improve tool (`pr_code_suggestions` section), use the following template: ``` /improve --pr_code_suggestions.some_config1=... --pr_code_suggestions.some_config2=... ``` With a [configuration file](https://pr-agent-docs.codium.ai/usage-guide/configuration_options/), use the following template: ``` [pr_code_suggestions] some_config1=... some_config2=... ```
    Enabling\disabling automation
    When you first install the app, the [default mode](https://pr-agent-docs.codium.ai/usage-guide/automations_and_usage/#github-app-automatic-tools-when-a-new-pr-is-opened) for the improve tool is: ``` pr_commands = ["/improve --pr_code_suggestions.summarize=true", ...] ``` meaning the `improve` tool will run automatically on every PR, with summarization enabled. Delete this line to disable the tool from running automatically.
    Utilizing extra instructions
    Extra instructions are very important for the `improve` tool, since they enable to guide the model to suggestions that are more relevant to the specific needs of the project. Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify relevant aspects that you want the model to focus on. Examples for extra instructions: ``` [pr_code_suggestions] # /improve # extra_instructions=""" Emphasize the following aspects: - Does the code logic cover relevant edge cases? - Is the code logic clear and easy to understand? - Is the code logic efficient? ... """ ``` Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.
    A note on code suggestions quality
    - While the current AI for code is getting better and better (GPT-4), it's not flawless. Not all the suggestions will be perfect, and a user should not accept all of them automatically. - Suggestions are not meant to be simplistic. Instead, they aim to give deep feedback and raise questions, ideas and thoughts to the user, who can then use his judgment, experience, and understanding of the code base. - Recommended to use the 'extra_instructions' field to guide the model to suggestions that are more relevant to the specific needs of the project, or use the [custom suggestions :gem:](https://pr-agent-docs.codium.ai/tools/custom_suggestions/) tool - With large PRs, best quality will be obtained by using 'improve --extended' mode.
    More PR-Agent commands
    > To invoke the PR-Agent, add a comment using one of the following commands: > - **/review**: Request a review of your Pull Request. > - **/describe**: Update the PR title and description based on the contents of the PR. > - **/improve [--extended]**: Suggest code improvements. Extended mode provides a higher quality feedback. > - **/ask \**: Ask a question about the PR. > - **/update_changelog**: Update the changelog based on the PR's contents. > - **/add_docs** 💎: Generate docstring for new components introduced in the PR. > - **/generate_labels** 💎: Generate labels for the PR based on the PR's contents. > - **/analyze** 💎: Automatically analyzes the PR, and presents changes walkthrough for each component. >See the [tools guide](https://pr-agent-docs.codium.ai/tools/) for more details. >To list the possible configuration parameters, add a **/config** comment.
    See the [improve usage](https://pr-agent-docs.codium.ai/tools/improve/) page for a more comprehensive guide on using this tool.
    zed-wong commented 3 months ago

    @Faouzijedidi1 maybe we put it in new repo idk? or convert draft to pr and we merge it

    zed-wong commented 3 months ago

    Let place this in new repo