GitAuto: [FEATURE] Grab taxes and levies from BCB for all banks

Resolves #141

What is the feature

Implement functionality to fetch taxes and levies data from the Banco Central do Brasil (BCB) for all banks, based on their ISPB codes. This involves retrieving data from the BCB's personal and corporate URLs for each bank and extracting the relevant information into JSON files separate from the banks' data files. Additionally, integrate data from the SFA - Open Finance participants directory into the application.

Why we need the feature

This feature enhances the application by providing up-to-date taxes and levies information for all banks, which is crucial for users requiring accurate financial data. Automating the retrieval and updating process ensures that the information remains current without manual intervention. Integrating data from the SFA - Open Finance participants directory enriches the application's dataset, offering a more comprehensive view of bank participants in the open finance ecosystem.

How to implement and why

Implementation Steps:

Add Required URLs to Constants.cs

Include the base URLs for personal and corporate taxes/levies pages:

public const string BcbTaxesPersonalUrl = "https://www.bcb.gov.br/fis/tarifas/htms/{0}.asp?seg=&frame=1";
public const string BcbTaxesCorporateUrl = "https://www.bcb.gov.br/fis/tarifas/htms/{0}a.asp?seg=&frame=1";

Why: Centralizing URLs in Constants.cs makes future updates easier and keeps the URLs organized.

Add New Enum Item in Source.cs
- Add BcbTaxes and SfaOpenFinance to the Source enum.
```
public enum Source
{
 // Existing sources...
 BcbTaxes,
 SfaOpenFinance
}
```
- Why: Enumerating the new data sources allows for easy reference and management within the application.
Implement Loading Methods in Reader.cs
- LoadBcbTaxes()
  - Loop through all banks' ISPB codes.
  - For each bank, construct personal and corporate URLs using the ISPB.
  - Fetch HTML content from both URLs.
  - Parse HTML to extract taxes and levies information.
- LoadSfaOpenFinance()
  - Fetch data from the SFA - Open Finance participants URL.
  - Parse and integrate the data into the application's data models.
- Why: Separate methods for loading data maintain code organization and readability.
Parse HTML Content
- Use an HTML parser like HtmlAgilityPack.
```
var web = new HtmlWeb();
var doc = web.Load(url);
// Parse document...
```
- Extract relevant data such as tax names, rates, and descriptions.
- Why: Proper parsing is essential to accurately extract required information from HTML content.

Add New Fields to Bank.cs

Extend the Bank class:

public class Bank
{
 // Existing fields...
 public List<TaxInfo> PersonalTaxes { get; set; }
 public List<TaxInfo> CorporateTaxes { get; set; }
}
public class TaxInfo
{
 public string Name { get; set; }
 public decimal Rate { get; set; }
 public string Description { get; set; }
}

Why: Modifying the data model allows the application to store and manage the new data effectively.

Implement Merging Logic in Seeder.cs
- MergeBcbTaxes()
  - Merge taxes data with existing bank data based on ISPB.
  - Ensure banks have mandatory fields before adding new records.
  - Why: Merging ensures the new data integrates smoothly without disrupting existing records.
Update AcquireData in Program.cs
- Call the new methods:
```
var reader = new Reader();
reader.LoadBcbTaxes();
reader.LoadSfaOpenFinance();
```
- Why: Including new data sources in the data acquisition process ensures they are processed.
Update Writer.cs
- Modify output methods to include new fields:
  - SaveCsv()
  - SaveMarkdown()
  - SaveSql()
  - Example for CSV:
```
csv.WriteField("PersonalTaxes");
csv.WriteField("CorporateTaxes");
```
- Why: Updating output methods allows users to access the new data in all supported formats.
Testing
- Run the application locally to verify:
  - Data is fetched and parsed correctly.
  - Output files in the result directory contain the new data.
- Handle exceptions and edge cases (e.g., missing data, network errors).
- Why: Comprehensive testing ensures reliability and correctness of the new feature.
Documentation
- Update the README.md with information about the new data sources.
- Provide instructions on how to use and interpret the new data fields.
- Why: Keeping documentation up-to-date helps users and contributors understand the changes.

About backward compatibility

Data Model Compatibility:
- New fields are added to the Bank class without altering existing fields.
- Existing functionality remains unaffected, ensuring current users experience no disruptions.
Output Formats:
- The structure of CSV, Markdown, and SQL outputs remains consistent.
- New data is appended, preserving the format for users parsing these files.
Code Integration:
- Existing methods and workflows are extended, not replaced.
- New methods are added in a way that they don't interfere with current processes.
Why: Maintaining backward compatibility is crucial to prevent breaking changes for users who depend on the current implementation.

Test these changes locally

git checkout -b gitauto/issue-#141-e0b4dd12-d9e2-486d-a6f4-745dae1bf2ac
git pull origin gitauto/issue-#141-e0b4dd12-d9e2-486d-a6f4-745dae1bf2ac

[!IMPORTANT]

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

🪧 Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit , please review it.` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (Invoked using PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. ### Other keywords and placeholders - Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. - Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description. - Add `@coderabbitai` anywhere in the PR title to generate the title automatically. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

guibranco / BancosBrasileiros-MergeTool