PLG 522/asset sender - Githubissues

kevin-paladin commented 3 months ago

This is a migration of the asset related code in data-shipper in order that we can customize it later for asset specific changes.

I anticipate discussion around a few practices in order that we can choose an agreeable solution.

Type of change

[x] New feature (non-breaking change which adds functionality)

How Has This Been Tested?

I've tested manually so far, primarily with gcp data.

I've verified a few different types (vminstance, cloudstorage, cloudfunction) against the existing data-shipper, comparing specific documents.

Checklist:

[x] My code follows the style guidelines of this project
[x] My commit message/PR follows the contribution guidelines of this project
[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

Summary by CodeRabbit

New Features
- Introduced asset processing and management functionality, including sending messages to Amazon SQS and managing asset groups.
- Added handling of database interactions and S3 operations.
Enhancements
- Improved asset statistics collection and compliance information retrieval.
- Enhanced error processing and Elasticsearch updates for better data management.
Configurations
- Added new configuration constants for development, Elasticsearch, RDS, S3, and sender components.
- Introduced a configuration service to manage and retrieve configuration properties.

coderabbitai[bot] commented 3 months ago

Walkthrough

The recent changes introduce new functionalities for asset management and processing within the assets-subsystem. Several helper classes facilitate interactions with AWS services, databases, and Elasticsearch. New methods and classes enhance asset handling, grouping, compliance checks, and error processing. Additionally, configuration settings and authentication mechanisms have been updated to support these enhancements.

Changes

File(s)	Change Summary
`AssetSenderJob.java`	Introduced functionality for processing assets, extending `JobExecutor` with necessary constants and logging.
`SQSHelper.java`	Introduced a class to facilitate sending messages to Amazon SQS, including methods for serialization and message handling.
`AssetGroups.java`	Added methods for managing asset groups, fetching compliance data, and updating aliases.
`Assets.java`	Implemented methods to manage assets, fetch data from S3, and upload to Elasticsearch.
`LoadErrors.java`	Added a method to process error data from S3 and update Elasticsearch.
`DatabaseHelper.java`	Added methods for executing SQL queries and inserting data into a database.
`S3Helper.java`	Introduced functionality to list objects, fetch data, and check for object existence in S3.
`AssetDocumentFields.java`	Introduced an interface defining fields for Elasticsearch asset documents.
`AssetGroupStatsCollector.java`	Added methods to collect and upload various statistics related to asset groups.
`AssetsCounts.java`	Introduced a class handling asset counts for different platforms.
`DataSourceHelper.java`	Introduced a class to fetch asset groups, accounts, and aliases using Elasticsearch and database helper.
`AuthHelper.java`	Added methods for managing access tokens and authorize API operations.
`ConfigConstants.java`	Introduced interfaces for handling various configuration constants.
`ConfigService.java`	Added functionality to retrieve and set configuration properties from a specified URL.
`gcp-cloudsql.json`	Introduced a JSON structure containing records with various attributes.

Assessment against linked issues

Objective	Addressed	Explanation
Introduce functionality for processing assets (PLG-522)	✅
Add methods for asset group management and compliance checks (PLG-522)	✅
Implement helper classes for AWS, DB, and Elasticsearch interactions (PLG-522)	✅
Update configuration settings and authentication mechanisms (PLG-522)	✅

Poem

In the land of code where assets roam,
We've built new paths for them to call home.
Helpers in S3 and SQS do sing,
Fetching, sending, all in a swing.
With every byte and every log,
Our system grows, no longer a cog.
Cheers to changes, both bold and small,
In this realm of tech, we stand tall.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.