Cleanup database client

mrheinen commented 1 month ago

PR Type

Enhancement, Bug fix

Description

Refactored API endpoints to remove 'GET all' methods and introduce segment-based retrieval for improved efficiency and scalability.
Updated database client interface to use search methods instead of 'GET all' methods for various resources.
Introduced external version tracking (ext_version and ext_uuid) for Content, ContentRule, and Application models.
Enhanced WHOIS and honeypot lookups to use new search methods with improved error handling.
Updated UI components (RuleForm and RulesList) to use new segment-based API endpoints.
Modified database schema to include new columns for version tracking.
Improved rule loading in the backend to use batches for better performance.
Removed dependency on github.com/vingarcia/ksql from the API package.

Changes walkthrough 📝

Relevant files

Enhancement

8 files

api_server.go `Refactor API endpoints for improved efficiency` cmd/api/api_server.go Removed several 'GET all' endpoints for various resources Updated router to use new segment-based endpoints	+0/-6
server.go `Streamline API server handlers and improve WHOIS lookup` pkg/api/server.go Removed 'GET all' handler functions Updated WHOIS lookup to use new search method Refactored error handling in some functions	+9/-73
backend.go `Enhance backend server with improved data retrieval` pkg/backend/backend.go Updated honeypot lookup to use new search method Modified rule loading to use batches Added error handling for honeypot lookup	+38/-14
database.go `Refactor database client for efficient data retrieval` pkg/database/database.go Removed 'GET all' methods from DatabaseClient interface Added new search methods for various resources Introduced ExternalDataModel interface for version tracking	+86/-157
rdap.go `Enhance WHOIS lookup with new search method` pkg/whois/rdap.go Updated WHOIS lookup to use new search method Improved error handling and logging	+12/-6
RuleForm.vue `Update RuleForm to use new API endpoint` ui/src/components/container/RuleForm.vue - Updated app loading to use new segment-based endpoint	+1/-1
RulesList.vue `Update RulesList to use new API endpoint` ui/src/components/container/RulesList.vue - Updated app loading to use new segment-based endpoint	+1/-1
database.sql `Update database schema for version tracking` config/database.sql Added ext_version and ext_uuid columns to content, content_rule, and app tables	+6/-2

Documentation

1 files

SEARCH_KEYWORDS.md

Update search keywords documentation

SEARCH_KEYWORDS.md

Added new fields for external version and UUID to Content,
ContentRule, and Application

+6/-0

Dependencies

1 files

BUILD.bazel

Update Bazel build file

pkg/api/BUILD.bazel - Removed dependency on github.com/vingarcia/ksql

+0/-1

💡 PR-Agent usage: Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

github-actions[bot] commented 1 month ago

Preparing PR description...

github-actions[bot] commented 1 month ago

Preparing review...

codiumai-pr-agent-pro[bot] commented 1 month ago

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Key issues to review

API Changes
Significant changes to the DatabaseClient interface, including removal of several 'GET all' methods and addition of new search methods. This may impact existing code that relies on these methods. Performance Concern
The new rule loading method uses batches, which could potentially lead to performance issues if not implemented carefully. The maximum number of batches is hardcoded to 10, which might not be suitable for all scenarios. Error Handling
The error handling for the WHOIS lookup has been changed. It now logs the error but continues execution, which might lead to unexpected behavior if the database query fails.

codiumai-pr-agent-pro[bot] commented 1 month ago

PR Code Suggestions ✨

Category	Suggestion	Score
Data integrity	Add validation for the ExtUuid field to ensure it's a valid UUID ___ Consider adding validation for the `ExtUuid` field to ensure it's a valid UUID format when setting or updating it. [pkg/database/database.go [73]](https://github.com/mrheinen/lophiid/pull/37/files#diff-1adb887d06a44193c36fc1c5708be385f3129cd59c2f2aa555faa065941ed877R73-R73) ```diff ExtUuid string `ksql:"ext_uuid" json:"ext_uuid" doc:"The external unique ID of the content"` +func (c Content) SetExtUuid(uuid string) error { + if !isValidUUID(uuid) { + return fmt.Errorf("invalid UUID format") + } + c.ExtUuid = uuid + return nil +} + +func isValidUUID(uuid string) bool { + r := regexp.MustCompile("^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-4[a-fA-F0-9]{3}-[8\|9\|aA\|bB][a-fA-F0-9]{3}-[a-fA-F0-9]{12}$") + return r.MatchString(uuid) +} + ``` - [ ] Apply this suggestion* Suggestion importance[1-10]: 8 Why: This suggestion improves data integrity by ensuring the UUID is in a valid format before setting it.	8
Thread safety	Implement an atomic method to increment the ExtVersion field ___ Consider implementing a method to increment the `ExtVersion` field atomically to ensure thread-safety when updating the version. [pkg/database/database.go [72]](https://github.com/mrheinen/lophiid/pull/37/files#diff-1adb887d06a44193c36fc1c5708be385f3129cd59c2f2aa555faa065941ed877R72-R72) ```diff ExtVersion int64 `ksql:"ext_version" json:"ext_version" doc:"The external numerical version of the content"` +func (c Content) IncrementExtVersion() { + atomic.AddInt64(&c.ExtVersion, 1) +} + ``` - [ ] Apply this suggestion* Suggestion importance[1-10]: 8 Why: This suggestion enhances thread safety when updating the version, which is important for concurrent operations.	8
Maintainability	Define a constant for the default external version value ___ Consider using a constant or an enum for the default external version value (1) to improve maintainability and reduce the risk of inconsistencies. [pkg/database/database.go [52-57]](https://github.com/mrheinen/lophiid/pull/37/files#diff-1adb887d06a44193c36fc1c5708be385f3129cd59c2f2aa555faa065941ed877R52-R57) ```diff +const DefaultExternalVersion = 1 + type ExternalDataModel interface { ModelID() int64 ExternalVersion() int64 ExternalUuid() string SetModelID(id int64) } ``` - [ ] Apply this suggestion Suggestion importance[1-10]: 7 Why: Using a constant improves code maintainability and reduces the risk of inconsistencies, but it's a minor improvement.	7
Enhancement	Add a method to generate a new ExtUuid for Content objects ___ Consider adding a method to generate a new `ExtUuid` when creating a new `Content` object to ensure uniqueness. [pkg/database/database.go [85-88]](https://github.com/mrheinen/lophiid/pull/37/files#diff-1adb887d06a44193c36fc1c5708be385f3129cd59c2f2aa555faa065941ed877R85-R88) ```diff func (c Content) ModelID() int64 { return c.ID } func (c Content) ExternalVersion() int64 { return c.ExtVersion } func (c Content) ExternalUuid() string { return c.ExtUuid } func (c Content) SetModelID(id int64) { c.ID = id } +func (c Content) GenerateNewExtUuid() { + c.ExtUuid = uuid.New().String() +} + ``` - [ ] Apply this suggestion* Suggestion importance[1-10]: 6 Why: While this suggestion adds a useful feature, it's not crucial and the current implementation may already handle UUID generation elsewhere.	6

mrheinen / lophiid