Concurrent File Duplicate Finder - Nabil

[Nabil-Salah]

Create a command-line tool that concurrently searches for duplicate files in a given directory tree using file hashes (e.g., MD5).

Requirements:

1. File Scanning:

• Recursively walk through a specified directory tree
• Calculate MD5 hash for each file
• Handle large files efficiently (read in chunks)

2. Concurrency:

• Use goroutines to process multiple files simultaneously
• Handle SIGTERM and SIGINT
• Properly cancel all of the goroutines when the app is exiting

3. Duplicate Detection:

• Store file hashes and paths in a concurrent-safe data structure
• Identify files with matching hashes as duplicates

4. Output:

• Display groups of duplicate files
• Provide JSON output too
• Show file paths and sizes
• Optionally, provide a summary (total space that could be saved)

5. User Interface (CLI):

• Accept directory path as a command-line argument
• Provide flags for optional features (e.g., minimum file size to consider)

6. Testing:

• Write unit tests for core functions (e.g., hash calculation, duplicate detection)
• Implement integration tests with a sample directory structure

7. Documentation:

• Provide clear comments explaining concurrency patterns used
• Include a README with usage instructions and performance considerations

Optional Enhancements:

• Allow user to choose hash algorithm (MD5, SHA-256, etc.)
• Add an option to automatically delete or move duplicate files
• Support for ignoring certain file types or directories

Acceptance Criteria:

• Tool correctly identifies all duplicate files in a given directory tree
• Concurrency is properly implemented
• Code is well-documented and passes all tests