Concurrent File Duplicate Finder

[xmonader]

Create a command-line tool that concurrently searches for duplicate files in a given directory tree using file hashes (e.g., MD5).

Requirements:

1. File Scanning:

   • Recursively walk through a specified directory tree
   • Calculate MD5 hash for each file
   • Handle large files efficiently (read in chunks)

2. Concurrency:

   • Use goroutines to process multiple files simultaneously
   • Handle SIGTERM and SIGINT
   • Properly cancel all of the goroutines when the app is exiting

3. Duplicate Detection:

   • Store file hashes and paths in a concurrent-safe data structure
   • Identify files with matching hashes as duplicates

4. Output:

   • Display groups of duplicate files
   • Provide JSON output too
   • Show file paths and sizes
   • Optionally, provide a summary (total space that could be saved)

5. User Interface (CLI):

   • Accept directory path as a command-line argument
   • Provide flags for optional features (e.g., minimum file size to consider)

6. Testing:

   • Write unit tests for core functions (e.g., hash calculation, duplicate detection)
   • Implement integration tests with a sample directory structure

7. Documentation:

   • Provide clear comments explaining concurrency patterns used
   • Include a README with usage instructions and performance considerations

Optional Enhancements:

• Allow user to choose hash algorithm (MD5, SHA-256, etc.)
• Add an option to automatically delete or move duplicate files
• Support for ignoring certain file types or directories

Acceptance Criteria:

• Tool correctly identifies all duplicate files in a given directory tree
• Concurrency is properly implemented
• Code is well-documented and passes all tests