Enhancement: Add duplicate detection for files in agent knowledge base

[dekdynamics]

Hi,

I found that it is possible to upload the same file over and over to the agent file repository.

It would be handy if the upload file flow indicated which files were already present and prevented their re-upload, or if the duplicates were purged after the upload.

[greptile-apps[bot]]

To add duplicate detection for files in the agent knowledge base, follow these steps:

1. Update AgentFile Model: Add a method to check for duplicates.

public static function isDuplicate($agentId, $fileName, $fileHash) {
    return self::where('agent_id', $agentId)
                ->where(function($query) use ($fileName, $fileHash) {
                    $query->where('name', $fileName)
                          ->orWhere('hash', $fileHash);
                })->exists();
}

2. Modify Create and Edit Components: Check for duplicates before uploading.

foreach ($this->files as $file) {
    $fileHash = md5_file($file->getRealPath());
    if (AgentFile::isDuplicate($agent->id, $file->getClientOriginalName(), $fileHash)) {
        $this->alert('error', 'Duplicate file detected: ' . $file->getClientOriginalName());
        continue;
    }
    // Existing upload logic...
}

3. Update Migration: Add a hash column to agent_files table.

Schema::table('agent_files', function (Blueprint $table) {
    $table->string('hash')->nullable();
});

4. Update submit Method: Save file hash when uploading.

$agent->documents()->create([
    'name' => $filename,
    'path' => $filenametostore,
    'url' => $url,
    'disk' => $disk,
    'type' => $file->getClientMimeType(),
    'hash' => $fileHash,
]);

These changes will prevent duplicate files from being uploaded and provide feedback to the user.

References

/app/Http/Controllers/Api/v1/DocumentsController.php /app/Models/AgentFile.php /app/Livewire/Agents/Create.php /app/Livewire/Agents/Edit.php /app/Livewire/Agents/Modals/DocumentDelete.php /database/migrations/2024_04_18_172758_create_agent_files_table.php /resources/views/livewire/agents/create.blade.php /resources/views/livewire/agents/edit.blade.php /resources/views/livewire/agents/partials/documents.blade.php

Ask Greptile