search

Nekmo / telegram-upload

Upload and download files from Telegram up to 4 GiB using your account
MIT License
1.07k stars 233 forks source link

Feature request - Preventing uploading of duplicate files #114

Open tissole opened 3 years ago

tissole commented 3 years ago

I wish that the app could detect duplicate files and skipping them. Maybe storing the names and hashes of files uploaded in a database that keeps track of them. This will still not prevent uploading the files that were already in a channel, uploaded by other means.

So, a solution could be indexing the files from the channel in a database too, for future reference. This option could be turned on/off if the user wishes so.

This feature can be implemented?

Nekmo commented 3 years ago

Calculating the hash of the files would imply a significant disk and cpu usage. Research is needed.

tissole commented 3 years ago

Obviously, there's no need here for a heavily cryptographic hash algorithm because security is not a concern. A lighter non-cryptographic algorithm can be used and I found one blazing fast xxHash. It has a python implementation too.

Nekmo commented 2 years ago

Of course @tissole , I was thinking about sha1 or sha256.