Use of a thumbnails database for tdlib

[Tobias-B-Besemer]

I think it would be better, to use a thumbnails database for tdlib.

First, Microsoft on Windows, Google on Android and Picture Cloud Storage like Googles and at least Samsungs, use all a thumbnails database! I think all those companies found reasons why such a db is better then store for each picture a single, small thumbnail! But if even this is not a argument to do it, I think there other good arguments to do it...

Very small files don't use all the space that is reserviert for them. This means every file waste a bit of unused disk space and with thousands of files this gets a lot. A database would waste disk space and the content in it could even get compressed to save more space. Then, there should be normally a management of those thumbnails... Means: To don't have unnecessary files on the disk (and waste disk space with them, too), thumbs, if they are just "space holders" for not yet downloaded picture and after the download of the pics don't be needed anymore, should normally get deleted after the pic is downloaded. Then, maybe the user have defined a maximum space the cache can/should use. To keep cache in size there must be a deletion logic programmed. I don't know how tdlib handle this at the moment, but I guess - to keep it simple - the first download files get deleted first. But IMHO would be better to prefer the deletion of thumbs a bit over (maybe with mobile data) downloaded content. With a db files could be deleted first that not the first downloaded, no, it would be possible to delete those thumbs first where the pics in the chat history get posted first. (Yes, with a normal cache db, too...) Next that should get managed is, that a user maybe delete a chat, channel, or group where thumbnails got downloaded for. At least for privacy & security reasons this thumbs should get deleted immidiatly! All in all a lot of thumbs management that is AFAIK not implemented yet, and if get implemented, can maybe get implemented with a future use of a thumbs db.

Finally there is IMHO a use case problem on Android what wasn't intensionally not so designed, but comes to more and more users: A lot of Picture Galleries ignore by design the different Technics to hide (pics) folders and show the user all folders with pics found by default and even more, add all those pics to the "complete view" of all pics found on the device.

And yeah, I forgot a other prob of the latest trends on Android Phones! Cloud Storage like those from Google and Samsung and for sure from others, too, scan the device for pics and move them with different rules into the cloud. It's just high likely that those "pics mover" move the older thumbs into the cloud, too, what can result into that tdlib thinks that there must be thumbs that can be used, but finally don't find any (with errors generated), because the pics are already moved into any cloud!

OK, and at least the prob, that the pics (thumbs) (should) be already in a db! Then because I know, Android generate from every pic found on the device (yes, from the tdlib thumbs, too) a thumb and save this thumb into the Android thumbs db! This means if I don't be wrong, that there is even more disk space waste, because the tdlib thumbs didn't just get hold in the TG Cache folder, no, they get hold in the Android Thumbs DB, too!

[Tobias-B-Besemer]

OK, I must confess, I'm a bit a geek! I like to have all the digital private pics I ever made on my device! Yeah, you never know when you want to show an old one to somebody... ^^

...but anyway: Beside the prob, that the Facebook apps save every sticker they download (and they download in advance sticker themes, you never used before) in a new folder, what lead to the phanome, that I have over 2.000 pic folders on my device... ...I have over 100.000 pics (!!!) now on my phone where at least (no, I have no space prob, I have 256GB internal ^^) 90% should be TG Thumbs where the most of them have not 1kb and got downloaded at any time in the past maybe for channels, or groups, I have left long time ago...

...and all this leads to the point, that I have a Android System Thumbnails Database from over 7GB (!!!) on my system, where should be thousends of pics (TG Thumbs) included that get first calculated as new data/thumbs for the db and then included, even not necessary at all!

[Tobias-B-Besemer]

Just FYI, if this info on StackExchange is right, for every pic on the device get at least 10kb used in the Android Thumbs DB even if the downloaded thumb from Telegram have often 1kb, or less! ^^ https://android.stackexchange.com/questions/66221/android-huge-thumbdata4-file-in-dcim-folder

[Tobias-B-Besemer]

Will in the following days have a look on my Android System Thumbnails Database to see if I'm right... ...just in case anyone else will do it , too, I test the following two apps...

https://play.google.com/store/apps/details?id=uni.unseengallery.ke

https://play.google.com/store/apps/details?id=com.futuretech.unseen.images

[levlam]

Microsoft on Windows, Google on Android and Picture Cloud Storage like Googles and at least Samsungs, use all a thumbnails database!

The mentioned companies use thumbnail databases to not generate thumbnails every time photos are viewed and to not create a lot of additional files in user directories. Both reasons are irrelevant to TDLib.

This means every file waste a bit of unused disk space and with thousands of files this gets a lot.

Databases have overhead on stored data also, which is usually bigger than the corresponding overhead of a file system,

if they are just "space holders" for not yet downloaded picture and after the download of the pics don't be needed anymore, should normally get deleted after the pic is downloaded.

Thumbnails are not just placeholders, they often used to speed up image reading and showing.

Then, maybe the user have defined a maximum space...

Most of this is already implemented in TDLib. There is no need to change the way files are stored for that..

A lot of Picture Galleries ignore by design the different Technics to hide (pics) folders

If a Gallery app ignores .nomedia file, then it is better to report that to developer of the Gallery app.

It's just high likely that those "pics mover" move the older thumbs into the cloud, too, what can result into that tdlib thinks that there must be thumbs that can be used, but finally don't find any (with errors generated), because the pics are already moved into any cloud!

The descibed behavior will never result in an errors generated. The files will be redownloaded without any issue.

Then because I know, Android generate from every pic found on the device

It should ignore files from TDLib cache folders because the folders contain .nomedia file.

...and all this leads to the point, that I have a Android System Thumbnails Database from over 7GB (!!!) on my system or every pic on the device get at least 10kb used in the Android Thumbs DB

Have you just told that Thumbnail database is a very ineffecive way to store photos?

To summarize, I don't see enough reasons to move files to a database from a designed specially for storing files database called file system. This move will make much harder for applications using TDLib to access the files and needlessly complcate file usage for a general app.

[Tobias-B-Besemer]

Ignoring .nomedia files and showing the content of folders start with a dot, is a intended/implemented feature of some galleries to give users picture access to the pictures they e.g. recived with WhatsApp or e.g. have recived/be already downloaded into cache of Telegram. Normally a great feature!

I think, that Cloud Storages like Google Photos and Samsung Cloud will move TG Thumbnails into the cloud! Maybe this is no problem for tglib, but it gets a big problem for the user! First it is a waste of cloud space, even - because of the small size of the thumbs - this can maybe get ignored. But, I had this problem with all those Clouds that auto-upload pictures (and maybe move them completely) - and I tested at the beginning many like e.g. Amazon Photos/Cloud, Dropbox & MEGA - they will at least catch, use & show pictures already downloaded from the normal Telegram Client and create with them a gigant, absolutely useless photo stream, where the real pictures the user want normally in such a stream, are "hidden" in a amout of other stuff coming from the Telegram Channels & Groups! Even if this Cloud Storages would respect the .nomedia files and ignore the folders starting with a dot - and I'm sure this can't by the lately fast growing amount of Cloud Storage that came up be promised - this would show, that there is an issue at least in the normal Telegram Client, because I had really a lot of this files in the folders & streams!

And then I thought, that one goal of tdlib - beside e.g. have the network and server communication in a library where the developer of an alternative client can easily use without the need of deep background knowledge - is, to have an "abstraction layer" for e.g. (file management of) the cache, so he must not really care about how thumbs are stored and managed, and can easily build his alternative client on top and can concentrate on new user features and/or maybe an alternative user interface design, animations, or other handling. So may question to this is then: Isn't e.g. a perfectionised, automated and unfied cache managment and the file or database handling if it a goal, or why should this have a big impact on an alternative client developer?

[levlam]

Ignoring .nomedia file is a problem of a Gallery app. If some app autouploads every file it can find to the cloud and doesn't allow to customize the behavior, this is also a problem of the app.

TDLib supports getting file content using readFilePart method, which allows developers to not care about where files are stored. But this method will be slower than direct reading from a file and needlessly complcates some use cases.

So, there is a possibility to move all files to a database files, but there are no enough reasons to complicate things there. File system is almost a perfect database for storing files.