search

tvdstaaij / telegram-history-dump

Backup Telegram chat logs using telegram-cli
Other
664 stars 84 forks source link

JSON Dump line at the end of file #74

Open bikhial opened 7 years ago

bikhial commented 7 years ago

Hi Currently dumper dumps json line at the beginning of the file. Is it possible to have new message line at the end of file? I change some lines of dumper_prepender.rb file but i cant do it.

Thanks for your help

tvdstaaij commented 7 years ago

The thing is, the Telegram history downloading progress goes like this for every dialog (n is the chunk size):

This is repeated until the oldest message in the dialog is reached (on the first dump) or until the most recent message from the last dump is reached (when doing an incremental dump).

This means that new-to-old is the "natural order" of the dump. As far as I know, there is no possibility to set an offset from the oldest message, only an offset from the newest message. It would be possible to work around this of course, for example by writing chunks to temporary files and concatenating them in reverse order at the end of the dialog. It might be a good idea to start doing this in a next major version (to any users reading this: opinions welcome). I'd say that from a processing point of view it makes more sense to have the data from old to new. It could also open up a way for solving the formatter memory issue (#57) seeing how all formatters currently reverse loop a RAM-buffered version of the message objects.

For now I would advise to process the JSON lines in reverse order (you can find reverse line reader code for many languages),

tvdstaaij commented 7 years ago

The above commit is an implementation of the temporary file strategy I mentioned earlier. You can try it out if you want, note that this is on the branch dump-old-to-new and not on master. Usage notes:

tvdstaaij commented 7 years ago

Formatters should also work after the above commit, and again requires a fresh backup.