tgdrive / teldrive

Telegram Drive
MIT License
1.8k stars 266 forks source link

Files Successfully Uploaded with RCLONE but Unable to Download Completely (Critical Issue) #165

Closed cj-neo closed 4 months ago

cj-neo commented 8 months ago

I have been having trouble uploading large files with rclone, and I am getting the following error:

failed to close object after copy: uploaded failed

Despite this error, rclone will still repeatedly upload the file multiple times, consuming a lot of time and traffic until it succeeds.

To solve this problem, I followed the suggestion I saw here and added the --multi-thread-streams=0 parameter. After that, the error above no longer appeared, and the upload was very smooth and successful. However, I found that many files seemed to disappear after being uploaded to less than half of the percentage, and then the number of successfully transferred files would increase by one. To test this, I uploaded the same files again with rclone, and the result was also correct and skipped the re-upload.

Since it took so little time and it felt strange, I tried to download the files with rclone, but I found that the download speed would drop to 0 and get stuck forever after reaching a certain percentage (each file is different, about 20-30% or even less), and there was no error (no error message in rclone and teldrive backend).

I also tried to download the files directly from the website, but I found that they could not be opened at all and displayed

ERR_EMPTY_RESPONSE.

I tested a dozen large files, and only one or two files that were uploaded successfully multiple times before adding the --multi-thread-streams=0 parameter could be downloaded normally.

It feels like the original error display and detection have been blocked, and the actual problem has not been solved. On the contrary, because there is no error, it is impossible to know that the file upload has actually failed, which may lead to file loss.

I have tried using different browsers and different internet connections,would appreciate any help in resolving this issue.

Thank you.

Linden10 commented 8 months ago

What a particular problem, while I have the issue of rclone doing the "fail and retry" the files play fine though I wonder why rclone doesn't function right when using that command...

I hope someone looks into this! For now, while I have no way to help you out I do have something that can help with detecting files that weren't properly uploaded to telegram/teldrive. You can type this in the sql editor on Neon DB or whatever program you use to handle your database.


Command to list files with id:0 (aka not uploaded to telegram/missing/can't open or play.) select * from teldrive.files where parts::text like '%{"id": 0}%'

Command to delete missing files: delete from teldrive.files where parts::text like '%{"id": 0}%'

Another command to delete files by letting teldrive deal with it itself.

UPDATE teldrive.files
SET status = 'pending_deletion'
WHERE parts::text like '%{"id": 0}%';

Thanks to the folks on the discord for these commands that can be used to easily detect files that aren't uploaded! The first command will list every file that has "id:0" which means that it wasn't uploaded to telegram properly, thus if you try to play the file it'll not work at all.

Once you confirmed what files need to be re-uploaded, you can then execute the second or third command and it'll automatically delete them. Hope this helps somewhat!

cj-neo commented 8 months ago

@Linden10 Thanks for providing the testing method. However, I found that the file seems to have been uploaded to TELEGRAM and even occupies compatible space, but the content is incomplete. I'm not using this software temporarily. I'll consider using it again when the author fixes the issue.

Linden10 commented 8 months ago

@Linden10 Thanks for providing the testing method. However, I found that the file seems to have been uploaded to TELEGRAM and even occupies compatible space, but the content is incomplete. I'm not using this software temporarily. I'll consider using it again when the author fixes the issue.

That’s a problem alright if the files aren’t being fully uploaded to telegram properly which is strange!

Hmm…say, have you restarted teldrive and tried uploading again? If yes then IDK…I hope the author or someone can figure something out.