Catch ValueError during file upload

[tissole]

• telegram-upload version: 0.4.0
• Python version: 3.9.5
• Operating System: Windows 10
• Dependencies list (run pip freeze): cffi==1.14.5 click==7.1.2 cryptg==0.2.post4 hachoir==3.1.2 Pillow==8.2.0 pyaes==1.6.1 pyasn1==0.4.8 pycparser==2.20 pywin32==300 rsa==4.7.2 telegram-upload==0.4.0 Telethon==1.21.1

Description

I tried to upload a folder with a large number of small images (all under 1Mb) and got this error: "OSError: [Errno 24] Too many open files"

What I Did

telegram-upload --to Memes --directories recursive 'D:\Memes'

I believe that this error was triggered by the huge number of files so I split the big folder into small folders of 2000 files each. Then I got this error: "telethon.errors.rpcerrorlist.PhotoInvalidDimensionsError: The photo dimensions are invalid (hint: pip install pillow for send_file to resize images) (caused by SendMediaRequest)"

I installed pillow and I tried again, but no luck, this time the error was: "ValueError: read less than 131072 before reaching the end; either file_size or read are wrong"

So I'm stuck here. Any ideas on how to resolve this?

[Nekmo]

Can you put the full traceback to find where the problem occurs? Thanks!

[tissole]

The first test is with a very big folder containing almost 21.000 small images, the majority under 1 MB each, mostly jpg, and a few png and gif. The log:

File "D:\TeleGram\telegram-upload-master\venv\Scripts\telegram-upload-script.py", line 33, in sys.exit(load_entry_point('telegram-upload==0.4.0', 'console_scripts', 'telegram-upload')()) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\exceptions.py", line 54, in wrap return fn(*args, kwargs) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\click\core.py", line 829, in call return self.main(args, kwargs) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\click\core.py", line 782, in main rv = self.invoke(ctx) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\click\core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\click\core.py", line 610, in invoke return callback(args, kwargs) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\management.py", line 107, in upload files = list(files) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\files.py", line 87, in next return next(self._iterator) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\files.py", line 116, in get_iterator yield self.process_normal_file(file) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\files.py", line 119, in process_normal_file return File(file, force_file=self.force_file, thumbnail=self.thumbnail, caption=self.caption) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\files.py", line 135, in init super().init(path) OSError: [Errno 24] Too many open files

The second test was done with a much smaller folder containing only 2000 such image files. This is the log:

File "D:\TeleGram\telegram-upload-master\venv\Scripts\telegram-upload-script.py", line 33, in sys.exit(load_entry_point('telegram-upload==0.4.0', 'console_scripts', 'telegram-upload')()) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\exceptions.py", line 54, in wrap return fn(*args, kwargs) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\click\core.py", line 829, in call return self.main(args, kwargs) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\click\core.py", line 782, in main rv = self.invoke(ctx) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\click\core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\click\core.py", line 610, in invoke return callback(args, kwargs) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\management.py", line 111, in upload client.send_files(to, files, delete_on_success, print_file_id, forward) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\client.py", line 173, in send_files message = self._send_file_message(entity, file, thumb, progress) File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telegram_upload\client.py", line 123, in _send_file_message message = self.send_file(entity, file, thumb=thumb, File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telethon\sync.py", line 39, in syncified return loop.run_until_complete(coro) File "C:\Users\karuso\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete return future.result() File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telethon\client\uploads.py", line 375, in send_file file_handle, media, image = await self._file_to_media( File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telethon\client\uploads.py", line 692, in _file_to_media file_handle = await self.upload_file( File "d:\telegram\telegram-upload-master\venv\lib\site-packages\telethon\client\uploads.py", line 603, in upload_file raise ValueError( ValueError: read less than 131072 before reaching the end; either file_size or read are wrong

[Nekmo]

It may be a bug in Windows, it requires checking, thanks.

[tissole]

I find a workaround for this issue. I run the script with git bash and is working :-) No more errors.

[pcaglar]

It may be a bug in Windows, it requires checking, thanks.

same error for me while running with bash script on linux.

Uploading "pics2.jpg" [##----------------------------------] 7% Traceback (most recent call last): File "/usr/local/bin/telegram-upload", line 8, in sys.exit(upload_cli()) File "/usr/local/lib/python3.8/dist-packages/telegram_upload/exceptions.py", line 64, in wrap return fn(*args, kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke return callback(args, kwargs) File "/usr/local/lib/python3.8/dist-packages/telegram_upload/management.py", line 129, in upload client.send_files(to, files, delete_on_success, print_file_id, forward) File "/usr/local/lib/python3.8/dist-packages/telegram_upload/client.py", line 187, in send_files message = self._send_file_message(entity, file, thumb, progress) File "/usr/local/lib/python3.8/dist-packages/telegram_upload/client.py", line 133, in _send_file_message message = self.send_file(entity, file, thumb=thumb, File "/usr/local/lib/python3.8/dist-packages/telethon/sync.py", line 39, in syncified return loop.run_until_complete(coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/local/lib/python3.8/dist-packages/telethon/client/uploads.py", line 392, in send_file file_handle, media, image = await self._file_to_media( File "/usr/local/lib/python3.8/dist-packages/telethon/client/uploads.py", line 713, in _file_to_media file_handle = await self.upload_file( File "/usr/local/lib/python3.8/dist-packages/telethon/client/uploads.py", line 622, in upload_file raise ValueError( ValueError: read less than 131072 before reaching the end; either file_size or read are wrong

https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072

could this be the solution?

Thanks

[Nekmo]

@CcTemplar the error seems to indicate a error in the file or during the upload. Try uploading the file again.

[santosadrian]

Same here. I post details, maybe is usseful:

(base) user@rig:~/Pictures$ python --version
Python 3.9.13
(base) user@rig:~/Pictures$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:    22.04
Codename:   jammy
(base) user@rig:~/Pictures$ conda --version
conda 22.9.0
(base) user@rig:~/Pictures$ telegram-upload resiste-no-estas-solo-JDelR-yt-banner.png

(base) user@rig:~/Pictures$ telegram-upload resiste-no-estas-solo-JDelR-yt-banner.png 

Traceback (most recent call last):
  File "/home/user/.Anaconda/bin/telegram-upload", line 8, in <module>
    sys.exit(upload_cli())
  File "/home/user/.Anaconda/lib/python3.9/site-packages/telegram_upload/exceptions.py", line 64, in wrap
    return fn(*args, **kwargs)
  File "/home/user/.Anaconda/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/.Anaconda/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/user/.Anaconda/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/.Anaconda/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/user/.Anaconda/lib/python3.9/site-packages/telegram_upload/management.py", line 172, in upload
    client.send_files(to, files, delete_on_success, print_file_id, forward)
  File "/home/user/.Anaconda/lib/python3.9/site-packages/telegram_upload/client.py", line 183, in send_files
    message = self._send_file_message(entity, file, thumb, progress)
  File "/home/user/.Anaconda/lib/python3.9/site-packages/telegram_upload/client.py", line 133, in _send_file_message
    message = self.send_file(entity, file, thumb=thumb,
  File "/home/user/.Anaconda/lib/python3.9/site-packages/telethon/sync.py", line 39, in syncified
    return loop.run_until_complete(coro)
  File "/home/user/.Anaconda/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/user/.Anaconda/lib/python3.9/site-packages/telethon/client/uploads.py", line 392, in send_file
    file_handle, media, image = await self._file_to_media(
  File "/home/user/.Anaconda/lib/python3.9/site-packages/telethon/client/uploads.py", line 713, in _file_to_media
    file_handle = await self.upload_file(
  File "/home/user/.Anaconda/lib/python3.9/site-packages/telethon/client/uploads.py", line 622, in upload_file
    raise ValueError(
ValueError: read less than 131072 before reaching the end; either `file_size` or `read` are wrong

[Nekmo]

Does this always happen with the same file? Was the file open or being tampered with at the time of the upload?

[bootsector]

Having this issue as well Edit: Adding --force-file seems to help.

[ericvlog]

any solution on this? I also face same problem, above comments say --force-file will help, but I upload media files, --force-file will disable the preview.

Any updated?