IncompleteRead Issue

[ghost]

Issue: https://github.com/xbmc/xbmc/issues/24490

see my comment: https://github.com/urllib3/urllib3/issues/3277#issuecomment-1890830176

You need to point to the real code that causes the error.

It seems you are experimenting:

https://github.com/henryjfry/repository.thenewdiamond/blob/9768b3e3973d5778cabff788f8b39fceecbd3c61/script.diamondinfo/a4kscrapers_wrapper/subs.py#L71-L77

Kodi looks interesting, if you need some help with your project here, let me know (give me an email, or an invite to some non-public project)

[ghost]

I cant find a public file I can reproduce the error with, but its very definetly an error thrown by "_error_catcher" in Urllib3 with these problen Urls, i think its the header response.

A working header (eg https://sample-videos.com/video321/mp4/720/big_buck_bunny_720p_1mb.mp4)

 HTTPHeaderDict({'Date': 'Sun, 14 Jan 2024 10:32:36 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'})

Versus the RealDebrid response header contiaing the wrong "Content-Length"

HTTPHeaderDict({'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381782703-381848239/381848239', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'})

The exact same code works on the command line and not in kodi because kodi is using a different version of urllib3.

@henryjfry , can you

• post the HTTPHeaderDict of the working RealDebrid content header (from the command line)?
• post this "exact same code"?

[henryjfry]

So the working URL header from the commandline:

{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}

And the working code:

import struct, os
import urllib
__64k = 65536
__longlong_format_char = 'q'
__byte_size = struct.calcsize(__longlong_format_char)

def temp_file():
    import tempfile
    file = tempfile.NamedTemporaryFile()
    filename = file.name
    return filename

def is_local(_str):
    from urllib.parse import urlparse
    if os.path.exists(_str):
        return True
    elif urlparse(_str).scheme in ['','file']:
        return True
    return False

def hashFile_url(filepath): 
    #https://trac.opensubtitles.org/projects/opensubtitles/wiki/HashSourceCodes
    #filehash = filesize + 64bit sum of the first and last 64k of the file
    name = filepath
    if is_local(filepath):
        local_file = True
    else:
        local_file = False
    if local_file == False:
        from urllib import request
        f = None
        #opener = None
        url = name
        request.urlcleanup()
        f = request.urlopen(url)
        filesize = int(f.headers['Content-Length'])
        if filesize < __64k * 2:
            try: filesize = int(str(f.headers['Content-Range']).split('/')[1])
            except: pass
        first_64kb = temp_file()
        last_64kb = temp_file()
        import requests
        headers = {"Range": 'bytes=0-%s' % (str(__64k))}
        r = requests.get(url, headers=headers)
        with open(first_64kb, 'wb') as f:
            for chunk in r.iter_content(chunk_size=1024): 
                if chunk: # filter out keep-alive new chunks
                    f.write(chunk)
        if filesize > 0:
            headers = {"Range": 'bytes=%s-%s' % (filesize - __64k, filesize)}
        else:
            f.close()
            os.remove(first_64kb)
            return "SizeError" 
        try:
            r = requests.get(url, headers=headers)
            with open(last_64kb, 'wb') as f:
                for chunk in r.iter_content(chunk_size=1024): 
                    if chunk: # filter out keep-alive new chunks
                        f.write(chunk)
        except:
            f.close()
            if os.path.exists(last_64kb):
                os.remove(last_64kb)
            if os.path.exists(first_64kb):
                os.remove(first_64kb)
            return 'IOError'
        f = open(first_64kb, 'rb')
    try:
        longlongformat = '<q'  # little-endian long long
        bytesize = struct.calcsize(longlongformat) 
        if local_file:
            f = open(name, "rb") 
            filesize = os.path.getsize(name) 
        hash = filesize 
        if filesize < __64k * 2: 
            f.close()
            if local_file == False:
                os.remove(last_64kb)
                os.remove(first_64kb)
            return "SizeError" 
        range_value = __64k / __byte_size
        range_value = round(range_value)
        for x in range(range_value): 
            buffer = f.read(bytesize) 
            (l_value,)= struct.unpack(longlongformat, buffer)  
            hash += l_value 
            hash = hash & 0xFFFFFFFFFFFFFFFF #to remain as 64bit number  
        if local_file:
            f.seek(max(0,filesize-__64k),0) 
        else:
            f.close() 
            f = open(last_64kb, 'rb')
        for x in range(range_value): 
            buffer = f.read(bytesize) 
            (l_value,)= struct.unpack(longlongformat, buffer)  
            hash += l_value 
            hash = hash & 0xFFFFFFFFFFFFFFFF 
        f.close() 
        if local_file == False:
            os.remove(last_64kb)
            os.remove(first_64kb)
        returnedhash =  "%016x" % hash 
        return returnedhash
    except(IOError): 
        if local_file == False:
            os.remove(last_64kb)
            os.remove(first_64kb)
        return 'IOError'

[ghost]

{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}

That's A different request.

kodi failing:

HTTPHeaderDict({'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381782703-381848239/381848239', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'})

Now what's the working one on cli? Same request please.

[henryjfry]

Ok so the working command line examples with the:

/usr/lib/python3/dist-packages/urllib3/response.py

Modified to print the header

{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
{'Date': 'Tue, 16 Jan 2024 10:08:20 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '1055736', 'Connection': 'close', 'Content-Type': 'video/mp4'}
RealDebrid
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
381926360
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
URLLIB_HEADERS
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
381926360
698f18f7323fc01e
https://sample-videos.com/video321/mp4/720/big_buck_bunny_720p_1mb.mp4
{'Date': 'Tue, 16 Jan 2024 10:08:23 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:23 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:23 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:23 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:23 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:23 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:23 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:23 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
1055736
{'Date': 'Tue, 16 Jan 2024 10:08:24 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:24 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:24 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:24 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:24 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:24 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:24 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
URLLIB_HEADERS
{'Date': 'Tue, 16 Jan 2024 10:08:24 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
1055736
b3fbc599afa3b9fc

[henryjfry]

And the same run in kodi with the header output in:

/home/osmc/.kodi/addons/script.module.urllib3/lib/urllib3/response.py

As kodi was throwing the error before it reached the relevant line in the code

2024-01-16 10:29:59.490 T:8579     info <general>: {'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}__MESSAGE__
2024-01-16 10:30:00.142 T:8579     info <general>: {'Date': 'Tue, 16 Jan 2024 10:09:38 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '1055736', 'Connection': 'close', 'Content-Type': 'video/mp4'}__MESSAGE__
2024-01-16 10:30:00.143 T:8579     info <general>: RealDebrid__MESSAGE__
2024-01-16 10:30:00.542 T:8579     info <general>: {'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}URLLIB_HEADERS
2024-01-16 10:30:00.606 T:8579     info <general>: Skipped 7 duplicate messages..
2024-01-16 10:30:00.606 T:8579     info <general>: {'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}__MESSAGE__
2024-01-16 10:30:00.606 T:8579     info <general>: 381926360__MESSAGE__
2024-01-16 10:30:00.838 T:8579     info <general>: {'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}URLLIB_HEADERS
2024-01-16 10:30:00.922 T:8579     info <general>: Skipped 7 duplicate messages..
2024-01-16 10:30:00.922 T:8579     info <general>: ('Connection broken: IncompleteRead(65536 bytes read, 1 more expected)', IncompleteRead(65536 bytes read, 1 more expected))===>A4K_Wrapper
2024-01-16 10:30:00.925 T:8579     info <general>: IOError__MESSAGE__
2024-01-16 10:30:00.926 T:8579     info <general>: https://sample-videos.com/video321/mp4/720/big_buck_bunny_720p_1mb.mp4__MESSAGE__
2024-01-16 10:30:02.276 T:8579     info <general>: {'Date': 'Tue, 16 Jan 2024 10:09:40 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}URLLIB_HEADERS
2024-01-16 10:30:02.592 T:8579     info <general>: Skipped 7 duplicate messages..
2024-01-16 10:30:02.592 T:8579     info <general>: {'Date': 'Tue, 16 Jan 2024 10:09:40 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}__MESSAGE__
2024-01-16 10:30:02.592 T:8579     info <general>: 1055736__MESSAGE__
2024-01-16 10:30:03.281 T:8579     info <general>: {'Date': 'Tue, 16 Jan 2024 10:09:41 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}URLLIB_HEADERS
2024-01-16 10:30:03.595 T:8579     info <general>: Skipped 7 duplicate messages..
2024-01-16 10:30:03.595 T:8579     info <general>: {'Date': 'Tue, 16 Jan 2024 10:09:41 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}__MESSAGE__
2024-01-16 10:30:03.597 T:8579     info <general>: 1055736__MESSAGE__
2024-01-16 10:30:03.647 T:8579     info <general>: b3fbc599afa3b9fc__MESSAGE__

[ghost]

'Content-Range': 'bytes 0-65536/381926360'

You are requesting 65537 bytes (0-65536)

You need to request 65536 bytes (0-65535).

headers = {"Range": 'bytes=0-%s' % (str(__64k -1 ))}

[henryjfry]

I am requesting 65536 bytes:

64k = 65536 headers = {"Range": 'bytes=%s-%s' % (filesize - 64k, filesize)}

[henryjfry]

File hashes wouldnt match the example file if i were requesting the wrong number of bytes

[ghost]

You need to get it together an listen:

this is the fix, just keep silent, and try it out.

headers = {"Range": 'bytes=0-%s' % (str(__64k -1 ))}

[henryjfry]

Using your header range:

headers = {"Range": 'bytes=0-%s' % (str(__64k -1 ))}

For the last 64kb the test file on their wiki:

https://trac.opensubtitles.org/projects/opensubtitles/wiki/HashSourceCodes https://static.opensubtitles.org/addons/avi/breakdance.avi

Produces the hash:

1b0736e88cf061a8MESSAGE

Versus my original method which matches the online hash:

8e245d9679d31e12MESSAGE

So my original method is requesting the correct bytes and the prior version of Urllib will not return the error and will return the correct file hash

[ghost]

What hash? This goes step by step. Did the error message disapear for the first request?

[henryjfry]

The hash that "hashFile_url" in the code I posted produces.

This is supposed to match the wiki method for a local file Therefore it must request the first 64kb and the last 64kb of a file

[ghost]

Friend, are you able to liste and go debug step by step? Stop pushing info. (i know what the hash is, question was rhetorical).

My question: Did the error message disapear for the first request? After adding -1?

(yes/no)

[henryjfry]

The first part never gave the error, it was only the last64kb that gave the error. I can't test it right now as my computer is locked up doing something.

But that new header was for the first 64kb right?

But I'll test it in a moment.

Fyi I've already tried negative 1's to change the request and see if that had any difference and a failing request would always report 1 more byte than requested.

[ghost]

very well, still fix the 1st part. 2nd part should be:

~headers = {"Range": 'bytes=%s-%s' % (filesize - __64k + 1, filesize)}~ ~NOTICE: + 1~

~you should notice then 'Content-Length': '65536' 9instead of 65537) everywhere.!~

[ghost]

Fyi

No FYI, no talk, just tests & logs.

(btw, the system looks nice, I may get a trial month)

[henryjfry]

Ok so using the following code:

import struct, os
import urllib
__64k = 65536
__longlong_format_char = 'q'
__byte_size = struct.calcsize(__longlong_format_char)

try:
    import xbmc
except:
    pass

def temp_file():
    import tempfile
    file = tempfile.NamedTemporaryFile()
    filename = file.name
    return filename

def is_local(_str):
    from urllib.parse import urlparse
    if os.path.exists(_str):
        return True
    elif urlparse(_str).scheme in ['','file']:
        return True
    return False

def logger(msg):
    try: xbmc.log(str(msg)+'__MESSAGE__', level=xbmc.LOGINFO)
    except: print(str(msg))

def hashFile_url(filepath): 
    #https://trac.opensubtitles.org/projects/opensubtitles/wiki/HashSourceCodes
    #filehash = filesize + 64bit sum of the first and last 64k of the file
    name = filepath
    if is_local(filepath):
        local_file = True
    else:
        local_file = False
    if local_file == False:
        from urllib import request
        f = None
        #opener = None
        url = name
        request.urlcleanup()
        f = request.urlopen(url)
        filesize = int(f.headers['Content-Length'])
        if filesize < __64k * 2:
            try: filesize = int(str(f.headers['Content-Range']).split('/')[1])
            except: pass
        first_64kb = temp_file()
        last_64kb = temp_file()
        import requests
        headers = {"Range": 'bytes=0-%s' % (str(__64k))}
        headers = {"Range": 'bytes=0-%s' % (str(__64k -1 ))}
        r = requests.get(url, headers=headers)
        logger(dict(r.headers))
        logger(filesize)
        with open(first_64kb, 'wb') as f:
            for chunk in r.iter_content(chunk_size=1024): 
                if chunk: # filter out keep-alive new chunks
                    f.write(chunk)
        if filesize > 0:
            headers = {"Range": 'bytes=%s-%s' % (filesize - __64k, filesize)}
            headers = {"Range": 'bytes=%s-%s' % (filesize - __64k + 1, filesize)}
        else:
            f.close()
            os.remove(first_64kb)
            return "SizeError" 
        try:
            r = requests.get(url, headers=headers)
            logger(dict(r.headers))
            logger(filesize)
            with open(last_64kb, 'wb') as f:
                for chunk in r.iter_content(chunk_size=1024): 
                    if chunk: # filter out keep-alive new chunks
                        f.write(chunk)
        except:
            f.close()
            if os.path.exists(last_64kb):
                os.remove(last_64kb)
            if os.path.exists(first_64kb):
                os.remove(first_64kb)
            return 'IOError'
        f = open(first_64kb, 'rb')
    try:
        longlongformat = '<q'  # little-endian long long
        bytesize = struct.calcsize(longlongformat) 
        if local_file:
            f = open(name, "rb") 
            filesize = os.path.getsize(name) 
        hash = filesize 
        if filesize < __64k * 2: 
            f.close()
            if local_file == False:
                os.remove(last_64kb)
                os.remove(first_64kb)
            return "SizeError" 
        range_value = __64k / __byte_size
        range_value = round(range_value)
        for x in range(range_value): 
            buffer = f.read(bytesize) 
            (l_value,)= struct.unpack(longlongformat, buffer)  
            hash += l_value 
            hash = hash & 0xFFFFFFFFFFFFFFFF #to remain as 64bit number  
        if local_file:
            f.seek(max(0,filesize-__64k),0) 
        else:
            f.close() 
            f = open(last_64kb, 'rb')

        for x in range(range_value): 
            buffer = f.read(bytesize) 
            (l_value,)= struct.unpack(longlongformat, buffer)  
            hash += l_value 
            hash = hash & 0xFFFFFFFFFFFFFFFF 
        f.close() 
        if local_file == False:
            os.remove(last_64kb)
            os.remove(first_64kb)
        returnedhash =  "%016x" % hash 
        return returnedhash
    except(IOError): 
        if local_file == False:
            os.remove(last_64kb)
            os.remove(first_64kb)
        return 'IOError'

from urllib import request
test_url = 'https://20.download.real-debrid.com/d/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.mp4'
f = request.urlopen(test_url)
logger(dict(f.headers))
f = request.urlopen('https://sample-videos.com/video321/mp4/720/big_buck_bunny_720p_1mb.mp4')
logger(dict(f.headers))

logger('https://static.opensubtitles.org/addons/avi/breakdance.avi')

logger(hashFile_url('https://static.opensubtitles.org/addons/avi/breakdance.avi'))

logger('RealDebrid')

logger(hashFile_url(test_url))

logger('https://sample-videos.com/video321/mp4/720/big_buck_bunny_720p_1mb.mp4')

logger(hashFile_url('https://sample-videos.com/video321/mp4/720/big_buck_bunny_720p_1mb.mp4'))

I get a new error because it is now requesting the wrong amount of data presumably? This the command line error for visibility (same error):

{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
{'Date': 'Tue, 16 Jan 2024 11:06:59 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '1055736', 'Connection': 'close', 'Content-Type': 'video/mp4'}
https://static.opensubtitles.org/addons/avi/breakdance.avi
{'Date': 'Tue, 16 Jan 2024 11:27:21 GMT', 'Content-Type': 'video/x-msvideo', 'Content-Length': '65536', 'Connection': 'keep-alive', 'ETag': '"4126440223"', 'Last-Modified': 'Tue, 23 Jan 2018 08:35:06 GMT', 'X-Cache-Backend': 'fw2', 'Age': '112', 'X-Var-Cache': 'HIT', 'X-Var-Cache-Hits': '10', 'X-RateLimit-Remaining': '40', 'X-Via': 'fw2', 'Accept-Ranges': 'bytes', 'Content-Range': 'bytes 0-65535/12909756', 'CF-Cache-Status': 'DYNAMIC', 'Report-To': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=cAVKJv1Qjbm5lVA3vW%2FqYjh68Mruos0vFkYS9z2LtV6X1QOZTEBqdgoksQJqRr1v1B93BSXkoSNSzshHum%2Bdm7GwSwHoBfQmvcGwLyZ7vZt6hy4EP3yv4EAwn0i56eDCx9j5%2FsKYkxMAL3E%3D"}],"group":"cf-nel","max_age":604800}', 'NEL': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'Server': 'cloudflare', 'CF-RAY': '8466023f1a3b6337-LHR', 'alt-svc': 'h3=":443"; ma=86400'}
12909756
{'Date': 'Tue, 16 Jan 2024 11:27:21 GMT', 'Content-Type': 'video/x-msvideo', 'Content-Length': '65535', 'Connection': 'keep-alive', 'ETag': '"4126440223"', 'Last-Modified': 'Tue, 23 Jan 2018 08:35:06 GMT', 'X-Cache-Backend': 'fw2', 'Age': '113', 'X-Var-Cache': 'HIT', 'X-Var-Cache-Hits': '11', 'X-RateLimit-Remaining': '40', 'X-Via': 'fw2', 'Accept-Ranges': 'bytes', 'Content-Range': 'bytes 12844221-12909755/12909756', 'CF-Cache-Status': 'DYNAMIC', 'Report-To': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=KL2E3J0NFJw4efVjEQ513qBSmeTvaPb8lxJZUn73QXubVcaUi9nJuTGPrqxvu2CCyFkHGdIbpd%2By0FquJnRsU2diWdfVakHEWAqMtj1aDUBwfu6%2BvSBVtfu1kOdr%2FqUwa9OZX%2Bd877l7eKw%3D"}],"group":"cf-nel","max_age":604800}', 'NEL': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'Server': 'cloudflare', 'CF-RAY': '8466024119fc75d5-LHR', 'alt-svc': 'h3=":443"; ma=86400'}
12909756
Traceback (most recent call last):
  File "/home/osmc/.kodi/addons/script.extendedinfo/subs_file_hash3.py", line 138, in <module>
    logger(hashFile_url('https://static.opensubtitles.org/addons/avi/breakdance.avi'))
  File "/home/osmc/.kodi/addons/script.extendedinfo/subs_file_hash3.py", line 112, in hashFile_url
    (l_value,)= struct.unpack(longlongformat, buffer)
struct.error: unpack requires a buffer of 8 bytes

[henryjfry]

So thats with the sample file they provide.

[henryjfry]

Commenting out the new headers and the results from the commandline:

{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
{'Date': 'Tue, 16 Jan 2024 11:08:28 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '1055736', 'Connection': 'close', 'Content-Type': 'video/mp4'}
https://static.opensubtitles.org/addons/avi/breakdance.avi
{'Date': 'Tue, 16 Jan 2024 11:28:50 GMT', 'Content-Type': 'video/x-msvideo', 'Content-Length': '65537', 'Connection': 'keep-alive', 'ETag': '"4126440223"', 'Last-Modified': 'Tue, 23 Jan 2018 08:35:06 GMT', 'X-Cache-Backend': 'fw2', 'Age': '0', 'X-Var-Cache': 'MISS', 'X-RateLimit-Remaining': '40', 'X-Via': 'fw2', 'Accept-Ranges': 'bytes', 'Content-Range': 'bytes 0-65536/12909756', 'CF-Cache-Status': 'DYNAMIC', 'Report-To': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=OHGrcqJRNNVH4%2BETlKZHT%2B40G09%2BctYnfcGzCjN2c00D9G3RLzNopdXNc3NOwnw5PCmMI7EAgFCqJKFlTCm7JNsmd%2FrnGQiIPlpk%2BeQ3r2gJ7I1eHZ8ttNCKkvZg2909ueNlRQUcKzLaHlo%3D"}],"group":"cf-nel","max_age":604800}', 'NEL': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'Server': 'cloudflare', 'CF-RAY': '846604697e0a496e-LHR', 'alt-svc': 'h3=":443"; ma=86400'}
12909756
{'Date': 'Tue, 16 Jan 2024 11:28:50 GMT', 'Content-Type': 'video/x-msvideo', 'Content-Length': '65536', 'Connection': 'keep-alive', 'ETag': '"4126440223"', 'Last-Modified': 'Tue, 23 Jan 2018 08:35:06 GMT', 'X-Cache-Backend': 'fw2', 'Age': '0', 'X-Var-Cache': 'MISS', 'X-RateLimit-Remaining': '40', 'X-Via': 'fw2', 'Accept-Ranges': 'bytes', 'Content-Range': 'bytes 12844220-12909755/12909756', 'CF-Cache-Status': 'DYNAMIC', 'Report-To': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=2L%2BGoutni4btWAECEAtH2TX8pMyts19rCGyOfDAvONK2qHSl%2FHsgQ82Ojnxdx06d1l9WwK8KS1Ya2qba5kAXBa7c4wE7YDNYAj4jwW0%2Fnhtx8TMV1smBexRhjUMIIjJkoik%2BGVuaOr%2BS5wY%3D"}],"group":"cf-nel","max_age":604800}', 'NEL': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'Server': 'cloudflare', 'CF-RAY': '8466046af98971e6-LHR', 'alt-svc': 'h3=":443"; ma=86400'}
12909756
8e245d9679d31e12
RealDebrid
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
381926360
{'Server': 'Lity 2.0', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 381860824-381926360/381926360', 'Content-Disposition': 'attachment', 'Content-Type': 'application/force-download', 'Connection': 'close'}
381926360
698f18f7323fc01e
https://sample-videos.com/video321/mp4/720/big_buck_bunny_720p_1mb.mp4
{'Date': 'Tue, 16 Jan 2024 11:08:31 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65537', 'Content-Range': 'bytes 0-65536/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
1055736
{'Date': 'Tue, 16 Jan 2024 11:08:32 GMT', 'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23', 'Last-Modified': 'Tue, 20 Oct 2020 10:54:05 GMT', 'ETag': '"101bf8-5b2180cb74c20"', 'Accept-Ranges': 'bytes', 'Content-Length': '65536', 'Content-Range': 'bytes 990200-1055735/1055736', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'video/mp4'}
1055736
b3fbc599afa3b9fc

[ghost]

STOP IT, DELETE YOUR REPLIES!

[henryjfry]

Which replies?

[ghost]

the last ones, too much info.

[ghost]

(wait 2 min. i do it on my machine)

[ghost]

here you go:

headers = {"Range": 'bytes=%s-%s' % (filesize - __64k, filesize-1)};

[ghost]

Which means your issue is fixed, which means it is not an urllib3 issus (just the version 2 is more precise, and throws an error correctly).

[ghost]

spoke too soo, had my kodi "patch" implemented

ha! funny, ok. But it should(!) work now => classic code nonsense.

[henryjfry]

Sorry I was confused, looking at an old screen.

It does work. Doubly confirming now.

[henryjfry]

Yep, "filesize-1" was the fix:

headers = {"Range": 'bytes=%s-%s' % (filesize - __64k, filesize-1)};

[henryjfry]

Yeah fixing the two headers:

headers = {"Range": 'bytes=0-%s' % (str(64k -1 ))} headers = {"Range": 'bytes=%s-%s' % (filesize - 64k, filesize-1)}

Works. But I still dont understand why it then ends up requesting 64kb when its asking for 1 byte less?

(unless its a counting starts at 0 thing?)

[henryjfry]

Yeah kodi is pretty easy to get setup on windows. There is a good kodi reddit (avoid the official ones as soon as you mention real debrid youll get hit with the ban hammer).

https://www.reddit.com/r/Addons4Kodi/

For addons you want: tmdbhelper seren fen

And optionally my addon: extendedinfo mod

Tmdbhelper you need to install the player files, which basically responds to a playback request and returns playback links to your setup addons (ie seren/fen). The addons then scrape torrents, check RD and provide a playable link.

My addon (extendedinfo) provides a cool and pretty snappy ui to browse stuff and you can then play from tmdbhelper from its UI.

And this extendedinfo UI is accessed by opening the addon as a program rather than a video addon.

My addon has a bunch of extra functionality (unadvertised mostly) so if you are feeling adventurous I can talk you through it.

[henryjfry]

If you use trakt i advise you try tmdbhelper (and setup your trakt account) and my addon. There is an option to open you last played tv shows or movies and I pretty much live on that screen.

[henryjfry]

got kodi, looks nice, but all this drama with the addons etc. - maybe I try it out next week or so. currently have enough to watch on netflix

If you want the widest selection of content in the best quality kodi with realdebrid and with functioning addons shits all over netflix. But it does take some getting used to.

I advise you download repos as zip files: eg mine: https://henryjfry.github.io/repository.thenewdiamond/repository.newdiamond-2.2.zip

tmdbhelper: https://henryjfry.github.io/repository.thenewdiamond/repository.newdiamond-2.2.zip

seren: https://nixgates.github.io/packages/repository.nixgates-2.2.0.zip

And then install from the repo's

fen (addon he deactivated his repo) https://github.com/Tikipeter/tikipeter.github.io/blob/main/packages/plugin.video.fenlight-1.0.18.zip

And then potentially copy and paste some API keys into the addon userdata XML settings files which for some of the addons are needed as its easier than typing them into kodi.

And seren there is currently an error in v3 with db writes/trakt sync. So I could provide my copy of his addon which still has some issues but crashes far less (or just use Fen and dont bother with seren).

[henryjfry]

Oh yeah tmdbhelper: https://henryjfry.github.io/repository.thenewdiamond/repository.jurialmunkey-2.2.zip

[henryjfry]

kind of "kodi addon bundles"?

Yeah im not sure, the repo will be basically an xml file with the relevant addon details. So i think its to display correctly at the kodi end in the correct categories?

Kind of a ball ache actually as certain types of addons are effectively invisible (dependencies) and you need to know exactly where to look if you want a specific version

[ghost]

And don't forget to close the issues:

Issue: https://github.com/xbmc/xbmc/issues/24490

https://github.com/urllib3/urllib3/issues/3277