ERROR: BLOB longer than INT_MAX bytes

[tzulberti]

Describe the bug I am trying to download Ghost in the Shell, but even before the first file starts downloading I get an error:

anime: ./ghost-in-the-shell/ghost-in-the-shell_1.mp4
INFO:anime_downloader.downloader.base_downloader:./ghost-in-the-shell/ghost-in-the-shell_1.mp4

ERROR: BLOB longer than INT_MAX bytes

To reproduce

anime -ll DEBUG dl 'Ghost' --choice 2

2020-05-28 21:19:24 desktop anime_downloader.session[1196] DEBUG uncached request
DEBUG:anime_downloader.session:uncached request
send: b'GET /anime/ghostintheshell/%5Bbonkai77%5D.Ghost.in.the.Shell.(1995).%5BBD.1080p.Dual.Audio.x265.HEVC.10bit%5D.mp4 HTTP/1.1\r\nHost: edge-32.cdn.bunny.sh\r\nuser-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) Gecko/20100101Firefox/56.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nreferer: https://twist.moe/anime/ghostintheshell/[bonkai77].Ghost.in.the.Shell.(1995).[BD.1080p.Dual.Audio.x265.HEVC.10bit].mp4\r\nCookie: __cfduid=d732937cae412e05951d1302b2d8863b41590711526\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Fri, 29 May 2020 00:18:47 GMT
header: Content-Type: video/mp4
header: Content-Length: 2203322816
header: Connection: keep-alive
header: Last-Modified: Thu, 09 Aug 2018 22:51:29 GMT
header: ETag: "5b6cc571-835409c0"
header: X-Proxy-Cache: HIT
header: X-Proxy-Serve: EU-CDN
header: Accept-Ranges: bytes
2020-05-28 21:19:25 desktop anime_downloader.session[1196] DEBUG uncached request
DEBUG:anime_downloader.session:uncached request

Traceback (most recent call last):
  File "/home/tzulberti/envs/downloader/bin/anime", line 8, in <module>
    sys.exit(main())
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/anime_downloader/cli.py", line 53, in main
    cli()
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/anime_downloader/commands/dl.py", line 139, in command
    range_size=chunk_size)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/anime_downloader/sites/anime.py", line 385, in download
    downloader.download()
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/anime_downloader/downloader/base_downloader.py", line 55, in download
    self.check_if_exists()
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/anime_downloader/downloader/base_downloader.py", line 35, in check_if_exists
    self.source.stream_url, headers=headers, stream=True)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests_cache/core.py", line 136, in request
    **kwargs
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests_cache/core.py", line 109, in send
    return send_request_and_cache_response()
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests_cache/core.py", line 97, in send_request_and_cache_response
    response = super(CachedSession, self).send(request, **kwargs)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests/sessions.py", line 665, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests/sessions.py", line 665, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests/sessions.py", line 245, in resolve_redirects
    **adapter_kwargs
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests_cache/core.py", line 109, in send
    return send_request_and_cache_response()
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests_cache/core.py", line 99, in send_request_and_cache_response
    self.cache.save_response(cache_key, response)
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests_cache/backends/base.py", line 46, in save_response
    self.responses[key] = self.reduce_response(response), datetime.utcnow()
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests_cache/backends/storage/dbdict.py", line 166, in __setitem__
    sqlite.Binary(pickle.dumps(item)))
  File "/home/tzulberti/envs/downloader/lib/python3.6/site-packages/requests_cache/backends/storage/dbdict.py", line 130, in __setitem__
    self.table_name, (key, item))
OverflowError: BLOB longer than INT_MAX bytes

pd: This is an amazing app

[Blatzar]

I'm guessing this happens when the internal downloader encounters a file larger than the 32 bit integer limit. aria2 most likely handles this, are you using it?

[vn-ki]

Looks like this is a problem with requests cache.

[vn-ki]

We should have a flag to disable it.

[AbdullahM0hamed]

@tzulberti does this error still occur for you?

[AbdullahM0hamed]

I was unable to replicate it