mb makehashes crash (because of zsync feature?)

kelson42 commented 5 years ago

I have activated zsync in the Mirrorbrain configuration and have run the following to get the zsync "file" done:

$mb makehashes /var/www/download.kiwix.org/ -v --force -t /usr/share/mirrorbrain

The process went fine on a few files... but then crash systematically:

Hashing '/var/www/download.kiwix.org/dev/xz-5.0.3-windows.zip'... done.
Hash file updated: '/usr/share/mirrorbrain/dev/xz-5.0.3-windows.zip.size_927128'
Hash updated in database for 'dev/xz-5.0.3-windows.zip'
Hashing '/var/www/download.kiwix.org/dev/xz-5.0.3.tar.bz2'... done.
Hash file updated: '/usr/share/mirrorbrain/dev/xz-5.0.3.tar.bz2.size_1025913'
Hash updated in database for 'dev/xz-5.0.3.tar.bz2'
Hashing '/var/www/download.kiwix.org/dev/xz-5.0.4-windows.zip'... done.
Hash file updated: '/usr/share/mirrorbrain/dev/xz-5.0.4-windows.zip.size_935531'
Hash updated in database for 'dev/xz-5.0.4-windows.zip'
Hashing '/var/www/download.kiwix.org/dev/xz-5.0.4.tar.bz2'... done.
Hash file updated: '/usr/share/mirrorbrain/dev/xz-5.0.4.tar.bz2.size_1048328'
Hash updated in database for 'dev/xz-5.0.4.tar.bz2'
Hashing '/var/www/download.kiwix.org/dev/xz-5.2.4.tar.gz'... done.
Hash file updated: '/usr/share/mirrorbrain/dev/xz-5.2.4.tar.gz.size_1572354'
Hash updated in database for 'dev/xz-5.2.4.tar.gz'
Hashing '/var/www/download.kiwix.org/dev/zlib-1.2.8.tar.gz'... done.
Hash file updated: '/usr/share/mirrorbrain/dev/zlib-1.2.8.tar.gz.size_571091'
Hash updated in database for 'dev/zlib-1.2.8.tar.gz'
unlocking /usr/share/mirrorbrain/dev/LOCK
Examining directory /var/www/download.kiwix.org/hotspots
locked /usr/share/mirrorbrain/hotspots/LOCK
Hashing '/var/www/download.kiwix.org/hotspots/README'... done.
Hash file updated: '/usr/share/mirrorbrain/hotspots/README.size_243'
Hash updated in database for 'hotspots/README'
Hashing '/var/www/download.kiwix.org/hotspots/edupi_en_200gb_2017-03.img'... done.
Hash file updated: '/usr/share/mirrorbrain/hotspots/edupi_en_200gb_2017-03.img.size_196865949696'
Traceback (most recent call last):
  File "/usr/local/bin/mb", line 1677, in <module>
    r = mirrordoctor.main()
  File "/usr/local/lib/python2.7/dist-packages/cmdln.py", line 262, in main
    return self.cmd(args)
  File "/usr/local/lib/python2.7/dist-packages/cmdln.py", line 285, in cmd
    retval = self.onecmd(argv)
  File "/usr/local/lib/python2.7/dist-packages/cmdln.py", line 423, in onecmd
    return self._dispatch_cmd(handler, argv)
  File "/usr/local/lib/python2.7/dist-packages/cmdln.py", line 1124, in _dispatch_cmd
    return handler(argv[0], opts, *args)
  File "/usr/local/bin/mb", line 1134, in do_makehashes
    force=opts.force)
  File "/usr/local/lib/python2.7/dist-packages/mb/hashes.py", line 210, in check_db
    file_id])
psycopg2.InternalError: invalid memory alloc request size 1073741824

$ mb --version
2.18.1

Stuff seems to work fine without the zync option activated.

kelson42 commented 5 years ago

@poeml Might that be that the field zsums BYTEA is too small?

kelson42 commented 5 years ago

I have move the chunk size to 4MB, but mb makehashes still crash with a file of 200GB.

poeml / mirrorbrain

mb makehashes crash (because of zsync feature?) #176