thumbnails in ddr-densho-1000 on Dana's LOCAL VM not displaying

[pkikawa]

Dana's local vm got updated from 4.0.0 to 5.0.15 during which time thumbnails stopped displaying. Access files are in the repository, imagemagick is installed, thumbnail cache is present.

below is an example of an object page. [ddr-densho-1000-480-1 Workbench DDR.zip](https://github.com/denshoproject/ddr-local/files/5357490/ddr-densho-1000-480-1._.Workbench._.DDR.zip

[gjost]

Summary of chat log

@pkikawa upgraded Dana's VM from ddr-local 4.0.0 to 5.0.x. In the process the VM was also upgraded in place from Debian 9 to Debian 10. Made full-disk backup of old 4.0.0 before upgrade.

This is a special VM for working with ddr-densho-1000 where the repository is kept on the machine because of the size of the collection (running e.g. git status on a collection of this size on NFS is too slow). We thought for awhile there was a problem with fullsize binaries on this machine. Then it was pointed out to us (thank you @sarabeckman!) that fullsize binaries for this collection are large video files that are not present in the local collection. They are hosted at Internet Archive.

DDR configs (/etc/ddr/ddrlocal-local.cfg)

[local]
docstore_enabled=1
docstore_host=127.0.0.1:9200
docstore_index=ddrlocal
git_annex_whereis=False
vocabs_path=http://partner.densho.org/vocab/api/0.2
[public]
docstore_host=127.0.0.1:9200
docstore_index=ddrlocal
[debug]
debug=1

Media dir: /var/www/media/ddr symlinked to /media/ddr on this machine

densho@DDREditor:/var/www/media$ ls -la
total 16
drwxrwxr-x   3 ddr  ddr  4096 Oct  7 14:36 .
drwxr-xr-x   6 root root 4096 Jul 29 13:24 ..
drwxr-xr-x 229 ddr  ddr  4096 Oct 14  2019 cache
lrwxrwxrwx   1 ddr  ddr    11 Jul 18  2018 ddr -> /media/ddr/
-rw-r--r--   2 root root   67 Jun  2 13:05 README

DDR media path settings

densho@DDREditor:/opt/ddr-local$ sudo su ddr
[sudo] password for densho: 
ddr@DDREditor:/opt/ddr-local$ make shell
>>> from django.conf import settings
>>> settings.MEDIA_BASE; settings.MEDIA_ROOT; settings.MEDIA_URL; settings.STATIC_ROOT
'/var/www/media/ddr'
'/var/www/media'
'/media/'

Git and git-annex versions

(ddrlocal) ddr@DDREditor:/opt/ddr-local$ git version; git-annex version
git version 2.20.1
git-annex version: 7.20190129
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentPars
er MagicMime Feeds Testsuite
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1
 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 S
HA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B51
2 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BL
AKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
operating system: linux x86_64
supported repository versions: 5 7
upgrade supported from repository versions: 0 1 2 3 4 5 6

Nginx directory listing shows the files: http://192.168.56.101/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/

Access files can be viewed: http://192.168.56.101/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a.jpg

Thumbnails not appearing in editor Detail page shows "No image" where the thumbnail should be. "No image" instead of a broken image link.

Monitoring tail -f /var/log/ddr/local.log while loading a page reveals no log entries relating to thumbnails.

The app uses a SQLite3 database (/var/lib/ddr/ddrlocal.db) to store thumbnail metadata. Removing this database and restarting the app doesn't change or fix things.

Imagemagick is installed

densho@DDREditor:~$ convert --version
Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP
Delegates (built-in): bzlib djvu fftw fontconfig freetype jbig jng jp2 jpeg lcms
 lqr ltdl lzma openexr pangocairo png tiff wmf x xml zlib

Existing thumbnails are present in /var/www/media/cache/.

[gjost]

@pkikawa pls try running the DDR code that generates thumbnails.

This has to be run in the Python interpreter. If you are the ddr user you can just run django-admin shell. If you're not the ddr user, run this first: sudo su ddr; source /opt/ddr-local/venv/ddrlocal/bin/activate; django-admin shell. Then paste this code into the interpreter:

SRC='/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a.jpg'
DEST='/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a-thumb.jpg'
from DDR import imaging
imaging.thumbnail(src=SRC, dest=DEST, geometry='100x100')

The expected result is a blob of JSON like this:

{'src': '/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a.jpg', 'size_src': 127305, 'dest': '/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a-thumb.jpg', 'geometry': '100x100', 'analysis': {'path': '/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a.jpg', 'format': 'JPEG', 'frames': 1, 'can_thumbnail': None, 'std_out': '/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a.jpg JPEG 1024x576 1024x576+0+0 8-bit sRGB 127305B 0.010u 0:00.000', 'std_err': '', 'image': True}, 'convert': 'convert  "/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a.jpg"[0] -resize \'100x100\' /var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a-thumb.jpg', 'attempted': True, 'elapsed': '0:00:00.039944', 'status_code': 0, 'std_out': '', 'std_err': '', 'exists': True, 'size': 18221, 'islink': False}

Note: on Friday @pkikawa attempted to run this code but just put it in a SRC.sh file and ran it using Bash, producing this output which is probably misleading:

from: can't read /var/mail/DDR
./SRC.sh: line 4: syntax error near unexpected token `src=SRC,'
./SRC.sh: line 4: `imaging.thumbnail(src=SRC, dest=DEST, geometry='100x100')'

[pkikawa]

(ddrlocal) ddr@DDREditor:~$ django-admin shell

SRC='/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr- densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a.jpg' DEST='/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr -densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a-thumb.jpg' from DDR import imaging imaging.thumbnail(src=SRC, dest=DEST, geometry='100x100') {'src': '/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000-480/files/ddr- densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e59-a.jpg', 'size _src': 127305, 'dest': '/var/www/media/ddr/ddr-densho-1000/files/ddr-densho-1000 -480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine-c7c6795e5 9-a-thumb.jpg', 'geometry': '100x100', 'analysis': {'path': '/var/www/media/ddr/ ddr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr- densho-1000-480-1-mezzanine-c7c6795e59-a.jpg', 'format': 'JPEG', 'frames': 1, 'c an_thumbnail': None, 'std_out': '/var/www/media/ddr/ddr-densho-1000/files/ddr-de nsho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-1000-480-1-mezzanine- c7c6795e59-a.jpg JPEG 1024x576 1024x576+0+0 8-bit sRGB 127KB 0.000u 0:00.009', ' std_err': '', 'image': True}, 'convert': 'convert "/var/www/media/ddr/ddr-densh o-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-densho-10 00-480-1-mezzanine-c7c6795e59-a.jpg"[0] -resize \'100x100\' /var/www/media/ddr/d dr-densho-1000/files/ddr-densho-1000-480/files/ddr-densho-1000-480-1/files/ddr-d ensho-1000-480-1-mezzanine-c7c6795e59-a-thumb.jpg', 'attempted': True, 'elapsed' : '0:00:00.040682', 'status_code': 0, 'std_out': '', 'std_err': '', 'exists': Tr ue, 'size': 18209, 'islink': False}

[pkikawa]

line breaks are from the cygwin window formatting

[gjost]

Looks like it successfully created the thumbnail. the problem must be elsewhere. Could you try the web UI again?

[pkikawa]

still no thumbnails being shown.

[GeoffFroh]

Not sure about status -- may be related to problems with VirtualBox install on sthelens

[gjost]

OK, I think I've got the thumbnails issue on Dana's machine sorted.

Most objects starting with ddr-densho-1000-448 now have thumbnails. Older objects do not have thumbnails because the *-a.jpg files are not present on the VM. It's impossible to make thumbnails from non-existent access files.

Thumbnails for newer objects were not being created because of some problem with the Sqlite3 database used to keep thumbnail data (/var/lib/ddr/ddrlocal-thumbnails.dbm.db). I kept getting PermissionErrors when trying to add a file in a test collection. Django sorl-thumbnail data is disposable and can just be recreated, so I tried removing the ddrlocal-thumbnails.dbm.db file and it seems to have worked.

Along the way I uncovered a bug in the ddr-cmdln Entity object when non-int values somehow found their way into the sort field. I added the code to the develop branch for the next release, and pulled the branch to Dana's VM.

[pkikawa]

fixed - forgot to close