Race condition between cache cleaning and temporary file creation

[giorgiosironi]

https://github.com/loris-imageserver/loris/blob/development/bin/loris-http_cache_clean.sh#L59 removes empty directories. However, if the cache clean runs while traffic is being served, there is a race condition with the creation of temporary files:

Traceback (most recent call last):
  File "/opt/loris/loris/webapp.py", line 395, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/loris/loris/webapp.py", line 346, in wsgi_app
    response = self.route(request)
  File "/opt/loris/loris/webapp.py", line 389, in route
    return self.get_img(request, ident, region, size, rotation, quality, fmt, base_uri)
  File "/opt/loris/loris/webapp.py", line 552, in get_img
    src_fp, src_format = self.resolver.resolve(ident)
  File "/opt/loris/loris/resolver.py", line 350, in resolve
    cached_file_path = self.copy_to_cache(ident)
  File "/opt/loris/loris/resolver.py", line 331, in copy_to_cache
    with tempfile.NamedTemporaryFile(dir=cache_dir, delete=False) as tmp_file:
  File "/usr/lib/python2.7/tempfile.py", line 475, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib/python2.7/tempfile.py", line 244, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
OSError: [Errno 2] No such file or directory: '/ext/loris/cache-resolver/lax/5e/aec/e8f/6e4/548/522/137/903/6f4/4d4/68d/tmpIMyGPB'

https://github.com/loris-imageserver/loris/blob/development/loris/resolver.py#L317 creates the cache dir, while https://github.com/loris-imageserver/loris/blob/development/loris/resolver.py#L331 creates the file in it. When the directory is removed between these two operations, the request fails with this exception.

[giorgiosironi]

However, it's not possible to add the same atime filter to the empty directories, as the find operations will have set it to a recent time.

[bcail]

I guess we could download the file to the tmp directory, and then create the cache dir and move the file in without having the requests call in-between. I think there would still technically be a race condition, though.

[giorgiosironi]

Would mitigate, but the same problem reappears between the creation and the movement. If I find a solution I'll post it here.

[giorgiosironi]

My solution is to create two servers behind a load balancer, so that when performing cache cleaning the server can be detached first to avoid interference from real traffic: https://github.com/elifesciences/iiif-formula/blob/master/salt/iiif/config/usr-local-bin-loris-cache-clean It's not a portable solution in general, but someone may find it useful.