Better handling of the ValueError during the collectstatic

[karimone]

Hello everyone, I got inspiration from a code over internet and I subclassed the Pipeline Storage to avoid the raise of the ValueError in case of problem processing the css.

import logging
from pipeline.storage import PipelineCachedStorage

logger = logging.getLogger(__name__)

class MyPipelineStorage(PipelineCachedStorage):
    """
    Remove the annoying ValueError problem in collectstatic
    """
    def hashed_name(self, name, content=None):
        try:
            out = super(MyPipelineStorage, self).hashed_name(name, content)
        except ValueError:
            logger.warning('The following file is missing %s' % name)
            out = name
        return out

In the log I see which files are missing, some are from vendor packages and some not. I would like to print in the log also the file that generated the error and which line, but I can't get it in the hashed_name function. Do you have any idea how to get it?

[karolyi]

the problem is that since you catch that error, the original code exits at the error raising, where it should continue collection.

my solution was the same, except copying and overloading the whole hashed_name function, and replacing the ValueError raising part with a warning emitter.

I'll post it here on interest, but I guess you can figure this out for yourself.

There is a related issue (closed): https://github.com/jazzband/django-pipeline/issues/302

[leonardoo]

i think with a option in CSS, it can be resolved it, but i dont know what thing @cyberdelia about this issue

[karolyi]

Unfortunately it's not on django-pipeline, but on the django storage itself.

what the pipeline people could do here, is to check and remove the nonexistent files from the collection list, so that django's storage.py logic wouldn't raise an exception.

[karolyi]

pasting my custom storage, mentioned in https://github.com/jazzband/django-pipeline/issues/484#issuecomment-169493118:

import warnings

class WsStorage(
        PipelineMixin, OptimizedFilesMixin, ManifestStaticFilesStorage):

    def hashed_name(self, name, content=None):
        """
        This is a slight modification of the hashed name function in
        django/contrib/staticfiles/storage.py's HashedFilesMixin class.

        It does not raise an error when pipeline instructs it to find a
        file that is not there. Update this function from time to time,
        as the original changes.
        """
        parsed_name = urlsplit(unquote(name))
        clean_name = parsed_name.path.strip()
        opened = False
        if content is None:
            if not self.exists(clean_name) and settings.DEBUG:
                warnings.warn(
                    'The file \' % s\' could not be found with %r.' %
                    (clean_name, self), RuntimeWarning)
                # raise ValueError()
            try:
                content = self.open(clean_name)
            except IOError:
                # Handle directory paths and fragments
                return name
            opened = True
        try:
            file_hash = self.file_hash(clean_name, content)
        finally:
            if opened:
                content.close()
        path, filename = os.path.split(clean_name)
        root, ext = os.path.splitext(filename)
        if file_hash is not None:
            file_hash = '.%s' % file_hash
        hashed_name = os.path.join(path, '%s%s%s' %
                                   (root, file_hash, ext))
        unparsed_name = list(parsed_name)
        unparsed_name[2] = hashed_name
        # Special casing for a @font-face hack, like url(myfont.eot?#iefix")
        # http://www.fontspring.com/blog/the-new-bulletproof-font-face-syntax
        if '?#' in name and not unparsed_name[3]:
            unparsed_name[2] += '?'
        return urlunsplit(unparsed_name)

@cyberdelia, while I realize this is not the problem of django-pipeline, I think it would be beneficial to add it to the documentation, so other people could use it too, who struggle with the same problem.

[dangayle]

@karolyi Thank you for this, I'm running into the same exact issues. I don't mind getting warned that a css file is missing, but it's super annoying to have the whole site go 500 error on me :(

[pythdasch]

Is there any update about this ?

Because I've this issue and he's saying that the output_filename is not there. Or normally pipeline should generates it.

here are the log : 2017-02-14T12:03:30.772180423Z app[web.1]: ERROR:django.request:Internal Server Error: / 2017-02-14T12:03:30.772371825Z app[web.1]: Traceback (most recent call last): 2017-02-14T12:03:30.772382362Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response 2017-02-14T12:03:30.772387135Z app[web.1]: response = self.process_exception_by_middleware(e, request) 2017-02-14T12:03:30.772391066Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response 2017-02-14T12:03:30.772395162Z app[web.1]: response = wrapped_callback(request, *callback_args, **callback_kwargs) 2017-02-14T12:03:30.772419462Z app[web.1]: File "/app/sphax/core/views.py", line 69, in index 2017-02-14T12:03:30.772423520Z app[web.1]: return render(request, 'core/landing.html', data) 2017-02-14T12:03:30.772427954Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/shortcuts.py", line 67, in render 2017-02-14T12:03:30.772431971Z app[web.1]: template_name, context, request=request, using=using) 2017-02-14T12:03:30.772435281Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/loader.py", line 97, in render_to_string 2017-02-14T12:03:30.772437951Z app[web.1]: return template.render(context, request) 2017-02-14T12:03:30.772440186Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/backends/django.py", line 95, in render 2017-02-14T12:03:30.772442755Z app[web.1]: return self.template.render(context) 2017-02-14T12:03:30.772445028Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 206, in render 2017-02-14T12:03:30.772447461Z app[web.1]: return self._render(context) 2017-02-14T12:03:30.772449619Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 197, in _render 2017-02-14T12:03:30.772451912Z app[web.1]: return self.nodelist.render(context) 2017-02-14T12:03:30.772453930Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 992, in render 2017-02-14T12:03:30.772463425Z app[web.1]: bit = node.render_annotated(context) 2017-02-14T12:03:30.772473492Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 959, in render_annotated 2017-02-14T12:03:30.772501624Z app[web.1]: return self.render(context) 2017-02-14T12:03:30.772503685Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/loader_tags.py", line 173, in render 2017-02-14T12:03:30.772505790Z app[web.1]: return compiled_parent._render(context) 2017-02-14T12:03:30.772507802Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 197, in _render 2017-02-14T12:03:30.772509835Z app[web.1]: return self.nodelist.render(context) 2017-02-14T12:03:30.772511733Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 992, in render 2017-02-14T12:03:30.772513831Z app[web.1]: bit = node.render_annotated(context) 2017-02-14T12:03:30.772515673Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 959, in render_annotated 2017-02-14T12:03:30.772517734Z app[web.1]: return self.render(context) 2017-02-14T12:03:30.772519531Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/pipeline/templatetags/pipeline.py", line 140, in render 2017-02-14T12:03:30.772521610Z app[web.1]: return self.render_compressed(package, package_name, 'css') 2017-02-14T12:03:30.772523601Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/pipeline/templatetags/pipeline.py", line 69, in render_compressed 2017-02-14T12:03:30.772525672Z app[web.1]: package_type) 2017-02-14T12:03:30.772529181Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/pipeline/templatetags/pipeline.py", line 82, in render_compressed_output 2017-02-14T12:03:30.772531363Z app[web.1]: return method(package, package.output_filename) 2017-02-14T12:03:30.772533355Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/pipeline/templatetags/pipeline.py", line 147, in render_css 2017-02-14T12:03:30.772535422Z app[web.1]: 'url': mark_safe(staticfiles_storage.url(path)) 2017-02-14T12:03:30.772537359Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 131, in url 2017-02-14T12:03:30.772539428Z app[web.1]: hashed_name = self.stored_name(clean_name) 2017-02-14T12:03:30.772541404Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 280, in stored_name 2017-02-14T12:03:30.772543473Z app[web.1]: cache_name = self.clean_name(self.hashed_name(name)) 2017-02-14T12:03:30.772545462Z app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 94, in hashed_name 2017-02-14T12:03:30.772547519Z app[web.1]: (clean_name, self)) 2017-02-14T12:03:30.772551781Z app[web.1]: ValueError: The file 'compiled/compiled.min.css' could not be found with <core.custom_storage.GzipManifestPipelineStorage object at 0x7fe5934567d0>.

What should I do ?? I'm on django 1.9.8 and can't figure out what to do. I checked every files listed in my settings and they are all there.

The lonely thing is that there is no folder compiled. should I create it ? is it because I putted compiled.min.css ? Because it worked before. and suddently doesn't work anymore