collectstatic with compiling and remote storage is broken

[skirsdeda]

I'm using django-pipeline with S3 as storage. Less compiler is used for css.

When running collectstatic this happens: 1) Existing static files are copied to S3 For every less file: 2) less file is compiled and it's compiled version appears in source directory (same directory as less file) 3) after compiling pipeline tries to compress the css file and cannot find it because it's not uploaded to S3 yet

Maybe static files finders could be used in compressing stage instead of trying to open compiled file path on storage.

[skirsdeda]

Even if a compiled css is found in storage when compressing, it might be outdated! In case it was changed and compilation happened just before compressing, a newly compiled version is not yet uploaded to storage!

[jplehmann]

@skirsdeda I reported #486 but after analyzing it more realize it's the same issue you were having. Did #475 fix your problem? I see it's failing the test.

I would also love to have our expert @cyberdelia speak into this. Surely we are configuring something wrong here -- we can't be the first people who've ever used pipeline+s3+less?!

[jplehmann]

And is this related to the fact that our compiled CSS is showing up in same directory as our source tree? That may be an indication that we've configured something wrong which is causing this problem. This was discussed in 202 but it was unclear what the resolution was.

[skirsdeda]

@jplehmann Well yes, #475 fixed this problem and I'm using it in production. The only problem is compiled CSS showing up in same directory. When I have some more time for this I'll look into #202 as well as build errors in my pull request but I'm pretty certain this is a bug and not a configuration issue.

[skirsdeda]

@jplehmann While #475 might not be the best solution, it works because compressors look for local files instead of in remote storage. And compilers (like lessjs) output files to local directories, that's why compressors cannot find compiled js or css in remote storage.

But in case no compressors were used, compiled files wouldn't appear in remote storage (at least not until collectstatic is run for a consecutive time and compiled files are found in local directories).

So the best solution would be to fix compiler code to output compiled files straight to remote storage while compressors should remain unchanged (look for compiled files in remote storage as they do now). I'll fix the code in #475 (hopefully next week).

[jplehmann]

Thanks @skirsdeda! The fix works for me too. Hopefully we can get it passing the test and merged, or get input from @cyberdelia on how this was supposed to work.

[jcalazan]

+1 Fix worked for me as well.

[steinbachr]

+1 fix worked perfectly

[peec]

having this problem as well, atm deployment of our app is stuck here.

Stack trace:

  Traceback (most recent call last):
  File "markant_app/manage.py", line 10, in <module>
  execute_from_command_line(sys.argv)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
  utility.execute()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
  self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 393, in run_from_argv
  self.execute(*args, **cmd_options)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
  output = self.handle(*args, **options)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle
  collected = self.collect()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
  for original_path, processed_path, processed in processor:
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/storage.py", line 26, in post_process
  packager.pack_stylesheets(package)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
  variant=package.variant, **kwargs)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/packager.py", line 106, in pack
  content = compress(paths, **kwargs)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 73, in compress_css
  css = self.concatenate_and_rewrite(paths, output_filename, variant)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 137, in concatenate_and_rewrite
  content = self.read_text(path)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 220, in read_text
  content = self.read_bytes(path)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 214, in read_bytes
  file = staticfiles_storage.open(path)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/files/storage.py", line 37, in open
  return self._open(name, mode)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 366, in _open
  raise IOError('File does not exist: %s' % name)
  IOError: File does not exist: static/scss/main.css
   (ElasticBeanstalk::ExternalInvocationError)

[thomasyip]

I wish I had discovered this thread before I have the patch. Regardless, here is my PR: https://github.com/cyberdelia/django-pipeline/pull/502.

I came to the same conclusion as @skirsdeda regarding the problem. But, I was trying to solve the problem by copying the file to s3. If you would like the intermediate file or if you are not compressing the css, my patch might be helpful.

[jplehmann]

I wonder why our leader @cyberdelia would not share wisdom on this issue and the PR contribution after 2 months? Surely he must be delayed be real-life.

[thomasyip]

@jplehmann, he is probably not actively working in a related area in the last few weeks.

He might want to look at the code before making comments and it is very easy to be carry away by day-to-day busi-ness.

[peec]

Also tried many other previous versions (that did not work). 1.4.0-1.5.4, didn't get pipeline to work on any of the versions with S3.

[tobiasmcnulty]

There is a related problem which is that pipeline tries to write CSS files to the local file system along side the original LESS file, rather than in the STATIC_ROOT directory or a temporary file of its choice, before uploading to S3:

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle
    collected = self.collect()
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
    for original_path, processed_path, processed in processor:
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/pipeline/storage.py", line 27, in post_process
    packager.pack_stylesheets(package)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
    variant=package.variant, **kwargs)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/pipeline/packager.py", line 119, in pack
    paths = self.compile(package.paths, force=True)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/pipeline/packager.py", line 99, in compile
    paths = self.compiler.compile(paths, force=force)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/pipeline/compilers/__init__.py", line 58, in compile
    return list(executor.map(_compile, paths))
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/concurrent/futures/_base.py", line 581, in result_iterator
    yield future.result()
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/pipeline/compilers/__init__.py", line 45, in _compile
    outdated=outdated, force=force)
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/pipeline/compilers/less.py", line 24, in compile_file
    return self.execute_command(command, cwd=dirname(infile))
  File "/Users/tobias/.virtualenvs/pipeline-test/lib/python2.7/site-packages/pipeline/compilers/__init__.py", line 104, in execute_command
    raise CompilerError(stderr)
pipeline.exceptions.CompilerError: /bin/sh: /Users/tobias/caktus/pipeline-test/test_project/static/less/login.css: Permission denied

This is using the documented S3PipelineCachedStorage (see http://django-pipeline.readthedocs.org/en/latest/storages.html#using-with-other-storages) which claims not to need access to the local filesystem (I tried S3PipelineManifestStorage too and it also fails).

I've verified that when django-pipeline runs without S3 configured, it does not try to write any files outside of STATIC_ROOT (the correct behavior).

Here are my settings:

AWS_STORAGE_BUCKET_NAME = 'tobias-pipeline-test-staticfiles'
AWS_ACCESS_KEY_ID = '<snip>'
AWS_SECRET_ACCESS_KEY = "<snip>"
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_AUTO_CREATE_BUCKET = True

STATIC_ROOT = os.path.join(BASE_DIR, 'static') # pipeline-test/static/, NOT pipeline-test/test_project/static/
STATIC_URL = "https://%s/" % AWS_S3_CUSTOM_DOMAIN
STATICFILES_STORAGE = 'test_project.storage.S3PipelineCachedStorage'

The fixes in PR #475 and PR #502 do not address this issue.

[itsMichaelBates]

Is there any updates on this? I'm using django-pipeline 1.6.9 with S3PipelineCachedStorage, using the sass compiler and yuglify compressor and I get the same error as above:

Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/raven/contrib/django/management/__init__.py", line 41, in new_execute
    return original_func(self, *args, **kwargs)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 193, in handle
    collected = self.collect()
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 139, in collect
    for original_path, processed_path, processed in processor:
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/pipeline/storage.py", line 26, in post_process
    packager.pack_stylesheets(package)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
    variant=package.variant, **kwargs)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/pipeline/packager.py", line 106, in pack
    content = compress(paths, **kwargs)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/pipeline/compressors/__init__.py", line 75, in compress_css
    css = self.concatenate_and_rewrite(paths, output_filename, variant)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/pipeline/compressors/__init__.py", line 139, in concatenate_and_rewrite
    content = self.read_text(path)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/pipeline/compressors/__init__.py", line 222, in read_text
    content = self.read_bytes(path)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/pipeline/compressors/__init__.py", line 216, in read_bytes
    file = staticfiles_storage.open(path)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/django/core/files/storage.py", line 38, in open
    return self._open(name, mode)
  File "/home/ubuntu/code/venv/lib/python3.4/site-packages/storages/backends/s3boto.py", line 383, in _open
    raise IOError('File does not exist: %s' % name)
OSError: File does not exist: css/about.css

It copies all the files to S3, but when it gets to compiling and compressing them this error occurs.

[jplehmann]

Just noting that three years later, and I'm updating from 1.5.2 to 1.6.14 and still no patch has been completed/merged. Oh well, continuing to use a patched branch.

[skirsdeda]

@jplehmann most people have moved on to javascript bundlers like webpack or whatever, this has little use now.

[ceiphr]

This tool is great, I would like this to be resolved already because it has been roughly four years.

[hugotacito]

6 years and counting.

[jplehmann]

most people have moved on to javascript bundlers like webpack or whatever, this has little use now.

@skirsdeda was entirely correct, though I didn't like to hear it at the time. We have finally migrated to webpack fully as of today, and for the first time in forever, local runs as fast as production. I moved my thumb down over to a thumb up.

[sowinski]

So the issue still exists? Is this project dead?

[sowinski]

Just noting that three years later, and I'm updating from 1.5.2 to 1.6.14 and still no patch has been completed/merged. Oh well, continuing to use a patched branch.

Which branch do you use?