Skip post process if already exists

evansd / whitenoise

Radically simplified static file serving for Python web apps

https://whitenoise.readthedocs.io

MIT License

2.51k stars 148 forks source link

Skip post process if already exists #250

Open thenewguy opened 4 years ago

thenewguy commented 4 years ago

I am working on integration testing and run collectstatic before testing so that the same static backend can be used during the tests.

This is fast with the default storage because it skips files that are already in the static root and have not changed. However, it takes some time to make all of the files by this backend and it slows down iterative testing and is somewhat of a pain point.

It seems like it should be possible to only rebuild if needed, otherwise just emit the name and path to go in the manifest. Perhaps via a setting?

thenewguy commented 4 years ago

@evansd Would you accept a PR with this for immutable files? I.e. if admin/js/vendor/jquery/jquery.min.220afd743d9e.js exists, do not re-upload it?

evansd commented 4 years ago

Is this issue specific to Whitenoise or do you get the same behaviour with Django's ManifestStaticFilesStorage?

thenewguy commented 4 years ago

@evansd This is a feature request I would like to implement for Whitenoise. Sorry for the poor wording

It offers similar benefit to WHITENOISE_KEEP_ONLY_HASHED_FILES. This would greatly speed up the collectstatic part of whitenoise's storage backend for us.

I have not investigated how the actual approach would work because the concept has not been approved. But the pseudo code would be something like this:

if WHITENOISE_SKIP_EXISTING_IMMUTABLE_FILES, then check to see if the hashed file already exists before copying it. If it does exist, add file to manifest and continue to the next file. If it does not exist, proceed as normal.

Ideally this would let us skip recreating the .gz and .br files and assuming the compression routine has already done its job. So we would just detect if the .gz file and the .br file exist, and if so, add them to the manifest but skip recreating them

evansd commented 4 years ago

Ah OK, I understand. I can see that a feature to skip recompressing files might be useful.

I'd say the best thing would be to implement this for your own use-case (either as a subclass of the Whitenoise backend or a forked copy) and check that it solves your problem and works as intended. And then we can think about the best way of getting that feature merged upstream.

thenewguy commented 4 years ago

@evansd good news. will implement in a way that makes it possible to contribute