Generate webp responsive images

nick-mon1 commented 2 weeks ago

Summary

Update responsive image scripts to convert jpg|png to webp. Support for webp is about 96% global users.

[!NOTE] Current support for this will only touch new image uploads going forward

Preview

[Link to Preview]()

Solution

Reducing the number of variant sizes to 2 covers most of our image needs and preferring webp format fits with current best practices for image serving.

Replace jpg|png variants to only generate webp images.
Generate only 800 for mobile/tablet and 1200 for desktop/widescreen
Update template to display 800 and 1200 depending on viewport

How To Test

Run local instance and upload test image with npx gulp upload
Should generate 800 & 1200 webp versions and original image
Should generate yaml image data file
3 images should be uploaded to S3
Images are not corrupted on S3, can be viewed from S3 link

What To Test

Visit it-warning-banners-how-gsa-is-working-to-stop-unnecessarily-frightening-users/
Check network tab is serving webp and not png.
Regression test evaluating-your-agency-accessibility-just-got-a-whole-lot-easier-with-gsa-openacr-editor still serves png image

github-actions[bot] commented 2 weeks ago

:mag: Preview in Federalist

RileySeaburg commented 2 weeks ago

Just noticed this as port of the review @nick-mon1

RileySeaburg commented 2 weeks ago

@clmedders I'll await your review to merge.

RileySeaburg commented 2 weeks ago

everything looks good locally im not able to check the S3 im not sure where that link is tbh i need to find it and verify again i have access

@clmedders you can click the link in the yaml that Nick generated for now.

We'll make sure you have access today.

nick-mon1 commented 2 weeks ago

@RileySeaburg

I'm seeing some inconsistencies on my local when running npx gulp upload or using the gulpWatch.

[x] yaml file is generated
[x] updated images-stream.js to display new webp images
[x] 3 variants are created

[!WARNING] Inconsistencies in race conditions, sometimes the original image is 0KB

Console Logs

I'm seeing race conditions between copying creating directories and files the upload task.

Change of Static files detected, rebuilding site (#10).
2024-08-30 10:03:53.181 -0400
Syncing /dist/js/main.js to /
Syncing /dist/js/main.js.LICENSE.txt to /
[10:07:14] Starting 'fileTidy'...
Moving file from content/uploads/_inbox/test-upload-5-8-30.jpg to content/uploads/_working-images/to-process/test-upload-5-8-30.jpg
[10:07:14] Finished 'fileTidy' after 2 ms
[10:07:14] Starting 'cleanInbox'...
[10:07:14] Finished 'cleanInbox' after 6.32 ms
[10:07:14] Starting 'writeDataFile'...
[10:07:14] Finished 'writeDataFile' after 15 ms
[10:07:14] Starting 'processImages'...
[10:07:14] Finished 'processImages' after 5.97 ms
[10:07:14] Starting 'removeProcessedImage'...
[10:07:14] Finished 'removeProcessedImage' after 482 μs
[10:07:14] Starting 'uploadTask'...
All uploads completed successfully.
file is written
Cleanup completed
[10:07:14] Finished 'uploadTask' after 3.49 ms
Error processing variant /Users/nlyons/Documents/gsa/digitalgov.gov/content/uploads/_working-images/to-process/test-upload-5-8-30.jpg: Error: ./content/uploads/_working-images/processed/test-upload-5-8-30.jpg: unable to open for write
./content/uploads/_working-images/processed/test-upload-5-8-30_w1200.webp: unable to open for write
./content/uploads/_working-images/processed/test-upload-5-8-30_w800.webp: unable to open for write
unix error: No such file or directory
unix error: No such file or directory
unix error: No such file or directory
    at Sharp.toFile (/Users/nlyons/Documents/gsa/digitalgov.gov/node_modules/sharp/lib/output.js:89:19)
    at createResponsiveVariant (/Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:83:6)
    at /Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:103:5
    at Array.forEach ()
    at processImageVariants (/Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:102:34)
    at /Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:132:11
    at Array.map ()
    at /Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:128:41
    at FSReqCallback.oncomplete (node:fs:188:23)
    at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)
Error processing variant /Users/nlyons/Documents/gsa/digitalgov.gov/content/uploads/_working-images/to-process/test-upload-5-8-30.jpg: Error: ./content/uploads/_working-images/processed/test-upload-5-8-30.jpg: unable to open for write
./content/uploads/_working-images/processed/test-upload-5-8-30_w1200.webp: unable to open for write
./content/uploads/_working-images/processed/test-upload-5-8-30_w800.webp: unable to open for write
unix error: No such file or directory
unix error: No such file or directory
unix error: No such file or directory
    at Sharp.toFile (/Users/nlyons/Documents/gsa/digitalgov.gov/node_modules/sharp/lib/output.js:89:19)
    at processImageOriginal (/Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:64:41)
    at /Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:131:11
    at Array.map ()
    at /Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:128:41
    at FSReqCallback.oncomplete (node:fs:188:23)
    at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)
Error processing variant /Users/nlyons/Documents/gsa/digitalgov.gov/content/uploads/_working-images/to-process/test-upload-5-8-30.jpg: Error: ./content/uploads/_working-images/processed/test-upload-5-8-30.jpg: unable to open for write
./content/uploads/_working-images/processed/test-upload-5-8-30_w1200.webp: unable to open for write
./content/uploads/_working-images/processed/test-upload-5-8-30_w800.webp: unable to open for write
unix error: No such file or directory
unix error: No such file or directory
unix error: No such file or directory
    at Sharp.toFile (/Users/nlyons/Documents/gsa/digitalgov.gov/node_modules/sharp/lib/output.js:89:19)
    at createResponsiveVariant (/Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:83:6)
    at /Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:103:5
    at Array.forEach ()
    at processImageVariants (/Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:102:34)
    at /Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:132:11
    at Array.map ()
    at /Users/nlyons/Documents/gsa/digitalgov.gov/config/gulp/file-process.js:128:41
    at FSReqCallback.oncomplete (node:fs:188:23)
    at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)
[10:07:14] Starting 'fileTidy'...
[10:07:14] Finished 'fileTidy' after 516 μs
[10:07:14] Starting 'cleanInbox'...
[10:07:14] Finished 'cleanInbox' after 1.35 ms
[10:07:14] Starting 'writeDataFile'...
[10:07:14] Finished 'writeDataFile' after 1.27 ms
[10:07:14] Starting 'processImages'...
[10:07:14] Finished 'processImages' after 318 μs
[10:07:14] Starting 'removeProcessedImage'...
[10:07:14] Finished 'removeProcessedImage' after 243 μs
[10:07:14] Starting 'uploadTask'...
All uploads completed successfully.
Cleanup completed
[10:07:14] Finished 'uploadTask' after 396 μs

What To Test

run npm gulp upload
run local server and upload via workflow testing the gulpWatch command
confirm images are not corrupted on s3
test with jpg and png

RileySeaburg commented 2 weeks ago

@nick-mon1 these race conditions are like the ghost of Christmas past, the keep popping up. I'll take a look.

GSA / digitalgov.gov