EPIC: Clarifying Goals/Roadmap for Image Uploads Project

[nelsonic]

Apologies for not making this clear earlier. I thought it was. But on reflection it really wasn't. 🤦‍♂️ So I'm going to try and clarify. 🤞

We need a standalone image uploading App because images are used in many situations:

A. General Communication: we send many images (e.g: via Signal) each day; images often communicate an idea considerably faster than text. B. Business: accounting, banking, insurance, design, law, architecture, engineering, construction, etc. all employ image uploading, sharing, review & approval. If we have good image uploading and OCR we can build great business tools. C. Social: there’s a reason the most valuable valuable social networks (Instagram, Snapchat, TikTok, etc.) are based on images (and now video). It comes back to the communication speed; our eyes can process an image very fast and thousands of words can be conveyed in a single image.

We can build many different kinds of Apps if we get image uploading “right”.

The Goals of the Image Uploads (imgup) Project are:

1. A fully functional image uploader that works on any device (Mobile, Desktop & API) with an intuitive/familiar interface that a completely non-technical person can use to easily upload a image/photo. 📸 There should be no "landing page" or button/link to click to upload; zero friction.

2. A robust REST API with excellent error handling + messages that we can use from any other platform e.g. our Flutter App. 📱

3. Comprehensive documentation + instructions for recreating the upload code, interface and REST API that anyone can follow (yes, including ClosedAI... 🤖🙄). The better our docs the more people will consider contributing improvements. 🤞 The better the docs on a "showcase" project the much easier it is for you to get your next "job" or recruit people to your Startup. 💭

4. Real Tests that fail when something goes wrong. Not Mocks that always pass and give false positives/negatives. We have built projects that upload to AWS S3 in the past and written real tests. e.g: https://github.com/dwyl/aws-lambda-deploy/blob/master/test/05_upload.test.js If we cannot figure out how to create real tests (AKA "end to end tests") for our project(s) then we have failed. Yes, I know that this is ironic considering we have the aws-sdk-mock project, but that is for calling specific AWS methods in a complex JS project where people don't want end-to-end tests. 👩‍🔬

5. Logging which device made the upload so that we have some metrics: 📈

6. Auth to restrict who can upload files beyond a certain size e.g: allow anonymous uploads up to 1mb to test the demo App and then if people authenticate allow uploading up to 10mb and for paying customers e.g. Wedding Photographers taking 43 megapixel raw photos allow uploads of unlimited size. Spin up a dedicated Fly.io instance to avoid crashing on 100mb+ files.

7. Re-size and optimise (compress) large images to save bandwidth on mobile devices. 📱 We already have a good proof-of-concept for this thanks to @LuchoTurtle 🎉 But it currently only creates 200px thumbnails that are tiny on any screen. e.g: https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhe5aFXPKonoWchLRYo9yJDqWyUdUeTQ6MQQJsTWnzzNum.jpg Becomes: https://s3.eu-west-3.amazonaws.com/imgup-compressed/zb2rhe5aFXPKonoWchLRYo9yJDqWyUdUeTQ6MQQJsTWnzzNum.jpg We need to make resizing dynamic (dependent on the screen size of the device) or pick a frequently used screen size like iPhone and iPad and render for that. See: https://github.com/dwyl/imgup/issues/99

8. Use URL shortening & redirection (proxy) to serve images with shorter URLs e.g: Instead of requesting the image directly from S3: https://s3.eu-west-3.amazonaws.com/imgup-compressed/zb2rhe5aFXPKonoWchLRYo9yJDqWyUdUeTQ6MQQJsTWnzzNum.jpg The imgup app acts as a proxy and the URL can be much shorter: https://dwy.is/zb2rh uses the minimum number of characters from the CID so the URL is as short as possible. Request latency should be minimised to

Related Side Quest: https://github.com/dwyl/linky/issues/13 🔗

9. Allow requesting several sizes of compressed images via URL params without changing the filename e.g:

10. Eventually migrate off AWS S3 to something else to not waste cash... 💸
    e.g: BackBlaze https://github.com/dwyl/technology-stack/issues/63 -> https://github.com/dwyl/imgup/issues/98 or investigate IPFS But we need to run tests on request/response times (latency), durability and figure out how to do the resizing/compression. 💭

As you can tell from this list, we have already shipped a couple of the features. But there's still quite a lot left. I will work on splitting it all out into sub-issues/tasks. 🧑‍💻 ⏳

[nelsonic]

Long time friend @jupiter 🪐 created https://github.com/jupiter/rust-image-worker a few years back. ⭐ The demo still works: https://factorymethod.uk/image.png?mode=fill&width=400&height=400&origin=http://factorymethod.uk/Apollo_17.jpeg

400px:

40px:

This looks like a very good starting point for what we want in terms of dynamic (URL param) re-sizing. 🎉

[LuchoTurtle]

For URL param re-sizing, I've found https://dev.to/horaceshmorace/an-open-source-and-completely-serverless-image-resizing-service-in-aws-589b (with an accompanying https://github.com/HoraceShmorace/Image-Flex repo) for image re-sizing on AWS (what we're using).

It might entail a revamp of what we have and discarding essentially our resizing lambda function, compressed bucket and a few other services.

However:

While Image-Flex allows you to indicate a region to use other than us-east-1, CloudFront requires us-east-1. Until I figure out a workaround, don't attempt to deploy in any region other than us-east-1.

[nelsonic]

Consider the following image: https://world-celebs.com/public/media/resize/800x-/2019/8/5/porter-robinson-3.jpg

We can resize it dynamically without owning/hosting it: https://factorymethod.uk/image.png?mode=fill&width=200&height=200&origin=https://world-celebs.com/public/media/resize/800x-/2019/8/5/porter-robinson-3.jpg

The only drawback I can see is that it doesn't yet work for GIF or webp as per: https://github.com/jupiter/rust-image-worker/issues/3 But this is preferable to being forced to use the us-east-1 AWS region and then having to get consent from people in the EU because of GDPR ... 🙃 I would much rather use Cloudflare than AWS.

Full disclosure: 2% of my holdings are Cloudflare (NYSE: NET 📈 We know a few people that work there and love the work they're doing! 😍 Not saying AWS, S3 and Lambda are "bad", they are "fine". 👌 Oh and Jeff is more than rich enough already! 🤑

But this is definitely side-tracking and we should discuss implementation of "advanced" resizing in a dedicated issue rather than this EPIC. 💭

Did you have any further thoughts on any of the items in the list above?