Open hkbarton opened 4 years ago
hi @ac360 @medikoo can you please help to review this proposal? Thanks a lot!
@hkbarton it's definitely a painful issue that needs to be solved.
I wonder though, whether that can be tackled without introducing an extra configuration option (it'll be great to avoid jumping on a road, where we enrich our config with tens of properties that require fine tuning,as e.g. it's the case with Webpack now), ofc if there's no better way, then there's no argument.
Ideally our setup should be smart enough to detect what have changed and submit just diff, without asking a user to setup some config to have that working.
What if we could confirm with backend just by sending hashes which files differ against uploaded state, and having an answer upload only those changed? Then on component side, we can update archive with new files. Would that be possible?
@hkbarton it's definitely a painful issue that needs to be solved.
I wonder though, whether that can be tackled without introducing an extra configuration option (it'll be great to avoid jumping on a road, where we enrich our config with tens of properties that require fine tuning,as e.g. it's the case with Webpack now), ofc if there's no better way, then there's no argument.
Ideally our setup should be smart enough to detect what have changed and submit just diff, without asking a user to setup some config to have that working.
What if we could confirm with backend just by sending hashes which files differ against uploaded state, and having an answer upload only those changed? Then on component side, we can update archive with new files. Would that be possible?
@medikoo In my mind, this is definitely possible. While it's a great/good engineer challenge I think it's doable and I agree with you that less configuration leads to better developer experience. We can have a list of hashes of every file that will be upload (we already need this if we want to compute the one hash of entire folder) and pass that list to the backend, then the backend can compare that with the stored version of the hash list and return the diff. Then the CLI will be able to just pack the diff files and upload.
Problem
Today, if the user tries to deploy the project which including large source code, we'll upload the entire source code every time user runs the deploy command. Most parts of the source code are usually 3rd party dependencies or assets which don't change too often, e.g. normal
NextJS
project includes 100mb node modules and some static assets like images. All those stuff won't change too much, and usually, they only need to be deployed on the very first deployment. It's slow and unnecessary to upload them every time we run deploy.Solution
We can leverage layer to hold common/shared resources like node modules or static assets and only update layer when the content of those common/shared resources are changed. We can use hash computing and comparing to figure out whether those resources are changed or not during the deployment process and be able to avoid re-upload the same thing if they're not changed.
Design
The entire process of hash computing and uploading skip can be performed as follow:
serverless.yml
to tell the CLI tool which folders/content are subject to not change often and can be skipped if the content of those doesn't change by addingaccelerate
section intoinputs.src
. e.g.inputs: src: src: ./src accelerate: