Open bkniffler opened 5 years ago
bcomnes
commented
5 years ago The client uploads 15 files at a time, which helps speed up uploading of many small files.
Netlify's API has a fairly low rate limit by default though, so you may be running into that if you are experiencing slow uploads.
bkniffler
commented
5 years ago I've looked into the code, it is true. Forking child processes would help hugely too. If there really is rate limits kicking in while deploying, that would be a shame. Would be cool if netlify would accept single zip file deployment.
bcomnes
commented
5 years ago There is a Zip upload endpoint available, but there is no de-duplicated zip upload endpoint. The zip upload requires you to upload everything you want hosted for every deploy. The file manifest endpoint enables content de-duplication which is usually faster after a slower initial deploy.
One thing you may check for in your project is if every file in your site is changing or not (e.g. if you have a timestamp in your headers or something). If this is the case you end up re-uploading every file every build. If your builds deterministically regenerate unmodified pages, you can take advantage of content deduplication, where you only upload newly changed assets.
- Do you want to request a feature or report a bug? Feature
- What is the current behavior? Js-Client is very slow when you need to upload >10k pages. It doesn't take advantage of parallel uploading in a single thread (say 10 files in parallel) nor does it take advantage of multiple threads (child workers). I'm pretty confident deployment could be insanely faster with these.
See: https://github.com/netlify/cli/issues/318 A good example implementation of parallel and multithread work is nextjs export: https://github.com/zeit/next.js/tree/canary/packages/next/export