Closed stevensturkop closed 1 year ago
Philreact
commented
3 years ago I've had the same problem for the last 8 months. It seems many people are having the same problem. In my opinion this should be a priority to get fixed. I have tried almost everything: deactivating windows defender, using yarn instead of npm, but nothing has worked.
sometimes it takes 1 min to change pages in development. In production it's super fast.
I can't publish my code as well since it's copyrighted.
balazsorban44
commented
3 years ago Is this the same for a fresh create next app as well? it could potentially point out if the problem is with one of the dependencies maybe or something special with your machine and next.
Philreact
commented
3 years ago Is this the same for a fresh create next app as well? it could potentially point out if the problem is with one of the dependencies maybe or something special with your machine and next.
In my case I have a custom express server. It's when I started using getServerSideProps that it's been slow only in development. I have 2 other people also coding in this project and in their machines the same problem persists.
The weird thing is that in production it's lightning speed.
balazsorban44
commented
3 years ago Production is very different from a dev server. If your teammates experience the same problem, it's more likely that the problem is in your setup. Maybe a slow database call in the test environment where the DB has less resources?
Without a reproduction though, I can only guess. you could try to create a minimal reproduction that does not use code directly from work but still produces the same effect. you might even find the reason for the issue along the way.
stevensturkop
commented
3 years ago @balazsorban44 We are using DynamoDB hosted on an ARM AWS ECS large which is pretty fast. So I believe the database is not the problem at all (especially as we query only 50 elements at a time, so it's pretty light). A minimal reproduction wouldn't showcase the problem.
We have around 500 components in the project, so all things considered I'm wondering whether NextJs has some limitations where it cannot handle compiling in development big files/pages?
stevensturkop
commented
3 years ago I just analyzed the bundle, despite the very slow first page init after npm run dev (around 1minute) the targetted page Size is 120kB and the page first load JS is 1.2MB.
timneutkens
commented
3 years ago Hey Steven, could you send me a message on https://twitter.com/timneutkens, then we can investigate it without code access.
timneutkens
commented
3 years ago @PhillipLangMartinez same for you
kartikrao
commented
3 years ago Our app www.drive.com.au is the largest automotive site in Australia.
Creating a production build with next build takes > 20 minutes on a machine with 6 cores and 32 GB of RAM. Our builds have slowed significantly since updating to Next 11.x
@timneutkens - Happy to provide access to config and code, if its of wider benefit to the community, we are also happy to pitch in if helpful.
timneutkens
commented
3 years ago Could you reach out to me on twitter as well, thanks!
kartikrao
commented
3 years ago Could you reach out to me on twitter as well, thanks!
@timneutkens - My apologies, the only social network I'm on is LinkedIn, is there an alternative way to get in touch ?
My email is redacted
timneutkens
commented
3 years ago Sent you a short email with what to run 👍
billymcintosh
commented
2 years ago I admire you helping one to one, but shouldn't the potential solutions be posted here to help everyone that has this issue?
f16z
commented
2 years ago so much pain in development.. still hunting for solutions
timneutkens
commented
2 years ago To provide a trace which only includes metadata of the application:
npm install next@canary (or yarn add next@canary)ctrl + c).next/trace file here or send it to https://twitter.com/timneutkens if you don't want to share it publicly. You can use https://gist.github.com to create a private url for the file.
.babelrc please provide itnext.config.js please provide ittailwind.config.js⚠️ The metadata in .next/trace includes full file paths to all JS/CSS. It does not include the file contents.
⚠️ The main thing we need to help investigate your application is .next/trace, package.json and such are not helpful in investigating the issue.
but shouldn't the potential solutions be posted here to help everyone that has this issue?
So far all I've done is help investigate the slowdowns for the people that reached out. There's no specific solution as it highly depends on the app and I'm going to share the findings. . I've been working on a way that we can investigate these slowdowns without needing the application code itself but it requires a bunch of steps that up till a few days ago were quite involved.
Here's a list of what I've found so far:
http://localhost:3000/about Next.js will at that point start compiling the JS/CSS for pages/about.js, this is intentional to ensure that you can have thousands of pages in a Next.js app and it would not slow down your app, otherwise you'd pay this compilation penalty at bootup
fs.readFile seems to be slow, it takes over 100ms per file, we expanded the tracing to include timing for individual filespages/index.js reads and compiles about 6867 files. What I've found is that 5557 of those are coming from @material-ui/icons/esm, assuming that the app is not using 5.5K icons I'd recommend changing how they are imported so that not all files have to be compiled. Based on debugging this trace we're adding a separate chart that will allow us to visualize the dependency graph based on a trace filef16z
commented
2 years ago I have issues installing next@canary until I ran npm config set legacy-peer-deps true
AdamDiament
commented
2 years ago
- Send the file
.next/tracefile here or send it to https://twitter.com/timneutkens if you don't want to share it publicly
@timneutkens Thanks for looking into this. I am about to send you my files privately on twitter. I love working with next but this is a major pain point, so I really appreciate that you are prioritising it. During the time the project was on I first navigated between a few different pages. Each page load took between 5 and 15 seconds by my estimates, which is my major issue day to day. I then did a fast refresh. This was nice and quick, better than usual. To get things ready for submitting this test I upgraded to webpack 5 from 4, which might explain the improvement. Thanks again!
... I am having trouble sending you the files on twitter as they are an unsupported file type, including zip. Is there another way I can share the trace with you as I don't want to post my config files here.
timneutkens
commented
2 years ago @AdamDiament I've replied to your message on twitter and also updated the instructions here. https://gist.github.com is how you can share individual files 👍
ValentinH
commented
2 years ago I had a similar issue and it was also due to @material-ui/icons and loading all of them. If you import them directly from your application code, the tree-shaking works properly. However, in our case, we were importing them from an internal package installed in the node_modules of our application which resulted in all icons being imported (only in the server bundle).
We solved it by following this documentation and adding a babel plugin to automatically rename our named imports to default imports when building and publishing our internal package.
AdamDiament
commented
2 years ago @AdamDiament I've replied to your message on twitter and also updated the instructions here. https://gist.github.com is how you can share individual files 👍
Thanks Tim sent you a gist on twitter
balazsorban44
commented
2 years ago @stevensturkop checking in, what's the status here? Were you able to reach out to Tim? Is this issue resolved?
Sameedahmad
commented
2 years ago I am having this issue to be honest it sucks to wait for so long on development.
timneutkens
commented
2 years ago @Sameedahmad see https://github.com/vercel/next.js/issues/29559#issuecomment-938431883. Please provide a trace file.
Sameedahmad
commented
2 years ago @timneutkens Here are the files, https://gist.github.com/Sameedahmad/68271c9bf81883e4014f3ac3cbc32648 Let me know if you would need anything else from my side.
timneutkens
commented
2 years ago nextjs-wordpress-starter\functions\wordpress\blocks\displayBlock.js is taking a long time to compile.
It consists of:
nextjs-wordpress-starter\components\blocks\Gutenberg\BlockGravityForm\index.jsnextjs-wordpress-starter\components\atoms\Code\Code.jsreact-syntax-highlighterSeems that react-synax-highlighter seems to be compiling support for every language in existence for your case.
Besides that it seems that reading files from the filesystem is slow in your case 🤔 Given that you have no customization in babelrc you can remove the babelrc to leverage the new compiler: https://nextjs.org/docs/advanced-features/compiler
dottodot
commented
2 years ago @timneutkens I've sent you my trace on twitter
mustafa-hanif
commented
2 years ago @timneutkens I've added my trace here. https://gist.github.com/mustafa-hanif/8deae35eb4113684c3ace37fbc610568
timneutkens
commented
2 years ago @mustafa-hanif I had a look at your trace and it's the same issue as the others, you're importing all of the material-ui icons library (5.5K of them) in one module. That module seems to be packages/hotels/src/ui/src/Dialog/DialogWithHeading.js.
└─ module /Users/mustafa.hanif/elaa-web/packages/hotels/src/ui/src/Dialog/DialogWithHeading.js 1s (total 281 ms, self 21 ms) [next-swc-loader 19 ms] └─ module @mui/icons-material (@mui/icons-material/esm/index.js + 9899) 260 ms (self 🔥327s) [read-resource 🔥325s]
jason-yolabs
commented
2 years ago Is there a public tool available to parse the .next/trace file? Would love to dive deeper without bugging Tim :)
Edit: https://github.com/vercel/next.js/blob/canary/scripts/trace-to-tree.mjs is the tool (and it's awesome!)
Carduelis
commented
2 years ago I've experienced the issue with @mui/material and my own npm package @gorazdo/tomui
The problem comes from a lot of modules to process (11k) like in this article
Unfortunately, this solution doesn't work. The problem is the regex (which is a completely mysterious thing for a non-rust developer. I've even tried Rust Regex online tool ), with no luck.
decreased 3-5 times dev build/rebuild/fast-refresh time decreased amount of modules by 10 times from 11k to 1k modules.
I've managed it using the following config:
modularizeImports: {
'@gorazdo/tomui': {
transform: '@gorazdo/tomui/{{member}}',
},
'@mui/material': {
transform: '@mui/material/{{member}}',
},
'@mui/icons-material/?(((\\w*)?/?)*)': {
transform: '@mui/icons-material/{{ matches.[1] }}/{{member}}',
},
},But I had to update imports manually for useTheme, ThemeProvider, styled, lighten and darken in my case
from:
import { Box, lighten, Typography, styled } from '@mui/material'
// for lighten and styled the regex didn't work neither the simple {{member}} way. so it is manually changedto
import { Typography, Box } from '@mui/material'
import { styled, lighten } from '@mui/material/styles' To fix my library I had to use default export for each file which is going to be modularized.
@mui/material ?
joulev
commented
2 years ago @timneutkens Hi, I am having the same problem. You can look at the source code directly at https://github.com/joulev/ezkomment, or the trace file below
.next/traceI also ran trace-to-tree.mjs (and from that I identified one issue which led to a 20 seconds improvement in next dev). The server compilation time is reported to be 31 seconds (out of total 36.2s), but everything inside that looks alright to me, and I can't see how the numbers add up to 31s:
trace-to-tree.mjs outputCould you please see what is wrong here? Thanks.
oalexdoda
commented
2 years ago I've experienced the issue with
@mui/materialand my own npm package@gorazdo/tomuiThe problem comes from a lot of modules to process (11k) like in this article Unfortunately, this solution doesn't work. The problem is the regex (which is a completely mysterious thing for a non-rust developer. I've even tried Rust Regex online tool ), with no luck.My Solution (3-5 times faster)
decreased 3-5 times dev build/rebuild/fast-refresh time decreased amount of modules by 10 times from 11k to 1k modules.
I've managed it using the following config:
modularizeImports: { '@gorazdo/tomui': { transform: '@gorazdo/tomui/{{member}}', }, '@mui/material': { transform: '@mui/material/{{member}}', }, '@mui/icons-material/?(((\\w*)?/?)*)': { transform: '@mui/icons-material/{{ matches.[1] }}/{{member}}', }, },But I had to update imports manually for
useTheme,ThemeProvider,styled,lightenanddarkenin my casefrom:
import { Box, lighten, Typography, styled } from '@mui/material' // for lighten and styled the regex didn't work neither the simple {{member}} way. so it is manually changedto
import { Typography, Box } from '@mui/material' import { styled, lighten } from '@mui/material/styles'To fix my library I had to use
default exportfor each file which is going to be modularized.Questions
- How to deal with this regex to make it work with
@mui/material?- How to deal with modularization of files that have only named exports?
- Can we automate modularization somehow?
Hey @Carduelis , managed to find a regex that works with hooks and stuff? Thanks!
zigang93
commented
2 years ago I've experienced the issue with
@mui/materialand my own npm package@gorazdo/tomuiThe problem comes from a lot of modules to process (11k) like in this article Unfortunately, this solution doesn't work. The problem is the regex (which is a completely mysterious thing for a non-rust developer. I've even tried Rust Regex online tool ), with no luck.My Solution (3-5 times faster)
decreased 3-5 times dev build/rebuild/fast-refresh time decreased amount of modules by 10 times from 11k to 1k modules.
I've managed it using the following config:
modularizeImports: { '@gorazdo/tomui': { transform: '@gorazdo/tomui/{{member}}', }, '@mui/material': { transform: '@mui/material/{{member}}', }, '@mui/icons-material/?(((\\w*)?/?)*)': { transform: '@mui/icons-material/{{ matches.[1] }}/{{member}}', }, },But I had to update imports manually for
useTheme,ThemeProvider,styled,lightenanddarkenin my casefrom:
import { Box, lighten, Typography, styled } from '@mui/material' // for lighten and styled the regex didn't work neither the simple {{member}} way. so it is manually changedto
import { Typography, Box } from '@mui/material' import { styled, lighten } from '@mui/material/styles'To fix my library I had to use
default exportfor each file which is going to be modularized.Questions
- How to deal with this regex to make it work with
@mui/material?- How to deal with modularization of files that have only named exports?
- Can we automate modularization somehow?
this work really well.. I finally able to remove that .babelrc file
ctotheameron
commented
2 years ago 👋 @timneutkens - a bit late to the party here and not sure if you're still reviewing these, but I'm seeing some extreme startup times on next dev
event - compiled client and server successfully in 609.1s (1709 modules)Strangely, next build is relatively quick in comparison.
See attached: https://gist.github.com/ctotheameron/4737a779386aac592d3f7fde13016d83
Edit:
I should note that this just started happening to me when I upgraded to a newer version of node (16.14.0). When I rollback to 14.19.0 it's taking ~ 1660 ms. However, other engineers on my team are seeing the multi-minute startup times on any version of node 🤷 .
timneutkens
commented
2 years ago Hey @ctotheameron, I had a look at the trace file and it seems that the slowdown is coming from vanilla-extract. If you have a look at the output of our trace file interpreter: https://gist.github.com/timneutkens/52f5cf1d1991b2789fbaa81885638c00 (you can run it yourself, it's this script: https://github.com/vercel/next.js/blob/canary/scripts/trace-to-tree.mjs)
It seems the large majority of the time spent is caused by the plugin added: https://gist.github.com/ctotheameron/4737a779386aac592d3f7fde13016d83#file-next-config-js-L2-L3
gehaktmolen
commented
2 years ago @Carduelis which next version are you using? I can't use modularizeImports with version 12.2.0; " You have defined experimental feature (modularizeImports) in next.config.js that does not exist in this version of Next.js. Please remove it from your configuration."
I am experiencing the same problem as you though, when including material ui version 5.
Anyway. I started a new next project for giggles. Noticed the following already; Adding Sentry.io increases initial compile time from about 200ms to 10 seconds, and loading a clean index page increases compile time from 1 second to 15 seconds, with almost the same amount of modules (in my case 333).
Also making sure that you import your mui components as import Card from '@mui/material/Card, instead of import { Card } from '@mui/material'; will help out significantly.
pcmaffey
commented
2 years ago Is there a way to lazily compile linked pages in the background once the current page is done?
apostolos107
commented
2 years ago Hello, @timneutkens. We have a similar slowdown problem. Here is our trace Could you have a look?
shunsusa
commented
1 year ago Enabling swcMinify might speed things up.
// next.config.js
swcMinify: true,
bayucandra
commented
1 year ago Hi, attached is my trace result. Besides I have a common problem which is about material-ui, seems I have a problem with my redux import. Is that correct?
I am using Next v10.2.3. Node v16.
I am pretty confused about how to analyze that because it is in my project directory, not inside node_modules directory. Please let me know your advice guys. Thanks in advance.
apostolos107
commented
1 year ago we can't use swcMinify: true because we use babel in order to use antd components. Is any way to speed up our build or import antd components with swc compiler? As I posted above this our trace, babel config and next config
lveillard
commented
1 year ago In case it helps somebody, in our case the line causing the extremely slow compilation was this one:
//tailwind.config.js
/** @type {import('tailwindcss').Config} */
module.exports = {
// eslint-disable-next-line global-require, import/no-extraneous-dependencies
presets: [require('tailwind-preset')],
content: [
'./src/app/**/*.{js,ts,jsx,tsx}',
'./src/pages/**/*.{js,ts,jsx,tsx}',
'./src/components/**/*.{js,ts,jsx,tsx}',
'./src/layout/**/*.{js,ts,jsx,tsx}',
'../../packages/ui/**/*.{js,ts,jsx,tsx}', // THIS ONE!
],
};In our case we are using turborepo and when adding the package/ui the compilation time goes from 2-5 seconds to >500 seconds
agyarmati
commented
1 year ago For reference, in our case the slow dev experience was caused by the app getting compiled twice, which is a known issue in 12.2.5: https://github.com/vercel/next.js/discussions/39901
Updating to 12.3.1 fixed the issue completely.
anthonyalayo
commented
1 year ago To add to what @ValentinH said and @Carduelis said, the issue is not with Next.js, it is with @mui and webpack. I ended up filing an RFC on @mui here which covers a lot of the details.
While the link above has details explaining the issue, I can reiterate here. Webpack must traverse all imports when running in either dev or prod, as it does not know how to / can't optimize at the code level. It is only when terser (a webpack default plugin) minifies bundles that you end up with the "tree shaken" version.
With any barrel file, your development server will have all modules (and their imports... and their imports...) loaded into memory while it's running. This also happens when building, but it is more hidden since you aren't running an HMR server at the same time.
The best solution, which I also recommended @mui to do internally, is to configure either a babel or swc plugin to transform your imports. @mui ironically has docs on that here. This will allow you to keep the DX you want while significantly speeding up your webpack builds.
I think we should consider this issue closed.
Vallo
commented
1 year ago Is there a way to modularizeImports or somehow optimise the bundling when an external package is importing @mui/icons-material incorrectly?
Currently I am using @solana/wallet-adapter-material-ui
which uses @mui/icons-material like this:
import { FileCopy as CopyIcon, LinkOff as DisconnectIcon, SwapHoriz as SwitchIcon } from '@mui/icons-material';
When compiling with this library, it results in 14162 modules. If I remove it, I get just 3332 modules.
Is there a way to optimise it? Thanks
anthonyalayo
commented
1 year ago @Vallo not without changing it, I also had to do the same for a library I was using.
rtroncoso
commented
1 year ago In case it helps somebody, in our case the line causing the extremely slow compilation was this one:
//tailwind.config.js /** @type {import('tailwindcss').Config} */ module.exports = { // eslint-disable-next-line global-require, import/no-extraneous-dependencies presets: [require('tailwind-preset')], content: [ './src/app/**/*.{js,ts,jsx,tsx}', './src/pages/**/*.{js,ts,jsx,tsx}', './src/components/**/*.{js,ts,jsx,tsx}', './src/layout/**/*.{js,ts,jsx,tsx}', '../../packages/ui/**/*.{js,ts,jsx,tsx}', // THIS ONE! ], };In our case we are using turborepo and when adding the package/ui the compilation time goes from 2-5 seconds to >500 seconds
fixed it for me, thanks!
sneko
commented
1 year ago @anthonyalayo to be honest I tried what you pointed https://mui.com/material-ui/guides/minimizing-bundle-size/#option-two-use-a-babel-plugin and it did not do much about speed :(
It's a total disaster ^^... 30 seconds to launch my next dev, and for each page is about 10s, welcome to 2023 :D
If you have other tips I take 👍
pimmee
commented
1 year ago I think we should consider this issue closed. @anthonyalayo
I've added the modularizeImports MUI transforms and tried 10 other things mentioned in other places, but still the DX with NextJS is very bad for many people due to extremely slow page loads. The DX in other tools in the ecosystem (e.g. vite, even create-react-app) don't suffer from this, so I don't think this issue should be closed.
The development experience with NextJS is advertised greatly by Vercel, so it is frustrating when it does not live up to its promises.
This is how the compiling look like during a page navigation in my fairly large project:
Here's some screenshots when I tried adding dependencies one by one in a new project with next 13.1.6:
After adding date-fns:
After adding modularizeImport on date-fns:
So the modularizeImports definitely works, but only a small number of libraries organizes their imports in a way that is compatible.
After adding @mui/x-data-grid (@mui/material, @mui/icons-material, date-fns, @mui/x-data-grid in total)
With ~1400 modules compiled and taking 1.7 seconds for page load in an empty project just by adding a few dependencies, it's easy to see how it gets out of hand in a large project.
I wonder if a lot of the slow page loading is due to prefetching of pages? In the following screenshot I hovered my mouse over a few links in my sidebar before navigating which seems to create a waterfall effect and cause delay.
If NextJS cannot improve the page load performance due to Webpack, I still think it can help with tooling to improve the DX. For example:
modularizeImport and do it automatically
...etcHappy to share private repo with NextJS developers if it helps solving this issue.
timneutkens
commented
1 year ago You can visualize the trace yourself using canary/scripts/trace-to-tree.mjs if you're curious.
node scripts/trace-to-tree.mjs /path/to/traceOption to disable prefetching globally in development
Prefetching is disabled in development for pages. For app it's enabled but only on hover. If you want to completely disable prefetching you can pass prefetch={false}.
Debug option to log which modules was compiled
See above.
Automatically detect libraries compatible with modularizeImport and do it automatically
This is potentially breaking which is why we don't have a default config.
Overall the issues with large amounts of modules was one the reasons we started building Turbopack: https://turbo.build/pack. Which is currently in alpha. It's a completely new architecture to allow processing many more modules quickly, including for next build.
What version of Next.js are you using?
11.1.2
What version of Node.js are you using?
14.18.0
What browser are you using?
Chrome
What operating system are you using?
Windows 10
How are you deploying your application?
AWS ECS
Describe the Bug
I've been using NextJs for years and recently it has been very hard to work with because very slow in development. After
npm run dev, I go tolocalhost:3000. From there the page can take up to 60 seconds to display. Then when the first page finally displays, each code change fast refresh or page transition SSR (compilation) takes between 15-20 seconds, sometimes more than 30 seconds, and sometimes it even doesn't work so I have to refresh the page.Expected Behavior
Page load + compilation should be faster.
To Reproduce
Unfortunately, I cannot send a reproducible UI because the project I work on is pretty big and the UI is under NDA.
Maybe without a reproducible UI, you guys have thoughts about how to improve/fix the compilation time on windows. Maybe some of you have faced the same problem and fixed it somehow. I'm all ears.
My package.json
{ "name": "myname", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start -p 3002" }, "dependencies": { "@react-google-maps/api": "^2.0.2", "@sentry/browser": "^6.7.2", "@sentry/react": "^6.7.2", "@sentry/tracing": "^6.7.2", "@sentry/webpack-plugin": "^1.15.1", "@stripe/react-stripe-js": "^1.4.0", "@stripe/stripe-js": "^1.13.1", "accept-language-parser": "^1.5.0", "aws-sdk": "^2.802.0", "base-64": "^1.0.0", "cors": "^2.8.5", "dotenv": "^8.2.0", "hls.js": "^0.14.16", "iban": "0.0.14", "js-sha256": "^0.9.0", "libphonenumber-js": "^1.9.11", "localized-countries": "^2.0.0", "lodash": "^4.17.21", "moment": "^2.29.1", "next": "^11.1.2", "next-compose-plugins": "^2.2.1", "next-fonts": "^1.5.1", "next-images": "^1.8.1", "next-seo": "^4.17.0", "nookies": "^2.5.0", "platform": "^1.3.6", "qrcode": "^1.4.4", "randomstring": "^1.1.5", "react": "^17.0.2", "react-cropper": "^2.1.4", "react-datepicker": "^3.3.0", "react-device-detect": "^1.15.0", "react-dom": "^17.0.2", "react-dotdotdot": "^1.3.1", "react-ga": "^3.3.0", "react-geocode": "^0.2.2", "react-google-maps": "^9.4.5", "react-lines-ellipsis": "^0.14.1", "react-loading-skeleton": "^2.2.0", "react-query": "^3.13.2", "react-textarea-autosize": "^8.3.2", "sass": "^1.29.0", "stripe": "^8.137.0", "uuid": "^8.3.1", "video.js": "^7.11.0", "videojs-contrib-dash": "^4.0.0" }, "devDependencies": { "@svgr/webpack": "^5.5.0" } }