Performance issue after creating 100+ Module

[alpeshrajodiya94]

Hello,

I wanted to express my gratitude for the outstanding package provided.

While utilizing it, we've developed over 100 modules. However, we've encountered speed/performance issues. Our investigation revealed that the package loads all modules on every page during Laravel's boot process. For instance, on pages like the Login page, not all modules are necessary to load. Is there a way to skip loading certain modules selectively? For example, on Module 1's webpage, Module 10 isn't required, so can we skip it during the boot method?

We welcome any suggestions or insights from those who may have encountered similar challenges.

Thank you.

[kjostling]

Could Laravel octane be a solution? That would only boot you app once at startup. But, with 100+ modules I guess it's hard to tell if the app comply to the "dos and donts".

I usually remove stuff in the modules service provider that isnt needed for that specific module. For instance scanning for migrations in a folder is expensive, views aswell.

Otherwise perhaps its time to split the monolith into smaller pieces. 100+ modules, thats impressive!

[dcblogdev]

Looking into this, on a fresh Laravel install with no modules installed.

load time is 40 miliseconds.

I've installed 150 modules than load time goes to around 3 seconds.

I then enabled caching load time went down to around 800 miliseconds.

Installed octane but doesn't make any significant change.

There is not currently a way to selectively load modules. Looking to see if there are further improvements to be made.

[xavi7th]

@dcblogdev when you say you enabled caching, can you please share some insights on the caching strategy you implemented?

I am having the same issues with load time also and I might benefit greatly from some insights you share.

Thanks in advance.

[CasonWebDev]

I'm planning to construct a large application here where i work using modules with this library in my laravel project, knowing the quantity of modules can cause lower performance, i'm thinking to up my application in kubernetes pods where i'll isolate deployments by concerns, and depending of the concern i can disable a set of modules i'll not use in that concern, in this way i believe i'll able to stay working with multiple modules but enabling that group i'll need in certain concern or not. This architecture my make sense to you too.

The command to disable modules is available in the library with php artisan module:disable

[SneherAdor]

I then enabled caching load time went down to around 800 miliseconds.

It also works for me by enabled cache on modules configuration files.

[kjostling]

I get the following stats roughly: Clean install 0 modules: 75ms Created 150 modules: 11s Installed octane+swoole: 30ms Starting tinker: 11s Schedule run: 11s

Running scheduler (and tinker) with swoole takes about the same time as loading a page without swoole, which indicate that it's executing register & boot. If a slow scheduler is a problem one could perhaps rely on ticks & intervals in swoole rather then externally executed crontasks.

It's perhaps not a solution to the "make many modules fast" problem, rather a workaround.

[alpeshrajodiya94]

Thank you for your feedback/comments.

I'm considering if it's possible to skip loading modules based on the route. Instead of loading all modules during the boot method, we could determine which module loads on which URL or route.

For instance, we have modules named Google 2FA, Sales, Purchase, etc. Upon login, we can load Google 2FA. On the dashboard page, we can load all modules. On the Sales page route, we would load only the Sales module (skipping Google 2FA).

Any luck on this suggestion?

[solomon-ochepa]

I'm planning to construct a large application here where i work using modules with this library in my laravel project, knowing the quantity of modules can cause lower performance, i'm thinking to up my application in kubernetes pods where i'll isolate deployments by concerns, and depending of the concern i can disable a set of modules i'll not use in that concern, in this way i believe i'll able to stay working with multiple modules but enabling that group i'll need in certain concern or not. This architecture my make sense to you too.

The command to disable modules is available in the library with php artisan module:disable

I'll recommend you try microservices! Every module is a potential microservice. If anyone needs help in breaking your app down to microservices, I and my team will be more than willing to help.

[Wulfheart]

I can confirm this issue. I took a barebones repo and only added fifteen modules. This package took 150ms, around 80ms with cache, while internachi/modular has no measurable performance overhead whatsoever. To give you a baseline: On a bare Laravel application the booting process takes about 13ms. Yes, I could disable XDebug and all that stuff but I think it highlights the performance issue quite well.

[enterprime]

When looking at the file upload and the capabilities of the nwidart/modules package in Laravel, I can conclude that during the initial module creation and basic configuration, a plethora of processes are registered.

For instance, consider a package for an online store or an administrative module where migrations are created, followed by additional modules such as search, cart, personal account, and so forth.

During module initialization, the following processes are automatically registered:

// Command registration
$this->registerCommands();

// Command schedule registration
$this->registerCommandSchedules();

// Translation registration
$this->registerTranslations();

// Configuration registration
$this->registerConfig();

// View registration
$this->registerViews();

// Loading migrations from the specified path
$this->loadMigrationsFrom(module_path($this->moduleName, 'database/migrations'));

However, the extent to which these processes are utilized depends on the specific project requirements. For instance, if you have your own templating engine and configuration system, most of these functions may be disabled.

Additionally, it's worth noting the API route mapping process:

// API route mapping
$this->mapApiRoutes();

This process may be activated only in certain parts of the project, while route reading will always function.

Therefore, it's advisable to analyze each of these processes and enable only those that are necessary according to your project's requirements.

My opinion: these examples should be commented out when the module is created.

[dont-know-php]

hi there,

maybe someone will find it useful in Laravel modules optimization.

I'm using versions laravel: 8.75 and laravel-modules: 8, the modules cache is enabled

I had the same problem of slow Laravel initialization, although my project has only 15 modules with a lot of settings in a service providers (loading routes, migrations, templates, resources, etc..).

I profiled my application - the module_path helper function takes the most execution time (my code calls the function ~100 times and it takes ~180ms) because every time the method is called, a Module class object is created

I replaced all method calls with a direct path to the files, for example:

// before
$this->loadMigrationsFrom(module_path($this->moduleName, 'Database/Migrations'));

// after
$this->loadMigrationsFrom(__DIR__ . '/../Database/Migrations');

This way I saved ~180ms on the Laravel initialization

[alissn]

Hi everyone,

I’ve just created a new pull request (#1921) that addresses the performance issue we've been discussing. The changes in the PR aim to significantly improve the loading speed of the modules.

It would be great if you could test it out and provide feedback. Any suggestions for further improvement are welcome.

Thanks!

[alpeshrajodiya94]

@alissn It's worked. I appreciate your support!!!

We have made a demo on the local server and worked very well. Still, we will apply this fixing to the actual code (live server) and see how it goes.

[dcblogdev]

@alissn I've pulled down 11.1.0 I have 140 modules

Boot time is high, still need to look at this I think.

[barnicolly]

i have perfomance problem too

i think the problem is here, https://github.com/nWidart/laravel-modules/commit/d7f17015635d5dfa31ad4b61cf3747d361c06917 (removed cache, service providers load file system every time)

[dcblogdev]

that cache was actually slower then @alissn implementation but I don't think he's finished with it as yet.

[alissn]

Hi @dcblogdev,

I checked the source again and tested performance by creating 400 modules in the breeze-demo project. The boot time was around 1 second, which seems good. I’ve attached some images below, showing that around 2,500 files were loaded.

I need more information to investigate the issue further. For reference, my local setup is a Mac M1 2020, and I’m running the project with Herd.

[alissn]

Hi @barnicolly,

The cache has been deleted, and the scan method is now stored in a static property. This static property remains in memory until the cycle ends, which improves performance.

For testing purposes, this section is disabled, as tests typically create and delete 1 or 2 modules.

You can see the reference here: https://github.com/nWidart/laravel-modules/blob/f183d64e2abb06cb8e7ce50dbd480e6f9f91b76c/src/FileRepository.php#L143-L148

[dcblogdev]

ah turning on normal laravel caching really speeds things up.

In my Breeze demo with 140 modules after running php artisan optimize much quicker.

[dcblogdev]

this is with 500 modules!

no application needs this many modules. Just remember to run php artisan optimize

[Wulfheart]

this is with 500 modules!

no application needs this many modules. Just remember to run php artisan optimize

ah turning on normal laravel caching really speeds things up.

In my Breeze demo with 140 modules after running php artisan optimize much quicker.

still too slow in my opinion. But may fit 99% of the use cases.