lin-d-hop
commented
1 year ago We can scale N8N whenever we need to by scaling the server. This is very easy to do. It is easy to configure the amount of memory in Cloudron too. So probably the approach of throwing more metal at N8N when we need to will resolve our problems for quite some time :)
Description:
Since n8n is self hosted by OFN. There is a limited amount of storage/memory available for it (this can be increased/decreased by paying more/less for the hosting space). All the n8n instances/accounts share this storage/memory. Therefore, if multiple instances are running memory intensive workflows at the same time then some running workflows may crash. We would get an error like: n8n has run out of memory or execution can't be found etc. But those same workflows may work fine if then are run at different time.
I had this issue multiple times when I run a memory intensive workflows like working with bulk products API with an output of over 1000+ variants using function nodes or other modules like airtable.
As more and more people/instances start using n8n, we might face the memory issue often which will in turn make n8n automations less reliable.
There are ways/good practices to reduce memory usage of a workflow for ex. using sub-workflows for memory intensive tasks in conjunction with split in batches, avoid using code/function module if possible etc. Obviously we can have a list of methods/tips to reduce memory of a workflow, but it might not be that effective. We need a plan on how to increase in n8n usage overtime while limiting the memory issue. Alternatively we can look a hosting n8n in multiple locations. Instances with heavy use of n8n can self host their own n8n, while smaller instances can still use the common n8n server. Well theses are some thoughts as of now. Need to brainstorm and check what other options we have.