FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

[gauthierbuttez]

I am using flowise with railway. I use pinecode to store data.

I want to upload text files and my website pages. I am using the Text Node and Puppeter web crawler Node.

I faced a first issue :

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

So after investigation, I changed the chank size to 1000 and the bug was fixed for a text file. Then I added puppeteer web scrapper. I used the sitemap method, I get the urls in puppeteer web scraper. and the issue came back. Again a problem of memory limit.

So I tried with another web scraper node and the same issue.

I have 30 text files that represent 30 books. It is about 50 Mo in total.

I have some screenshots to see my configuration if necessary. I don't know what I am doing wrong. I am a newbie in Flowise and Pinecone.

here is railway logs :

2024-07-08 11:52:17 [INFO]: ⬆ POST /api/v1/vector/internal-upsert/ceee0b68-64a4-4f7b-b684-445d5ffb3c19

Jul 08 13:54:11

Jul 08 13:54:11

<--- Last few GCs --->

Jul 08 13:54:11

Jul 08 13:54:11

[36:0x7fa3236226c0] 483060 ms: Mark-Compact 245.4 (258.7) -> 244.7 (258.7) MB, 210.03 / 0.00 ms (average mu = 0.597, current mu = 0.017) allocation failure; scavenge might not succeed

Jul 08 13:54:11

[36:0x7fa3236226c0] 483211 ms: Mark-Compact 245.7 (258.7) -> 245.0 (259.2) MB, 147.53 / 0.00 ms (average mu = 0.435, current mu = 0.025) allocation failure; scavenge might not succeed

Jul 08 13:54:11

Jul 08 13:54:11

Jul 08 13:54:11

<--- JS stacktrace --->

Jul 08 13:54:11

Jul 08 13:54:11

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Jul 08 13:54:11

----- Native stack trace -----

Jul 08 13:54:11

Jul 08 13:54:31

container event container died

So I contacted railway support and they asked me to upgrade. So I subscribed the 5$ plan. But then I have still the same issue, itried other stuff and get another issue about memory:

[36:0x7fba4eb3b6c0]  1029892 ms: Mark-Compact 2019.9 (2094.9) -> 2015.4 (2094.9) MB, 174.44 / 0.01 ms  (average mu = 0.578, current mu = 0.185) allocation failure; scavenge might not succeed

[36:0x7fba4eb3b6c0]  1030101 ms: Mark-Compact 2023.7 (2098.7) -> 2019.2 (2098.7) MB, 163.30 / 0.01 ms  (average mu = 0.462, current mu = 0.221) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

----- Native stack trace -----

container event container died

So I came back to Railway support. They told me to contact the template creator. As I use your github template in railway to deploy, I am forwarding you this issue.

I added many screenshots for more info.

Can you help me to find a solution, please?

[gauthierbuttez]

I contacted railway support on discord. They replied :

you would need to take this up with the template creator
you used a template to deploy flowise, if there is something wrong with the template, you need to take that up with the creator

I deployed from github repository. Do you have any info about this issue?

[HenryHengZJ]

My guess is that the webpage is too large to scrape, taking too much RAM for cheerio to do the scraping.

My suggestion is to maybe limit the Cheerio web scrape limit to maybe 10 pages at a time, or use other services like Apify, Firecrawl etc