[BUG] allocation failure; scavenge might not succeed

[gauthierbuttez]

I am using flowise with railway. I use pinecode to store data.

I want to upload text files and my website pages. I am using the Text Node and Puppeter web crawler Node.

I faced a first issue :

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

So after investigation, I changed the chank size to 1000 and the bug was fixed for a text file. Then I added puppeteer web scrapper. I used the sitemap method, I get the urls in puppeteer web scraper. and the issue came back. Again a problem of memory limit.

So I tried with another web scraper node and the same issue.

I have 30 text files that represent 30 books. It is about 50 Mo in total.

I have some screenshots to see my configuration if necessary. I don't know what I am doing wrong. I am a newbie in Flowise and Pinecone.

here is railway logs :

2024-07-08 11:52:17 [INFO]: ⬆ POST /api/v1/vector/internal-upsert/ceee0b68-64a4-4f7b-b684-445d5ffb3c19

Jul 08 13:54:11

Jul 08 13:54:11

<--- Last few GCs --->

Jul 08 13:54:11

Jul 08 13:54:11

[36:0x7fa3236226c0] 483060 ms: Mark-Compact 245.4 (258.7) -> 244.7 (258.7) MB, 210.03 / 0.00 ms (average mu = 0.597, current mu = 0.017) allocation failure; scavenge might not succeed

Jul 08 13:54:11

[36:0x7fa3236226c0] 483211 ms: Mark-Compact 245.7 (258.7) -> 245.0 (259.2) MB, 147.53 / 0.00 ms (average mu = 0.435, current mu = 0.025) allocation failure; scavenge might not succeed

Jul 08 13:54:11

Jul 08 13:54:11

Jul 08 13:54:11

<--- JS stacktrace --->

Jul 08 13:54:11

Jul 08 13:54:11

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Jul 08 13:54:11

----- Native stack trace -----

Jul 08 13:54:11

Jul 08 13:54:31

container event container died

So I contacted railway support and they asked me to upgrade. So I subscribed the 5$ plan. But then I have still the same issue, itried other stuff and get another issue about memory:

[36:0x7fba4eb3b6c0]  1029892 ms: Mark-Compact 2019.9 (2094.9) -> 2015.4 (2094.9) MB, 174.44 / 0.01 ms  (average mu = 0.578, current mu = 0.185) allocation failure; scavenge might not succeed

[36:0x7fba4eb3b6c0]  1030101 ms: Mark-Compact 2023.7 (2098.7) -> 2019.2 (2098.7) MB, 163.30 / 0.01 ms  (average mu = 0.462, current mu = 0.221) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

----- Native stack trace -----

container event container died

So I came back to Railway support. They told me to contact the template creator. As I use your github template in railway to deploy, I am forwarding you this issue.

I added many screenshots for more info.

Can you help me to find a solution, please?

[gauthierbuttez]

Sorry, it seems I posted twice the same issue.

[HenryHengZJ]

closing this as duplicated here - https://github.com/FlowiseAI/Flowise/issues/2774