Closed shellac85 closed 3 years ago
Thank you for raising this issue! We will try and get back to you as soon as possible. Please make sure you format it properly, followed our code of conduct and have given us as much context as possible. /cc @tunnckoCore @GrosSacASac
Sorry about that. Stack traces doesn't help me much. :disappointed: Seems like some memory problem, which I think we had in the past, in v1 specifically.
Someone noticed an issue when they tried to upload a 65mb file but I am noticing that when I go over 30mb I get errors in the console of VS code.
Is it failing randomly only some times or it's constantly?
Also, could you please try the v2 canary. I'm seeing it's on a production system, but it worth trying, there's no big fundamental changes behind the scenes. The v1 has a ton of problems and there were similar issues posted in the past too.
Are they some specific files, specific format?
Thanks for the quick reply. Ye stack traces aren't great at all!
It's consistent when its comes to files over 30mb. I zipped up one file that was over 30mb and that brought it down to 27mb and it worked fine then.
I have a test environment, so I can try out the v2 canary and let you know how I get on. I have been testing pdfs so far.
when its comes to files over 30mb. I zipped up one file that was over 30mb and that brought it down to 27mb and it worked fine then.
Good catch it's around the 30mb mark, haha. Strange. Sounds like some system limit? Probably the firewalls, the network configurations? :thinking: Does testing is on the same, as close as possible, env as the prod?
so I can try out the v2 canary and let you know how I get on.
Okay, great. :)
Same story with v2 canary unfortunately.
Sounds like some system limit? Probably the firewalls, the network configurations?
This was my initial thought but I couldn't find any setting relating to this. Will have to look more into that area I think!
I just tried uploading 10 GB of files and it works fine. Can you reproduce the issue using node native http module ? If not, it means the issue is inside foalTS.
I dont think its a issue with Formidable. In our code there is a section that encrypts the code and I think its running out of memory with large files:
export const encryptFile: (
fsPath: string,
encryptionKey: string,
) => Promise<void> = async (fsPath, encryptionKey) => {
if (!encryptionKey) throw new Error('Customer encryption key not found');
const fileData = await readFile(fsPath);
const dataBase64 = fileData.toString('base64');
const encryptedFile = CryptoJS.AES.encrypt(dataBase64, encryptionKey);
const buffer = Buffer.from(encryptedFile.toString(), 'base64');
writeFileSync(fsPath, buffer);
};
I was trying to achieve the same with streams but no joy yet.
Use streams to encrypt files without memory overflow. Closing as it is not an issue in Formidable
Support plan
Context
What are you trying to achieve or the steps to reproduce?
Hi, Just trying to upload a file. Someone noticed an issue when they tried to upload a 65mb file but I am noticing that when I go over 30mb I get errors in the console of VS code. Please see the stack trace from the error. I was on version 1.2.1 but its not working on 1.2.2 either. If you need anymore info, I will be happy to provide it. Thank you :)
What was the result you got?
What result did you expect?
File uploaded I was hoping for. :)