search

mollersuite / monofile

File sharing over the Discord CDN
https://fyle.uk
The Unlicense
18 stars 3 forks source link

Large Files fail to upload #46

Closed CrazyDev05 closed 1 month ago

CrazyDev05 commented 7 months ago

when i try to upload a large file (in my case 500MiB) i get the following error

HeadersTimeoutError: Headers Timeout Error
    at Timeout.onParserTimeout [as _onTimeout] (/var/www/monofile/node_modules/undici/lib/client.js:902:28)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7) {
  code: 'UND_ERR_HEADERS_TIMEOUT'
}
nbitzz commented 7 months ago

I don't believe I've seen this before, maybe raise your requestTimeout in config.json?

CrazyDev05 commented 7 months ago

Good point...

CrazyDev05 commented 7 months ago

how large should i make it?

nbitzz commented 7 months ago

Whatever you think would work. Also, what branch are you on? Are you on a non-master/stable branch? Are you cloning a file?

CrazyDev05 commented 7 months ago

I'm on the main branch/i cloned the git

CrazyDev05 commented 7 months ago

is the timeout in ms?

nbitzz commented 7 months ago

is the timeout in ms?

Yeah. I'm not sure how Discord.js uploads the files, but I do believe I got a different error for timeouts... could be different now, though

If boosting requestTimeout doesn't fix it, maybe decrease your Discord file size? That fixed some issues on the main instances

The only place I know undici is used in is axios, so it could also be an issue with the file you're trying to clone

CrazyDev05 commented 7 months ago

i already decreased maxDiscordFileSize to 10485760

CrazyDev05 commented 7 months ago

Wait I misunderstood the cloning file question, I'm trying to upload a file from my pc (the file just contains Hello \n to test the upload)

nbitzz commented 7 months ago

Wait I misunderstood the cloning file question, I'm trying to upload a file from my pc (the file just contains Hello \n to test the upload)

That's... really weird. I'll check DJS's codebase when I have time. How fast is your internet?

CrazyDev05 commented 7 months ago

The server has 1gbit/s (up/down), I have 30mbit/s up and i still have the same error after changing the requestTimeout to 10800000 (failed after 3.5 min)

nbitzz commented 7 months ago

The server has 1gbit/s (up/down), I have 30mbit/s up

and i still have the same error after changing the requestTimeout to 10800000 (failed after 3.5 min)

I'll be honest... I have zero idea of what's happening.

git stash, git checkout stable, and git stash apply is the best advice I can give here, but I doubt that'd work; master and stable aren't that different...

CrazyDev05 commented 7 months ago

Maybe some kind of discord limitation?

nbitzz commented 7 months ago

Maybe some kind of discord limitation?

I know @Jack5079 got it to work with 10MiB chunks, and I'm not sure what's limiting you. What provider are you using for your server?

I just tested an upload on the main instance, so I know it's still working...

CrazyDev05 commented 7 months ago

I'm using a dedicated server from hetzner Hardware:

CrazyDev05 commented 7 months ago

It should already be fixed https://github.com/discordjs/discord.js/pull/8679

nbitzz commented 7 months ago

Hetzner was apparently banned from Discord's API once, but I'm not sure if this is still true, and the fact that you're getting a timeout over an actual proper error is strange. Try running locally or with another provider and see if that works, maybe...?

CrazyDev05 commented 7 months ago

interestingly i can upload small files, but not large ones

nbitzz commented 7 months ago

interestingly i can upload small files, but not large ones

Try uploading a file in the 10MiB range, then a file in the 100MiB+ range, and we'll see how that plays out...?

CrazyDev05 commented 7 months ago
CrazyDev05 commented 7 months ago

Trying 500MB again

CrazyDev05 commented 7 months ago

now i get http error 504 and no error in the console

<html>
   <head>
      <title>504 Gateway Time-out</title>
   </head>
   <body>
      <center>
         <h1>504 Gateway Time-out</h1>
      </center>
      <hr>
      <center>nginx</center>
   </body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page -->
nbitzz commented 7 months ago

now i get http error 504 and no error in the console

<html>
   <head>
      <title>504 Gateway Time-out</title>
   </head>
   <body>
      <center>
         <h1>504 Gateway Time-out</h1>
      </center>
      <hr>
      <center>nginx</center>
   </body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page --><!-- a padding to disable MSIE and Chrome friendly error page -->

Your server's just taking a while to upload the file after it receives it; fix your timeout in NGINX or just refresh your files page if you have an account until it finishes uploading

FYI, your admin account is admin, with a password of admin. Make sure to reset it if you plan to make this instance public...

CrazyDev05 commented 7 months ago

i already changed the admin password and how long should i make the timeout?

nbitzz commented 7 months ago

i already changed the admin password and how long should i make the timeout?

...Longer than it is currently?

CrazyDev05 commented 7 months ago

Tried 700MB and error 500 (HeadersTimeoutError) is back...

nbitzz commented 7 months ago

Tried 700MB and error 500 (HeadersTimeoutError) is back...

Hmm.... how much RAM do you have? I may be stupid What's the maximum file size on your instance? I know this probably won't help much, but I'd like to know more. I have no idea what's causing this...

CrazyDev05 commented 7 months ago

it seems to be a discord issue

CrazyDev05 commented 7 months ago

Maybe progressively uploading instead of bulk uploading the files could help?

nbitzz commented 7 months ago

Maybe progressively uploading instead of bulk uploading the files could help?

As in, one file per message, or streaming uploads to Discord? Upload streaming is set for monofile 2.0 (#16). If you'd like, we can add a maxFilesPerMessage parameter to config.json which would allow you to upload only one chunk at a time, which will likely be available in 2.0.

CrazyDev05 commented 7 months ago

i meant like one file per message and maybe also upload streaming

nbitzz commented 1 month ago

Prooooooooobably fixed with upload streaming i'll close this now

nbitzz commented 1 month ago

I mean I managed a 10gb file on the api-v1 test instance and pfile is serving 250gb of data now so I think we're good