Error in uploading files which takes time

[mahsashadi]

I am uploading a file in Os.js which take some times. Sometimes it causes an error as below:

It is considerable that for the same file, the first time it caused the error as above, but when I retried it succeeded (although it took time again) @andersevenrud @ajmeese7

[andersevenrud]

There's not enough information here for me the help. I have some thoughts:

1. Is this on localhost or on the internet ? If it is the internet then check the connection -- it might be dropping out and causing aborts
2. How large is the file ? Does it exceed the default limits ? https://github.com/os-js/osjs-server/blob/master/src/config.js#L43
3. Does this happen on multiple devices ?

[mahsashadi]

1-both on localhost and internet 2- No I set the limit to 100 mb, but my file was 50 mb, I think the time it takes was the issue.

module.exports = {
  root,
  port: 8000,
  public: path.resolve(root, 'dist'),
  express:{
    maxBodySize:'1000000kb',
    maxFileSize:mb(100)
  }
}

3- yes, on different deployments and devices we have that.

[mahsashadi]

Should I set something like timeout? (Although In this case it must always fails for time-taking uploads!) I read express req timeout deafult is 2 minutes.

[andersevenrud]

If there is a default timeout, then try to change it.

Your stack trace says socketOnClose which means it is an error, and not something that happens within the OS.js server code. The only thing used is formidable.

[mahsashadi]

No i did not set any timeout for server. Where should I set that?

And is there any possibility to have restriction on osjs client for time or file-size which can abort the request?

[andersevenrud]

Where should I set that?

You could try osjs.httpServer.setTimeout() before you run boot() in the bootstrap file.

https://nodejs.org/api/http.html#serversettimeoutmsecs-callback

And is there any possibility to have restriction on osjs client for time or file-size which can abort the request?

Only size, but that will not abort a request like this.

---

The only other thing I think of is that if you are for example writing to a custom filesystem, then there might be some other restrictions etc outide OS.js.

[maryam4s26]

Hello @andersevenrud , I don't have any problems in my localhost, but when I intentionally slow down the internet speed in localhost, I get the following problem.

You could try osjs.httpServer.setTimeout() before you run boot() in the bootstrap file.

Even after your comment, I have added the following line in src/server/index.js, but my problem is still not solved.

[andersevenrud]

Does this always fail consistently at ~5minutes like the screenshot says ?

Even after your comment, I have added the following line in src/server/index.js, but my problem is still not solved.

Yeah, I didn't really think this would solve anything because it's not a timeout. It's some kind of abort 🤔

[mahsashadi]

Does this always fail consistently at ~5minutes like the screenshot says ?

For me, for a sample file which was previously failed with the same error, it successfully uploaded even after it took time more than 5 min ( around 6 min ), so it does not have consistency.

[andersevenrud]

Sadly, I'm not home so I cannot try to reproduce this problem myself... so all I can do right now is just ask some fairly standard questions :(

Have you tried running in incognito mode or a completely clean browser (no extensions) ?

Any messages in the developer console ?

[andersevenrud]

Also, what is the complete error message in the dialog? (this should be in the console)

[maryam4s26]

Also, what is the complete error message in the dialog? (this should be in the console)

[maryam4s26]

Have you tried running in incognito mode or a completely clean browser (no extensions) ?

Yes, there is a problem in this case as well

[andersevenrud]

Ah. This is the first time I see the actual HTTP code... this makes things a little bit clearer.

Your connections is directly to the node server or some kind of reverse-proxy ?

[maryam4s26]

Your connections is directly to the node server or some kind of reverse-proxy ?

directly to the node server.

[mahsashadi]

We can see many inconsistencies 1- having successfull and also failed uploading in an internet-based deployment 2- having failed uploading in localhost tests

The issue must be something related to the client and also deployment

Your connections is directly to the node server or some kind of reverse-proxy ?

What happened if we have reverse proxy?

[andersevenrud]

The issue must be something related to the client and also deployment

Technically a HTTP 408 code means that the client connection failed to complete the request in some way. In some cases could be caused by an invalid/corrupt Content-Length header or simply that file transfer stopped inside the browser.

I will try to reproduce this when I get home.

What happened if we have reverse proxy?

I was just thinking if you had a reverse proxy then there could be some problems with the proxy configs.

[mahsashadi]

I was just thinking if you had a reverse proxy then there could be some problems with the proxy configs.

We have also another deployment that uses reverse proxy, if you have any advise.

[mahsashadi]

Also we have another service (different client than osjs client), which uses osjs server as its server. In that we also faced time out error after 30 minutes

I think the sever log should be something like bad Gateway error

[andersevenrud]

bad Gateway error

Sounds like this comes from a reverse proxy then. It means it lost communication with the upstream (node) server.

I can't see the error code here, but I assume 502 and not 408.

[andersevenrud]

I will try to reproduce this when I get home.

I've currently tried uploading two different files to a completely fresh install and simulated the speed so that they both completed in ~15 minutes, and so far no errors.

I need to get to bed really soon, but I will try to reproduce this more tomorrow.

[maryam4s26]

@andersevenrud Is it possible to add the keepAliveTimeout to the express parameter in the server configuration?

[andersevenrud]

It is not possible to set that in the configuration. But, you can do it via bootstrap:

osjs.httpServer.keepAliveTimeout = N;

By the way, I am fairly sure that all timeout settings are 0 by default in nodejs, which means "forever".

---

https://nodejs.org/api/http.html#serverkeepalivetimeout

[mahsashadi]

BTW, according to the default express configs, I think there is a contradiction:

  express: {
    maxFieldsSize: mb(20),
    maxFileSize: mb(200),
    maxBodySize: '100kb'
  }

Since maxBodySize encompasses the entire request body, including both file uploads and form fields, I think it should typically be equal to or larger than maxFileSize to accommodate the largest possible file uploads allowed by maxFileSize.

[andersevenrud]

The maxBodySize is for regular requests that contain JSON etc. The maxFieldsSize and maxFileSize is for multipart, which are used for uploads.

So this is correct.