Timeout uploading large files

[FTWinston]

Describe the bug When uploading a large media file to a new post, the upload repeatedly times out after 60s. As it's a big file (~50MB), I'd expect the upload to take longer than 60s, but the CMSs repeatedly aborts and retries after 60s. It looks like this is intentional, perhaps caused by #3649?

Console output below:

Pausing requests for 4 seconds due to fetch failures: Request timed out after 60 seconds
cms.js:118591 Done pausing requests
cms.js:118589 Pausing requests for 9 seconds due to fetch failures: Request timed out after 60 seconds
cms.js:118591 Done pausing requests
cms.js:118589 Pausing requests for 16 seconds due to fetch failures: Request timed out after 60 seconds
react_devtools_backend.js:2450 leave called too many times.
cms.js:118591 Done pausing requests
cms.js:118589 Pausing requests for 25 seconds due to fetch failures: Request timed out after 60 seconds
react_devtools_backend.js:2450 leave called too many times.
cms.js:118591 Done pausing requests
react_devtools_backend.js:2450 API_ERROR: Request timed out after 60 seconds
    at new t (cms.js:53893:25)
    at i._createSuperInternal (cms.js:796:283)
    at new i (cms.js:53886:27)
    at E.handleRequestError (cms.js:54745:17)
    at E._callee69$ (cms.js:54804:61)
    at tryCatch (cms.js:223192:40)
    at Generator.invoke [as _invoke] (cms.js:223421:22)
    at Generator.prototype.<computed> [as throw] (cms.js:223244:21)
    at asyncGeneratorStep (cms.js:223885:24)
    at _throw (cms.js:223911:9)

cms.js:223888 Uncaught (in promise) {type: "ENTRY_PERSIST_FAILURE", error: "Failed to persist entry", payload: {…}}

To Reproduce

1. Go to the CMS, add a new content entry, of a type with a "file" field.
2. Click "choose a file", then "upload" and pick a large file.
3. Save the entry, wait a few minutes.
4. See error.

Expected behavior Saving should eventually succeed.

Applicable Versions:

• netlify-cms-app: 2.13.1
• netlify-cms-core: 2.33.1
• netlify-cms: 2.10.66
• Git provider: GitHub
• OS: Windows 10
• Browser version chrome 86

CMS configuration config.yml

Additional context If this timeout were configurable, or could be disabled when uploading files, that would seem to me to prevent this problem.

Alternatively, if this is simply a much larger file than I ought to be storing in the CMS, then by all means tell me that.

Thanks in advance!

[erezrokah]

Thanks @FTWinston, you are correct it is a good idea to have an increased default timeout for POST requests and also make it configurable.

[FTWinston]

Either of those sound good to me. Both sound great!

On Sun, 8 Nov 2020, 11:20 Erez Rokah, notifications@github.com wrote:

Thanks @FTWinston https://github.com/FTWinston, maybe it would be a good idea to have an increased default timeout for POST requests and also make it configurable.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netlify/netlify-cms/issues/4538#issuecomment-723563209, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOG32MGOEJNTRRUEBHHDD3SOZ5INANCNFSM4TNVBKPQ .

[KancerEzeroglu]

I'm going to work on this issue 🎉

[KancerEzeroglu]

Hi,

I changed the timeout to 120 in the unsentRequest.js file.

https://github.com/netlify/netlify-cms/blob/dd080347b09606d651bb154a9a4870d704f0d48e/packages/netlify-cms-lib-util/src/unsentRequest.js#L13

Then I start getting the error below:

It's coming directly from GitHub and I couldn't find the root cause. How should we proceed?

[TristanSchorn]

@KancerEzeroglu I found this which seems to imply this is an HTTP post buffer size limitation. Is it possible to reduce the buffer Netlify CMS uses?

[KancerEzeroglu]

Hi,

I played with timeouts on our side but I still got errors from Github API, for me, it seems there are some limitations on the Github site. I couldn't find any option to set timeout and/or buffer size in the APIs.

As an improvement, we can show a more user-friendly error instead of logging an error to the console.

@erezrokah wdyt?

[erezrokah]

I missed the last comment on this issue.

If someone would like to pick this up, it would be great to share the error received from GitHub when we increase the timeout, and possibly report it to GitHub's support to see if that's expected.

As for ⬇️

@KancerEzeroglu I found this which seems to imply this is an HTTP post buffer size limitation. Is it possible to reduce the buffer Netlify CMS uses?

The CMS doesn't use git directly, so that setting is not relevant for the CMS

[modi98]

Has there been any update on this issue? I'm also getting this error on Gitlab, I've already tried updating the CMS to the latest version and adding Large Media to see if that helps but still the request times out before completing.

[erezrokah]

Hi @modi98, nothing had changed since https://github.com/netlify/netlify-cms/issues/4538#issuecomment-752636207 though we've been mostly focused on GitHub.

If someone would like to take a look at fixing this for GitLab by increasing the timeout, happy to accept a contribution

[tulsileathers]

I'm also experiencing this timeout with gitlab

[tulsileathers]

I could take a look at fixing it if someone pointed me in the right direction. Haven't looked at the netlify-cms code base before

[arm1n]

If I understand the Github docs regarding large files correctly, file uploads (via browsers) are limited to 25MB only: https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github#about-size-limits-on-github