search

mitodl / odl-video-service

building blocks for a basic video service for ODL
BSD 3-Clause "New" or "Revised" License
4 stars 1 forks source link

502 error at https://video.odl.mit.edu/ #325

Closed pdpinch closed 6 years ago

pdpinch commented 6 years ago

It appears that if you have a stale Touchstone session and access https://video.odl.mit.edu/ you will get a 502 Bad Gateway error. Once in this state, you can get out of it by entering the URL https://video.odl.mit.edu/collections

This is not reproducible in a Private Window.

pdpinch commented 6 years ago

Sar and I found a UWSGI error occurs when I load the page, and nginx complains about request size. Removing my odl.mit.edu cookies (in particularly the JWT cookie for open-discussions) seems to clear the issue.

shaidar commented 6 years ago

The error reported in /var/log/odl_video/uwsgi.log is: invalid request block size: 4298 (max 4096)...skip

shaidar commented 6 years ago

Added buffer-size=32768 to the uwsgi config in /etc/uwsgi-emperor/vassals/odl-video.ini on RC and need to test.

noisecapella commented 6 years ago

There are some nginx settings we needed to tweak when dealing with a similar issue: https://github.com/mitodl/micromasters/issues/3453#issuecomment-339132284

I don't think there's any reason why the JWT should be so large. It might be worth investigating which is the exact cookie that is large, since cookies over 4k are not supported

pdpinch commented 6 years ago

I don't think this is the same issue, but this URL will consistently give me a 502 Bad Gateway error: https://video.odl.mit.edu/videos/a03d8b4405ca47519e0e/ (after logging in)

Ferdi commented 6 years ago

@pdpinch same error for me

shaidar commented 6 years ago

I was able to reproduce the error (loading the page - https://video.odl.mit.edu/videos/a03d8b4405ca47519e0e/) in Chrome Incognito and tailed the uwsgi logs. The issue is that the video uuid is incorrect. Talked to @mbertrand about it and possibly having it return a 404 error instead.

pdpinch commented 6 years ago

In that case, #350 should fix this?

I'll move it into the ready column and add it to the sprint.

mbertrand commented 6 years ago

I can start working on #350 now if that's ok with you @pdpinch

pdpinch commented 6 years ago

Working with @blarghmatey, it looks like these manifest as "invalid request block size" errors in the uswgi log on the prod server.

We found we could reproduce the error by custom crafting a very large cookie.

Tobias increased the uwsgi buffer size and the error cleared. I can no longer reproduce the error.

@Ferdi let us know what you find.

pdpinch commented 6 years ago

Working for Ferdi now too, so it appears this is fixed.