CMS: Preview fails with "Error While Sending Preview Data"

[pdpinch]

It appears that when a particularly large draft page is being previewed in the CMS, it will fail with this error in the browser: "Error While Sending Preview Data"

Looking at the logs, this seems to be a buffer size issue in nginx:

2017-08-04T16:16:15.678263+00:00 app[web.2]: 2017/08/04 16:16:15 [error] 161#0: *3749 upstream sent too big header while reading response header from upstream, client: 10.152.233.4, server: _, request: "POST /cms/pages/19/edit/preview/ HTTP/1.1", upstream: "uwsgi://unix:/tmp/nginx.socket:", host: "micromasters.mit.edu", referrer: "https://micromasters.mit.edu/cms/pages/19/edit/"

In Kibana: https://logs.odl.mit.edu/app/kibana#/doc/logstash-*/logstash-heroku-2017.08.04/fluentd?id=AV2uDeaNEZmaosbJB9it&_g=()

See also ZenDesk #9908

[pdpinch]

I'm still getting 502 errors with the following text on the the CI server when I try to preview the page at /cms/pages/16/edit

About MIT Center for Transportation & Logistics

For more than four decades, the MIT Center for Transportation & Logistics (MIT CTL) has been a world leader in supply chain management education and research. MIT CTL has made significant contributions to supply chain logistics and has helped numerous companies gain competitive advantage from its cutting-edge research. 

About MIT Center for Transportation & Logistics

For more than four decades, the MIT Center for Transportation & Logistics (MIT CTL) has been a world leader in supply chain management education and research. MIT CTL has made significant contributions to supply chain logistics and has helped numerous companies gain competitive advantage from its cutting-edge research. 

Instructor Team
Chris Caplice

Director, MITx MicroMasters Program in Supply Chain Management, Silver Family Research Fellow, Executive Director, MIT Center for Transportation & Logistics (CTL)

Dr. Caplice has been teaching logistics and supply chain management at MIT for over a decade. He is also responsible for the planning and management of the research, education, and corporate outreach programs for the MIT Center for Transportation & Logistics as well as MIT’s Global SCALE Network. He is also the Chief Scientist for Chainalytics, a leading analytical supply chain consulting firm. He received a Ph.D. from MIT in 1996 in Transportation and Logistics Systems, a Master of Science in Civil Engineering from the University of Texas at Austin, and a Bachelor of Science in Civil Engineering from the Virginia Military Institute (VMI).
Eva Ponce

Executive Director, MITx MicroMasters Program in Supply Chain Management, Lecturer and Research Associate 

Eva Ponce, Ph.D., is a Research Associate at the MIT Center for Transportation & Logistics. Her current research focus is the design of urban distribution models (freight deliveries) with a special focus on omni-channel distribution strategies. She also leads research initiatives on Reverse Logistics and Closed-Loop Supply Chains. Dr. Ponce has over fifteen years of experience in teaching and research in supply chain management and quantitative models for industrial engineering. In 2008, she received her tenure as an Associate Professor in Supply Chain Management and Logistics at the School of Industrial Engineering of the Universidad Politécnica de Madrid (UPM).  
Yossi Sheffi

Elisha Gray II Professor of Engineering Systems, MIT, Professor, MIT Civil and Environmental Engineering, Director, MIT Center for Transportation & Logistics

Dr. Sheffi is a former director of MIT's Engineering Systems Division and holds a dual appointment at the Department of Civil and Environmental Engineering and at the Engineering Systems Division. He is an expert in systems optimization, risk analysis, and supply chain management, which are the subjects he teaches and researches at MIT. He is the author of dozens of scientific publications and three books. Outside the university, Professor Sheffi is an active entrepreneur, having founded five successful companies, and a sought-after speaker in corporate and professional events. He obtained his B.Sc. from the Technion in Israel in 1975, his S.M. from MIT in 1977, and Ph.D. from MIT in 1978.
James Rice

Deputy Director, MIT Center for Transportation & Logistics, Director, Integrated Supply Chain Management Program, Director, Supply Chain Exchange Program

Jim Rice joined the MIT Center for Transportation and Logistics (CTL) in 1995 and he was appointed as the Deputy Director of the Center in 2007. In this capacity, he oversees several research and outreach programs, CTL Executive Education Programs and outreach marketing activities. As part of these roles, Jim works with industry sponsors to develop research projects and educational and knowledge sharing events for the sponsors. He is also a Lecturer in the Supply Chain Management program teaching Supply Chain Finance. Prior to joining MIT, Jim managed manufacturing and distribution operations at Procter & Gamble and served as a sales and market manager at General Electric Company. Jim earned his MBA in Operations and Finance from the Harvard Business School and a Bachelors Degree in Mechanical Engineering from the University of Notre Dame.
Jarrod Goentzel

Director, MIT Humanitarian Response Lab, Lecturer and Research Associate

Dr. Goentzel is founder and director of the MIT Humanitarian Response Lab in the MIT Center for Transportation & Logistics. His research focuses on meeting human needs in resource-constrained settings through better supply chain management, information systems, and decision support technology. Dr. Goentzel has created graduate-level courses in humanitarian logistics, international operations, and supply chain finance, and has extensive experience using simulation games to build intuition and leadership skills. Dr. Goentzel received a Ph.D. from the School of Industrial and Systems Engineering at the Georgia Institute of Technology, an M.S. in applied mathematics from Colorado State University, and a B.A. in mathematics from Tabor College with studies at the Technical University of Budapest (Hungary).

If I cut out the last paragraph, it will preview OK.

FYI, @shaidar

[noisecapella]

I did some investigating and it looks like we need these options to increase the uwsgi buffer size, in addition to the change to uwsgi.ini:

uwsgi_buffers 8 32k;
uwsgi_busy_buffers_size 32k;
uwsgi_buffer_size 32k;

These options should be located at the http level. In the dev nginx.conf file we should also move the other buffer configuration options there too since they don't apply to the correct server.

[annagav]

Wagtail is storing the preview page in the session, and since we switched the setting for session middleware to store the session in a browser cookie instead of the database (PR #1861), the cookie length is exceeded. Cookie length should be less than or equal to 4096 characters.

[pdpinch]

Was there further discussion of this issue that wasn't recorded here?

@annagav @blarghmatey what happens if we go back to using the database for storing sessions?

[blarghmatey]

Then MM starts running a lot slower again. The reason for switching away from DB sessions was because it was crippling our performance during the load tests. We might be able to get away with a cached_db session, but we'll want to be sure to load test before and after to see what kind of impact it has.

[pdpinch]

Is there any way to limit the change just to the wagtail app?

On Nov 7, 2017, at 4:47 PM, Tobias Macey notifications@github.com wrote:

Then MM starts running a lot slower again. The reason for switching away from DB sessions was because it was crippling our performance during the load tests. We might be able to get away with a cached_db session, but we'll want to be sure to load test before and after to see what kind of impact it has.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[noisecapella]

@roberthouse54 ran into this as well. The cookie caused him to see an nginx 502 page when visiting micromasters.mit.edu. When he deleted the cookies the problem went away

[pdpinch]

@blarghmatey suggests that we make a change upstream to Wagtail. I am pretty sure that drafts are already saved to the database. It's not clear to me why the draft content would need to be saved to the session when previewing. Why can't the preview mode just pull the appropriate draft?

[amir-qayyum-khan]

@pdpinch Neither we can have multiple session_engin nor we can restrict wagtail to specific engin. Wagtail uses django session for preview.

• @mitodl/devops we can go for 'django.contrib.sessions.backends.cached_db'
• Need these config, which I think is already done.
• Also need to perform load testing.

[pdpinch]

I'd like to open an issue upstream on wagtail for getting the preview data from the database instead of the session.

[amir-qayyum-khan]

@pdpinch https://github.com/wagtail/wagtail/issues/4521

[tomdyson]

Checking in from the Wagtail project - thanks for the detailed issue report. We'd be happy to work with you to get this fixed.

[amir-qayyum-khan]

thanks @tomdyson

[amir-qayyum-khan]

@tomdyson what is you suggestion to fix this issue?

[pdpinch]

The change in #4049 still needs to be load tested before we can turn it on