Too large of total headers size hangs Shiny for 45s and requests fails. ~8KB

[hedrickbt]

Hi,

When we send a request to shiny server and the HTTP headers get to about 8KB, the shiny server will "hang" and not connect. After 45 seconds we get sent a FIN packet.

If I add 1 additional character to either my-header-1 or my-header-2, the shiny server will hang.

GET /dashboard/ HTTP/1.1 my-header-1: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam sed dolor ut neque semper fermentum. Aliquam erat volutpat. Nulla molestie euismod magna, tincidunt tempor mauris tristique semper. Integer vulputate, ligula ac pulvinar fermentum, massa massa luctus leo, et sagittis augue risus nec quam. Vestibulum commodo tellus pulvinar metus ultricies, a viverra sem vehicula. Duis ex nisl, semper eu hendrerit quis, molestie quis est. Nam commodo ut lacus et dignissim. Praesent et nulla metus. Pellentesque id turpis ullamcorper, pulvinar ante ut, malesuada nisi. Morbi molestie eget nunc eu ultricies.Phasellus elit eros, ullamcorper id ex sed, pulvinar auctor ante. Cras sit amet nunc erat. Nullam a magna et massa ultricies volutpat ac nec eros. Suspendisse potenti. Integer in magna sodales, euismod mauris consectetur, vulputate massa. Mauris euismod eros sapien, vel suscipit erat elementum at. Vestibulum scelerisque, ante ut fringilla feugiat, orci orci convallis velit, sit amet feugiat ante neque quis sapien. Ut massa risus, sollicitudin non eleifend eu, tempor id ligula.Nulla iaculis massa eu dictum dapibus. Aenean et blandit odio, in consectetur felis. Quisque eget urna quis justo molestie consectetur a sit amet urna. Cras ornare porta neque sit amet hendrerit. Lorem ipsum dolor sit amet, consectetur adipiscing elit. In nisi quam, interdum ut consequat vel, bibendum vitae velit. Sed commodo libero in luctus porttitor. Nulla nibh orci, dictum vulputate velit sodales, bibendum congue nulla. Integer eget feugiat lectus. Praesent aliquam mauris ut magna semper egestas. Sed nec ultrices tortor. Vestibulum dignissim ut metus in commodo.Nam turpis erat, consectetur nec tempor a, ultrices vel dui. Cras et tortor tellus. Curabitur enim diam, semper et pellentesque nec, placerat imperdiet metus. Sed faucibus odio risus, et aliquam justo placerat quis. Praesent nec lacus lorem. Aenean elementum lacus arcu, nec posuere lectus malesuada in. Sed laoreet cursus leo, et congue eros blandit in. Vivamus bibendum, magna iaculis hendrerit lobortis, erat augue pretium purus, vehicula sagittis neque massa sit amet nunc. Quisque quis elit vel libero semper elementum nec a enim. In iaculis semper libero in laoreet. Vivamus egestas quis tortor imperdiet tristique. Sed quis sagittis nunc, at viverra felis. Nam blandit, leo eu porttitor placerat, erat velit aliquam diam, non fringilla eros mi non erat. Nunc ac diam eu lectus pulvinar sodales at vitae nunc.Integer dignissim nibh sit amet pharetra ullamcorper. Phasellus non nisl velit. Praesent quis dignissim ligula. Sed non volutpat ligula. Proin urna arcu, ultricies eget sodales at, pharetra aliquam elit. Proin sagittis euismod mi eget scelerisque. Sed id vestibulum massa. Etiam malesuada sodales ex, at fermentum libero pretium quis. Proin quis rhoncus turpis. Fusce nec nibh mattis, malesuada ante pharetra, rutrum lectus. Integer eu tortor purus. Vestibulum pretium nulla molestie, luctus eros ut, accumsan turpis. Fusce quis diam hendrerit, congue metus at, rutrum neque. Cras faucibus id odio eget rhoncus. Sed eu pellentesque ante. Curabitur nisl quam, aliquam nec pharetra vitae, venenatis vitae sapien.Duis vitae consequat magna. Quisque et vestibulum est. Mauris et erat mauris. Sed eget molestie elit. Nulla facilisi. Suspendisse accumsan ullamcorper pharetra. Pellentesque quis elit sit amet nisl luctus volutpat non nec nunc. Nulla vel leo ut erat posuere placerat. Vivamus eget risus euismod, elementum quam non, fringilla urna. Donec eu lacus non mauris faucibus efficitur. Duis cursus ante lorem, a cursus nibh consequat vel.Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Sed feugiat scelerisque ullamcorper. Aenean convallis nisl id neque convallis tempor. Aenean vulputate turpis et nulla venenatis, eu vestibulum erat dignissim. Sed faucibus, odio sit amet fringilla ornare, neque velit eleifend urna, quis ultricies nibh ligula ac lorem. Nullam non mi quis nulla viverra pretium eu eget massa. Sed id tincidunt ipsum. Nam vehicula metus quis enim malesuada, nec venenatis lorem pretium. Donec nec mauris et ex facilisis commodo sit amet sed turpis. Morbi a felis in odio pulvinar sollicitudin.Fusce ut finibus magna. In malesuada facilisis luctus. Pellentesque velit dolor, faucibus quis semper eget, tincidunt vitae lectus. Donec pharetra mollis quam, quis vestibulum orci blandit a. Nulla gravida dictum odio, at ornare metus sodales id. Vestibulum sit amet scelerisque est. Proin sollicitudin orci ac tellus fringilla, nec ultricies tellus pharetra. Mauris at sapien sit amet turpis varius eleifend. Donec vel sem faucibus, eleifend risus nec, faucibus ante. Nulla convallis non massa maximus placerat. Aenean dignissim tincidunt orci, tincidunt tempor turpis fermentum et. Proin maximus auctor nunc, non efficitur eros cursus et. Aliquam ut dignissim orci. Vestibulum molestie lorem lacus, sit amet posuere lacus vulputate vel.Aliquam vel libero quam. Nullam dui urna, vehicula a vestibulum suscipit, sodales sed metus. Sed gravida, arcu ut congue tincidunt, magna felis sagittis nunc, ut semper lorem metus et augue. Suspendisse malesuada, magna non ornare sagittis, sapien libero dignissim sapien, a cursus est nibh non eros. Maecenas vehicula massa et nibh euismod tristique in porttitor lacus. Maecenas eget rhoncus felis. Nullam ullamcorper purus massa, quis feugiat sapien fermentum eu. Sed eget turpis vitae magna vehicula molestie at nec turpis. In et nunc non tortor aliquam tincidunt a quis sem. Aenean gravida tincidunt euismod. Phasellus quis fringilla tortor, eu ornare risus. Quisque non sodales ipsum. Quisque auctor nibh at ex blandit, sed feugiat lorem cursus.Donec posuere aliquam leo, eu tincidunt turpis porttitor porttitor. Donec at condimentum mauris. Maecenas enim elit, facilisis eget sodales non, commodo nec mauris. Vivamus vehicula lorem quis diam pharetra, sit amet iaculis erat egestas. Quisque suscipit fringilla mollis. Aenean ac lacinia arcu. Vivamus at pellentesque arcu. Nulla nec massa varius, tempus urna in, elementum diam. Nulla feugiat ex sed purus hendrerit, eu tincidunt ex tristique. Sed consequat pharetra lobortis. Nullam nec nulla non ante placerat ultrices. Duis non nunc ullamcorper, convallis tellus a, placerat dui. Morbi eros nibh, ullamcorper ut diam in, vestibulum placerat arcu. Pellentesque auctor orci sed sem bibendum fringilla. Sed vitae lobortis lectus.Ut commodo id lectus id dictum. Nulla finibus, velit sit amet molestie mollis, orci mauris rutrum lorem, a semper sapien nibh vel ante. Maecenas venenatis eros nec rhoncus ullamcorper. Cras a fermentum enim. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. In sollicitudin nibh eros, non pretium nunc malesuada ut. Praesent venenatis turpis lectus, vitae luctus est pretium in. Donec posuere dolor sed dolor molestie, quis varius odio rutrum. Nam elementum ipsum eu nibh posuere volutpat. Vivamus feugiat id aenean.Generated 11 paragraphs, 1043 words, 7000 bytes of Lorem Ipsum my-header-2: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam sed dolor ut neque semper fermentum. Aliquam erat volutpat. Nulla molestie euismod magna, tincidunt tempor mauris tristique semper. Integer vulputate, ligula ac pulvinar fermentum, massa massa luctus leo, et sagittis augue risus nec quam. Vestibulum commodo tellus pulvinar metus ultricies, a viverra sem vehicula. Duis ex nisl, semper eu hendrerit quis, molestie quis est. Nam commodo ut lacus et dignissim. Praesent et nulla metus. Pellentesque id turpis ullamcorper, pulvinar ante ut, malesuada nisi. Morbi molestie eget nunc eu ultricies.Phasellus elit eros, ullamcorper id ex sed, pulvinar auctor ante. Cras sit amet nunc 01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678 User-Agent: PostmanRuntime/7.24.1 Accept: / Cache-Control: no-cache Postman-Token: a236f54b-6ab7-4125-abbb-3c3fec3b4606 Host: devintshiny02.somedoma.com:3838 Accept-Encoding: gzip, deflate, br Connection: keep-alive

[jcheng5]

Thanks for the report! It looks like the Node.js http parser has a default max header size of 8K. Do you need large header sizes to work, or were you just expecting a faster failure rather than hang?

[jcheng5]

Update 5/26/2020: Don't do this, use NODE_OPTIONS as described in the next comment instead

---

As a workaround, you can raise the limit if you use the following command in place of /opt/shiny-server/bin/shiny-server,

/opt/shiny-server/ext/node/bin/shiny-server --max-http-header-size=16384 /opt/shiny-server/lib/main.js

For example, if your system uses systemd, then in /etc/systemd/system/shiny-server.service you can replace the ExecStart line with:

ExecStart=/usr/bin/env bash -c 'exec /opt/shiny-server/ext/node/bin/shiny-server --max-http-header-size=16384 /opt/shiny-server/lib/main.js >> /var/log/shiny-server.log 2>&1'

If you happen to be using Shiny Server Pro the incantation would be slightly different:

ExecStart=/usr/bin/env bash -c 'exec /opt/shiny-server/ext/node/bin/shiny-server --max-http-header-size=16384 /opt/shiny-server/main.min.js >> /var/log/shiny-server.log 2>&1'

[hedrickbt]

Thanks @jcheng5 ,

You got me pointed in the right direction! I am using Ubuntu 18.04. Shiny Server v1.5.13.944 Node.js v12.15.0

Here's what my systemd /etc/systemd/system/shiny-server.service file looked like.

Description=ShinyServer

[Service]
Type=simple
ExecStart=/usr/bin/env bash -c 'exec /opt/shiny-server/bin/shiny-server >> /var/log/shiny-server.log 2>&1'
KillMode=process
ExecReload=/usr/bin/env kill -HUP $MAINPID
ExecStopPost=/usr/bin/env sleep 5
Restart=on-failure
RestartSec=1
Environment="LANG=en_US.UTF-8"
StartLimitInterval=45
StartLimitBurst=3

[Install]
WantedBy=multi-user.target

I tried adding the --max-http-header-size=16384 as follows: ExecStart=/usr/bin/env bash -c 'exec /opt/shiny-server/bin/shiny-server --max-http-header-size=16384 >> /var/log/shiny-server.log 2>&1'

This did NOT work. I am wondering if the Shiny server executable isn't passing along the arguments where it needs to.

But, this lead me to Googling and finding: https://www.electronjs.org/docs/all which mentions a NODE_OPTIONS environment variable.

I removed the --max-http-header-size=16384 from ExecStart and added a new entry to the systemd unit: Environment="NODE_OPTIONS=--max-http-header-size=16384"

This worked as expected. I no longer saw the hang! The server responded immediately without issue.

My final systemd unit:

Description=ShinyServer

[Service]
Type=simple
ExecStart=/usr/bin/env bash -c 'exec /opt/shiny-server/bin/shiny-server >> /var/log/shiny-server.log 2>&1'
KillMode=process
Environment="NODE_OPTIONS=--max-http-header-size=16384"
ExecReload=/usr/bin/env kill -HUP $MAINPID
ExecStopPost=/usr/bin/env sleep 5
Restart=on-failure
RestartSec=1
Environment="LANG=en_US.UTF-8"
StartLimitInterval=45
StartLimitBurst=3

[Install]
WantedBy=multi-user.target

[jcheng5]

The reason the command line arg didn't work is because you were pointing to a different shiny-server path than I indicated (the correct path also ends with shiny-server but has ext/node/bin in it).

But NODE_OPTIONS is an even better solution, I'm glad you figured that out!

[hedrickbt]

Just FYI - the path that was being pointed to from /etc/systemd/system/shiny-server.service is how Ubuntu installed the package/service for the Shiny server. Not sure if that means the package maintainer needs to make a change to how the path is being specified in the service or not.

[hedrickbt]

Closing