search

sandstorm-io / sandstorm

Sandstorm is a self-hostable web productivity suite. It's implemented as a security-hardened web app package manager.
https://sandstorm.io
Other
6.73k stars 707 forks source link

Sandstorm stops responding #3324

Open polymersgreen opened 4 years ago

polymersgreen commented 4 years ago

Amazon AWS EC2 medium (2 cores, 4GB ram)

Every couple of hours the server stops responding. restarting it via ssh helps. (sudo sandstorm restart)

log: https://nopaste.nl/2OojNPmszB

ocdtrekkie commented 4 years ago

It'd be probably relevant to know what your config looks like: Are you using Sandcats? Do you have one of the new Let's Encrypt certs or a GlobalSign cert? Perhaps give us your sandstorm.conf file, substituting any private information with similar versions?

polymersgreen commented 4 years ago

Hello, This was a standard installation with a script, I am using sandcats dns. Will post a config soon.

-- Jakub Sadowski j.sadowski@me.com Carefully handcrafted on a mobile device. Please excuse my brevity.

On 6 May 2020, at 21:22, Jacob Weisz notifications@github.com wrote:

 It'd be probably relevant to know what your config looks like: Are you using Sandcats? Do you have one of the new Let's Encrypt certs or a GlobalSign cert? Perhaps give us your sandstorm.conf file, substituting any private information with similar versions?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

zenhack commented 4 years ago

Ug, those stack traces originate from inside malloc(); we've got a memory corruption bug somewhere. Yay.

For convenience, I'm attaching formatted versions of those stack traces, though I agree with @abliss's pessimism about their utility -- it looks like something is trampling on malloc()'s internal data structures somewhere, and presumably the problem has nothing to do with the particular call stack when it actually gets caught. Here's the sigabrt one:

trace-sigabrt.txt

And the one from openssl:

trace-openssl.txt

zenhack commented 4 years ago

@polymersgreen: Is this a new system that you've just set up? If not, when did you first notice the problem?

abliss commented 4 years ago

@zenhack ever tried running sandstorm under valgrind?

zenhack commented 4 years ago

Quoting Adam Bliss (2020-05-06 19:54:43)

@zenhack ever tried running sandstorm under valgrind?

Not yet. I'm trying to get it built with asan instrumentation, but am hitting issues building libsodium. Have put it down for the moment.

polymersgreen commented 4 years ago

The system is quite fresh, installed last week, but it only started freezing Tuesday.

zenhack commented 4 years ago

If you revert your system to 0.262 (and disable updates so it doesn't just switch back to current immediately), does the behavior change?

zenhack commented 4 years ago

@polymersgreen, I finally got around to putting together an asan instrumented build of sandstorm:

https://mirror.zenhack.net/pub/sandstorm-0.tar.xz

Could you install that and let us know how that goes?

You can do this by setting UPDATE_CHANNEL=none in sandstorm.conf and then running sandstorm update sandstorm-0.tar.xz.

If you'd prefer to build from source yourself, rather than install the binary I've provided, you can check out the asan-build branch on my fork and build per usual, though you'll also need to have capnproto installed system wide, in addition to the usual build dependencies.

zenhack commented 4 years ago

Ug, looks like that build pulled in some system libraries that aren't available everywhere; normally sandstorm is statically linked but I had to do dynamic linking to get asan to work. If the binary fails to start try building from source on the system that will be running sandstorm.