Sentry stopped accepting transaction data

[ingria]

Self-Hosted Version

24.3.0.dev0

CPU Architecture

x86_x64

Docker Version

24.0.4

Docker Compose Version

24.0.4

Steps to Reproduce

Update to the latest master

Expected Result

Everything works fine

Actual Result

Performance page shows zeros for the time period since the update and until now:

Project page shows the correct info about transactions and errors:

Stats page shows 49k transactions of which 49k are dropped:

Same for errors:

Event ID

No response

UPD

there are a lot of errors in clickhouse container:

2024.03.10 23:40:34.789282 [ 46 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
2. DB::ReadBufferFromPocoSocket::ReadBufferFromPocoSocket(Poco::Net::Socket&, unsigned long) @ 0x101540cd in /usr/bin/clickhouse
3. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6fd5 in /usr/bin/clickhouse
4. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
5. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
6. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
7. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
9. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
10. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
 (version 21.8.13.1.altinitystable (altinity build))

[hheexx]

I have this problem again after updating to 24.8.0!

[hheexx]

---except that reverting to python snuba does not work. No more errors but still does not work.

[khassad]

I have this problem again after updating to 24.8.0!

Same behavior here, transactions showed up partially (some data in stats area) but not in global projects pages or individual project page 😢

[DarkByteZero]

I had issues with stopped ingestion, but my issue was that I didn't have COMPOSE_PROFILES=feature-complete in my custom env

[hubertdeng123]

I had issues with stopped ingestion, but my issue was that I didn't have COMPOSE_PROFILES=feature-complete in my custom env

Ah yeah, that'll do it. Without that you'll only be ingesting errors.

[liukch]

This issue has been present for several months and remains unresolved. Do we have a schedule to fix this issue? Due to this issue, we are experiencing significant difficulties with the upgrade of our self-hosted Sentry version. @hubertdeng123

[aldy505]

---except that reverting to python snuba does not work. No more errors but still does not work.

@hheexx I helped someone on Discord a few days ago, both regular snuba consumer and snuba rust-consumer didn't work for him. He tried upgrading their server instance to a higher spec (previously 4 cores CPU + 16 GB RAM [AWS EC2 m6a.xlarge] --> 8 cores CPU + 32 GB RAM [AWS EC2 m6a.2xlarge]). See the Discord thread here: https://discord.com/channels/621778831602221064/1286099840480182272

Obviously I know bumping their server spec is not for everyone, even my initial hunch was on the IO/s (or IOps) limit.

[hheexx]

thanks @aldy505, you are right. I fixed it by moving msl to seperate ssd storage (vm is on hdd)

[klemen-df]

Same issues, nothing helps :(

[Mordreak]

Same issue here, on fresh install and latest commit on master, I do get a 200 response with ID of the transaction, but nothing shows in the performance tab panel. Tried several proposed solutions without any luck

[ethrgeist]

For my installation it looks like metrics are collected and displayed fine, still logs are flooded with:

clickhouse-1                                    | 2024.11.12 08:16:23.789391 [ 47 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
clickhouse-1                                    | 
clickhouse-1                                    | 0. Poco::Net::SocketImpl::error(int, String const&) @ 0x0000000015b3dbf2 in /usr/bin/clickhouse
clickhouse-1                                    | 1. Poco::Net::SocketImpl::peerAddress() @ 0x0000000015b40376 in /usr/bin/clickhouse
clickhouse-1                                    | 2. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptr<DB::IHTTPContext>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x0000000013154417 in /usr/bin/clickhouse
clickhouse-1                                    | 3. DB::HTTPServerConnection::run() @ 0x0000000013152ba4 in /usr/bin/clickhouse
clickhouse-1                                    | 4. Poco::Net::TCPServerConnection::start() @ 0x0000000015b42834 in /usr/bin/clickhouse
clickhouse-1                                    | 5. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b43a31 in /usr/bin/clickhouse
clickhouse-1                                    | 6. Poco::PooledThread::run() @ 0x0000000015c7a667 in /usr/bin/clickhouse
clickhouse-1                                    | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7893c in /usr/bin/clickhouse
clickhouse-1                                    | 8. ? @ 0x00007fa3e4e25609 in ?
clickhouse-1                                    | 9. ? @ 0x00007fa3e4d4a353 in ?
clickhouse-1                                    |  (version 23.8.11.29.altinitystable (altinity build))

It's a 2 week old install, HEAD detached at 24.9.0

[aldy505]

@ethrgeist see https://github.com/getsentry/snuba/issues/5707