Self-hosted - not getting any events in Sentry

danielitit commented 3 months ago

Environment

What are you trying to accomplish?

Hey,

we stopped getting events to our sentry and only errors that I do see at this moment is from clickhouse:

2024.06.12 11:59:21.145129 [ 314 ] {0d052f13-5d6e-4b25-85fa-b6bc0c31d8c3} <Error> TCPHandler: Code: 60. DB::Exception: Table default.sessions_raw_local does not exist. Maybe you meant functions_raw_local?. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bff71c4 in /usr/bin/clickhouse
1. DB::Exception::Exception<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x0000000007636ba8 in /usr/bin/clickhouse
2. DB::IDatabase::getTable(String const&, std::shared_ptr<DB::Context const>) const @ 0x0000000010882f14 in /usr/bin/clickhouse
3. DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::shared_ptr<DB::Context const>, std::optional<DB::Exception>*) const @ 0x0000000010aa5b18 in /usr/bin/clickhouse
4. DB::DatabaseCatalog::getTable(DB::StorageID const&, std::shared_ptr<DB::Context const>) const @ 0x0000000010aad800 in /usr/bin/clickhouse
5. DB::JoinedTables::getLeftTableStorage() @ 0x000000001123da58 in /usr/bin/clickhouse
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001116ee84 in /usr/bin/clickhouse
7. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000112039a8 in /usr/bin/clickhouse
8. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000010fc22b8 in /usr/bin/clickhouse
9. DB::InterpreterCreateQuery::execute() @ 0x0000000010fcf7a0 in /usr/bin/clickhouse
10. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000114f3094 in /usr/bin/clickhouse
11. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000114ef988 in /usr/bin/clickhouse
12. DB::TCPHandler::runImpl() @ 0x00000000121e2bb4 in /usr/bin/clickhouse
13. DB::TCPHandler::run() @ 0x00000000121f2cc8 in /usr/bin/clickhouse
14. Poco::Net::TCPServerConnection::start() @ 0x00000000146fe0e4 in /usr/bin/clickhouse
15. Poco::Net::TCPServerDispatcher::run() @ 0x00000000146ff5e0 in /usr/bin/clickhouse
16. Poco::PooledThread::run() @ 0x0000000014870edc in /usr/bin/clickhouse
17. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001486f084 in /usr/bin/clickhouse
18. start_thread @ 0x0000000000007624 in /lib/libpthread.so.0
19. ? @ 0x00000000000d149c in /lib/libc.so.6

How are you getting stuck?

I was trying to run some snuba commands:

snuba bootstrap --no-migrate --force snuba bootstrap migrate --force snuba bootstrap --force --log-level debug

Each command is passing but still no events. I was even tried to send events using sentry-cli and I'm getting for example

sentry-cli send-event -m "Hello from Sentry"
Event dispatched: adde451f-7d36-4f00-929d-4c8e3876c6c6

Where in the product are you?

Other

Link

No response

DSN

No response

Version

No response

getsantry[bot] commented 3 months ago

Assigning to @getsentry/support for routing ⏲️

erlangparasu commented 3 months ago

This may fix: https://github.com/getsentry/self-hosted/issues/3016#issuecomment-2100500968

aldy505 commented 3 months ago

From your error, it seems you have not run the necessary migrations. A few questions:

What version are you currently running?
Is this a fresh install, or an upgrade? If this is an upgrade, what's the previous version?
How did you perform the upgrade? Did you run the ./install.sh script?

ManideepKarimireddy commented 3 months ago

It is a fresh install in Kubernetes with version24.4.2. we see the events getting dropped rather than being accepted

schmitch commented 3 months ago

It is a fresh install in Kubernetes with version24.4.2. we see the events getting dropped rather than being accepted

a little bit OT: how did you install it in k8s?

ManideepKarimireddy commented 3 months ago

It is a fresh install in Kubernetes with version24.4.2. we see the events getting dropped rather than being accepted

a little bit OT: how did you install it in k8s?

using helm charts

azaslavsky commented 3 months ago

Are you using sentry-kubernetes? If so, that is a third-party project unrelated to this repository, and is not a supported workflow.

getsantry[bot] commented 2 months ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

getsentry / self-hosted