Closed trongquan1297 closed 1 week ago
You have upgraded a substantial amount of versions and we have completely reworked the performance product. Are new transactions that come in still not accounted for? Or are these old transactions that you are looking to preserve?
You have upgraded a substantial amount of versions and we have completely reworked the performance product. Are new transactions that come in still not accounted for? Or are these old transactions that you are looking to preserve?
Thank you for your reply. New transactions still come but are not counted in the project tab. I want to show like old version
I also have this issue. Project overview shows zero transactions. Performance tab and weekly email reports have transaction counts. This started after upgrading from 23.12.1
to 24.4.1
. Both old (pre-update) transactions and new transactions don't show in the overview.
Going to move this over to the performance team, since they may have better insight into what changed.
Assigning to @getsentry/support for routing ⏲️
Routing to @getsentry/product-owners-performance for triage ⏲️
Hey there @trongquan1297 can you take a screenshot of your Discover tab searching for that project with a filter of project:<project_name> event.type:transaction
and a count()
column (see screenshot below).
project:
event.type:transaction
Yes. transaction still come
@trongquan1297 thanks! We're taking a look, and will try to get back to you early next week when the folks who were helping you are back from being away
Hello, we encountered the same problem. Help is needed.
@ivacom are you using a self-hosted instance? If yes, what version of self-hosted are you running, and what was your previous version before the upgrade?
@trongquan1297 are you still experiencing this problem?
this is still happening for me with Sentry 24.6.0. edit: just updated to 24.7.0, issue persists
yes, I use self-hosted, updated from version 23.3.1 > 23.6.2 > 24.3.0 > 24.6.0.
It seems the problem appeared after updating to version 23.6.2.
<your.sentry.version> -> 23.6.2 -> latest
https://develop.sentry.dev/self-hosted/releases/
@ivacom @SCjona thanks for clarifying! I need a little more info from you to help debug this, bear with me. Here's what I need:
https://<your-sentry-url>/api/0/organizations/<your-organization>/projects/?dataset=<something>&query=id:<some-project-id> id:<some-project-id> &sessionStats=1...more URL parameters
@ivacom thanks! That part looks correct. Next, can you check the contents of the response? It should be an array that looks like this:
[
{
id: 15,
name: ...
team: ...etc
transactionStats
Can you paste the contents of transactionStats
?
@gggritso , Yes, "transactionStats": [ [1721336400, 21191], [1721340000, 19083], [1721343600, 19162], [1721347200, 24648], [1721350800, 35906], [1721354400, 55665], [1721358000, 83597], [1721361600, 108304], [1721365200, 133168], [1721368800, 166180], [1721372400, 196108], [1721376000, 194613], [1721379600, 197770], [1721383200, 191335], [1721386800, 180598], [1721390400, 174520], [1721394000, 169829], [1721397600, 159714], [1721401200, 151055], [1721404800, 134266], [1721408400, 117220], [1721412000, 97107], [1721415600, 74891], [1721419200, 9442] ],
@gggritso
https://sentry.hostname.tld/api/0/organizations/<org-name>/projects/?dataset=metricsEnhanced&query=id%3A7%20id%3A5%20id%3A4%20id%3A1%20id%3A6%20id%3A3%20id%3A2&sessionStats=1&statsPeriod=24h&transactionStats=1
Response transactionStats:
Got it! I'll continue looking at this on Monday. Thank you!
Hi! @gggritso Let's continue?
@ivacom I need a bit of time to come up with some hypotheses on what's happening, hang in there 👍 It also looks like you and @SCjona are experiencing related but different issues.
thanks @gggritso , yes, I most likely have a problem with displaying @SCjona no transactions.
https://sentry.hostname.tld/api/0/organizations/<org-name>/projects/?dataset=metricsEnhanced&query=id%3A7%20id%3A5%20id%3A4%20id%3A1%20id%3A6%20id%3A3%20id%3A2&sessionStats=1&statsPeriod=24h&transactionStats=1
[ 1721340000, 0 ], [ 1721343600, 0 ],
@SCjona This part of the request contains the project code query=id%3A7%20id%3A5%20id%3A4%20id%3A1%20id%3A6%20id%3A3%20id%3A2
Perhaps in transactionStats there are project data where there are no transactions [1721408400, 117220], [1721412000, 97107], [1721415600, 74891],
no we have transactions, this is just broken in the project overview/details, works fine in the performance tab
GET https://sentry.hostname.tld/api/0/organizations/<org>/events-meta/?dataset=metricsEnhanced&environment=master&project=3&query=event.type:transaction&statsPeriod=14d
returns count 0 (triggered in project overview)
GET https://sentry.hostname.tld/api/0/organizations/<org>/events-stats/?environment=master&interval=1h&partial=1&project=3&query=event.type:transaction&referrer=api.performance.generic-widget-chart.apdex-area&statsPeriod=28d&yAxis=apdex()&yAxis=tpm()&yAxis=failure_rate()
returns correct? data (triggered in performance view)
Interestingly GET https://sentry.hostname.tld/api/0/organizations/<org>/events-meta/?environment=master&project=3&query=event.type:transaction&statsPeriod=14d&utc=true
returns count > 0 (triggered in discover)
If i remove the utc=true
parameter i still get count > 0, but if I add dataset=metricsEnhanced
I get count = 0.
@SCjona this is interesting!
Could you check whether these features are enabled in your Sentry config?
'organizations:metrics-extraction'
'organizations:transaction-metrics-extraction'
'organizations:mep-rollout-flag'
If those features are not enabled, please turn them on. This will turn on metrics extraction from transactions, which will start collecting transaction metrics and start populating the charts. You should start getting data from that point.
Let me know if that works! In the meantime, I will talk to the team about doing a frontend fallback (which the Performance tab does automatically).
@ivacom do you mind checking those features on your instance, too?
yes, included
@gggritso Those features were missing from my sentry.conf.py
. I hadn't updated that one in a while, just did cp sentry.conf.example.py sentry.conf.py
. I now get transactions in the project overview as new data is coming in. Thank you!
@gggritso Hi! I updated from the very beginning 23.3.1 => 23.6.2 everything worked and transactions were displayed. I updated to version 23.12.1 and everything broke. Maybe it's worth adding a version between 23.6.2 and 23.12.1?
@SCjona glad to hear it! Sounds like the issue is resolve for you, yes?
@ivacom if can try a version between 23.6.2 and 23.12.1 that would help narrow down the issue! There are 4-5 releases in that range you could try. If you can specify which exact release first breaks the feature, that would help a lot
@gggritso updated 23.6.2-> 23.10.1 there is a problem. updated 23.6.2-> 23.8.0 no problem. It remains to check 23.6.2-> 23.9.1.
@gggritso Hi! Updated 23.6.2 -> 23.8.0 -> 23.9.1 no problem. Updated 23.9.1->23.10.1 there is a problem. It turns out that after 23.9.1 it will not be possible to update.
@ivacom thanks for narrowing it down! I'm going to see if I can figure out what broke between those versions
@ivacom I took a look at the diff between those versions, but didn't spot anything obvious. Unfortunately, I'm running out of ways to help you! Without access to your environment, there's not much else I can do. One of the downsides of self-hosted over SAAS.
At this point I need more information. If you're able to take a lead on debugging, that would help move this forward. Sentry's front end code has source maps, so you should be able to step through the browser's debugger and check a few points:
render
method of ProjectCard
is in charge of calculating the total transaction count. If you can add a breakpoint there, it should be possible to see if transactionStats
has any non-zero data points in it at any point_debouncedLoadStats
method is in charge of fetching the data from the endpoint. If you add a breakpoint there, you should be able to confirm/deny whether it's able to successfully fetch the results for your projectsAlso overall, watch out for any errors in the Network pane (maybe CORS or CSRF errors, 500s, or anything similar) or in the browser console (errors parsing responses, errors with undefined
values, or anything similar)
@gggritso hi! In general, we partially figured out the problem, but we found a problem point. The problem point is Relay. We have Relay in static mode, only redis and kafka are connected. On sentry version 23.3.1 everything works fine, starting with version 23.9.1 relay in static mode does not transfer data for display in projects or the main stack does not process this data. If relay is in managed mode with upstream: web:9000 specified, everything works and is displayed in projects.
Relay is in a closed network and it cannot be connected to upstream: web:9000, so relay is in static mode with kafka and redis connection. Kafka and redis use the same
please help.
@ivacom could you please check whether you have the feature flag for transaction metrics extraction enabled, as it is in the linked example?
yes, it is enabled as you can see from the message above https://github.com/getsentry/sentry/issues/70473#issuecomment-2244319673
as I said above, the problem is in static mode, in managed mode there is no problem with displaying the number of transactions in projects
@ivacom sorry, I had not read the entire conversation. What you're describing does sound like a Relay bug: static mode should forward all data unless configured differently. Would you be able to share your local .relay/config.yml
(feel free to redact any fields you don't want to share) ?
Sure! there is also a projects directory with a json file created according to the documentation https://docs.sentry.io/product/relay/projects/ and for clarification, the data in Issues and Discover are displayed correctly in this operating mode, only the total number of transactions in Projects is not displayed.
@ivacom thanks! mode: static
and processing.enabled: true
are actually mutually exclusive (think of managed+processing
as a fourth relay mode). We added an explicit validation step for proxy mode, but apparently forgot about static mode: https://github.com/getsentry/relay/pull/3146.
Relay is in a closed network and it cannot be connected to upstream: web:9000, so relay is in static mode with kafka and redis connection.
Would it be possible for you to switch to Managed mode? If Relay can connect to Kafka and Redis, why not allow it to the upstream as well?
Unfortunately, this is not possible, Kafka and redis are located in the same place as relay. Sentry can connect to kafka and redis, but reverse connection from a closed network from Relay to Sentry is not possible for security reasons. What config should I use for testing?
Let me remind you that on version 23.3.1 everything works without problems. We want to update to the latest working version, but after version 23.8.0 this is not possible due to display problems.
Let me remind you that on version 23.3.1 everything works without problems.
I'm surprised this even worked. What you could try to restore previous behavior is adding {"config": {"transactionMetrics": {"version": 6}}}
to your project config(s) in .relay/projects/*.json
.
I will discuss with the team whether we want to support processing in static mode in the future.
Added your config, the result is the same, zero transactions in Projects
Here is the full config
***The current version of Sentry and Relay is 24.8.0
Hi! Is there any solution?
And by the way, transactions in projects do not work in proxy mode either.
We really need your help, we cannot update.
@ivacom your use case is tricky because it's a setup we do not recommend. Essentially you need to create a project config to resemble closely a a project config that would get sent from sentry. I'm looking at this code in particular: https://github.com/getsentry/relay/blob/a1713f11ff1840023e81b15145166eada08c3b14/relay-server/src/services/processor.rs#L1376-L1384
You might also need to add a {"metricExtraction": {"version": 4}}
to your config. Do the relay logs contain anything that might point you to the problem?
@jjbayer added to the config, it did not give a result, previously in the projects there are zero transactions. there are no errors in the log.
maybe something needs to be added to the relay source code so that it works in our complex configuration?
Every time I run it for a while (about 1 or 2 days) the transactions in the projects page goes to 0, then I use docker-compose down and docker-compose up -d and the transactions value goes back to normal, then after a while it goes back to 0 again.
Every time I run it for a while (about 1 or 2 days) the transactions in the projects page goes to 0, then I use docker-compose down and docker-compose up -d and the transactions value goes back to normal, then after a while it goes back to 0 again.
@ivacom I discussed the matter with the team, we are going to explicitly forbid combining static mode + processing enabled because we cannot support it and it will become even harder to maintain in the future. May I ask why the HTTP connection in your case is a security concern, but the redis & kafka connections are not?
@SuperTommy007 what you are describing sounds like a dynamic sampling issue, is the transaction count actually zero or just a very low number?
@jjbayer Hi! where we have kafka and relay, this is a closed cloud server from which there is no access to the main sentry server, but there is access from the main sentry server to the cloud server. The restrictions are determined by the information and financial security service, and we cannot bypass them. we accept more than 1M transactions per hour, we really need the function of static mode + enabled processing, help us build a relay version at least up to 24.4.x with the required function, and then we will try to solve the issue with access from the cloud server. We really need to update
Self-Hosted Version
24.4.0
CPU Architecture
x64_86
Docker Version
20.10.21
Docker Compose Version
v2.12.2
Steps to Reproduce
Expected Result
The number of transactions is equal to the number of transactions in the Performance tab.
Actual Result
n/a
Event ID
No response