The number of transactions is zero in all projects after upgraded

[trongquan1297]

Self-Hosted Version

24.4.0

CPU Architecture

x64_86

Docker Version

20.10.21

Docker Compose Version

v2.12.2

Steps to Reproduce

1. Upgrading from 21.11.0 -> 23.6.2 -> 24.4.0
2. The number of transactions is zero in all projects

Expected Result

The number of transactions is equal to the number of transactions in the Performance tab.

Actual Result

n/a

Event ID

No response

[hubertdeng123]

You have upgraded a substantial amount of versions and we have completely reworked the performance product. Are new transactions that come in still not accounted for? Or are these old transactions that you are looking to preserve?

[trongquan1297]

You have upgraded a substantial amount of versions and we have completely reworked the performance product. Are new transactions that come in still not accounted for? Or are these old transactions that you are looking to preserve?

Thank you for your reply. New transactions still come but are not counted in the project tab. I want to show like old version

[SCjona]

I also have this issue. Project overview shows zero transactions. Performance tab and weekly email reports have transaction counts. This started after upgrading from 23.12.1 to 24.4.1. Both old (pre-update) transactions and new transactions don't show in the overview.

[azaslavsky]

Going to move this over to the performance team, since they may have better insight into what changed.

[getsantry[bot]]

Assigning to @getsentry/support for routing ⏲️

[getsantry[bot]]

Routing to @getsentry/product-owners-performance for triage ⏲️

[k-fish]

Hey there @trongquan1297 can you take a screenshot of your Discover tab searching for that project with a filter of project:<project_name> event.type:transaction and a count() column (see screenshot below).

[trongquan1297]

project: event.type:transaction

Yes. transaction still come

[gggritso]

@trongquan1297 thanks! We're taking a look, and will try to get back to you early next week when the folks who were helping you are back from being away

[ivacom]

Hello, we encountered the same problem. Help is needed.

[gggritso]

@ivacom are you using a self-hosted instance? If yes, what version of self-hosted are you running, and what was your previous version before the upgrade?

[gggritso]

@trongquan1297 are you still experiencing this problem?

[SCjona]

this is still happening for me with Sentry 24.6.0. edit: just updated to 24.7.0, issue persists

[ivacom]

yes, I use self-hosted, updated from version 23.3.1 > 23.6.2 > 24.3.0 > 24.6.0. It seems the problem appeared after updating to version 23.6.2. <your.sentry.version> -> 23.6.2 -> latest https://develop.sentry.dev/self-hosted/releases/

[gggritso]

@ivacom @SCjona thanks for clarifying! I need a little more info from you to help debug this, bear with me. Here's what I need:

1. Go to your Sentry's "Projects" page
2. Open the browser's Developer Tools
3. Refresh
4. In Developer Tools, in the "Network" panel find an outgoing HTTP GET request to a URL that looks like this: https://<your-sentry-url>/api/0/organizations/<your-organization>/projects/?dataset=<something>&query=id:<some-project-id> id:<some-project-id> &sessionStats=1...more URL parameters
5. Paste the URL here. You can omit your organization and project IDs from the request.

[ivacom]

@gggritso

https://sentry/api/0/organizations/sentry/projects/?dataset=metricsEnhanced&query=id%3A15%20id%3A23%20id%3A22%20id%3A27&sessionStats=1&statsPeriod=24h&transactionStats=1

[gggritso]

@ivacom thanks! That part looks correct. Next, can you check the contents of the response? It should be an array that looks like this:

[
  {
    id: 15,
    name: ...
    team: ...etc
    transactionStats

Can you paste the contents of transactionStats?

[ivacom]

@gggritso , Yes, "transactionStats": [ [1721336400, 21191], [1721340000, 19083], [1721343600, 19162], [1721347200, 24648], [1721350800, 35906], [1721354400, 55665], [1721358000, 83597], [1721361600, 108304], [1721365200, 133168], [1721368800, 166180], [1721372400, 196108], [1721376000, 194613], [1721379600, 197770], [1721383200, 191335], [1721386800, 180598], [1721390400, 174520], [1721394000, 169829], [1721397600, 159714], [1721401200, 151055], [1721404800, 134266], [1721408400, 117220], [1721412000, 97107], [1721415600, 74891], [1721419200, 9442] ],

[SCjona]

@gggritso

https://sentry.hostname.tld/api/0/organizations/<org-name>/projects/?dataset=metricsEnhanced&query=id%3A7%20id%3A5%20id%3A4%20id%3A1%20id%3A6%20id%3A3%20id%3A2&sessionStats=1&statsPeriod=24h&transactionStats=1

Response transactionStats:

```json { "transactionStats": [ [ 1721340000, 0 ], [ 1721343600, 0 ], [ 1721347200, 0 ], [ 1721350800, 0 ], [ 1721354400, 0 ], [ 1721358000, 0 ], [ 1721361600, 0 ], [ 1721365200, 0 ], [ 1721368800, 0 ], [ 1721372400, 0 ], [ 1721376000, 0 ], [ 1721379600, 0 ], [ 1721383200, 0 ], [ 1721386800, 0 ], [ 1721390400, 0 ], [ 1721394000, 0 ], [ 1721397600, 0 ], [ 1721401200, 0 ], [ 1721404800, 0 ], [ 1721408400, 0 ], [ 1721412000, 0 ], [ 1721415600, 0 ], [ 1721419200, 0 ], [ 1721422800, 0 ] ] } ```

[gggritso]

Got it! I'll continue looking at this on Monday. Thank you!

[ivacom]

Hi! @gggritso Let's continue?

[gggritso]

@ivacom I need a bit of time to come up with some hypotheses on what's happening, hang in there 👍 It also looks like you and @SCjona are experiencing related but different issues.

[ivacom]

thanks @gggritso , yes, I most likely have a problem with displaying @SCjona no transactions.

@gggritso

https://sentry.hostname.tld/api/0/organizations/<org-name>/projects/?dataset=metricsEnhanced&query=id%3A7%20id%3A5%20id%3A4%20id%3A1%20id%3A6%20id%3A3%20id%3A2&sessionStats=1&statsPeriod=24h&transactionStats=1

[ 1721340000, 0 ], [ 1721343600, 0 ],

@SCjona This part of the request contains the project code query=id%3A7%20id%3A5%20id%3A4%20id%3A1%20id%3A6%20id%3A3%20id%3A2

Perhaps in transactionStats there are project data where there are no transactions [1721408400, 117220], [1721412000, 97107], [1721415600, 74891],

[SCjona]

no we have transactions, this is just broken in the project overview/details, works fine in the performance tab

GET https://sentry.hostname.tld/api/0/organizations/<org>/events-meta/?dataset=metricsEnhanced&environment=master&project=3&query=event.type:transaction&statsPeriod=14d returns count 0 (triggered in project overview)

GET https://sentry.hostname.tld/api/0/organizations/<org>/events-stats/?environment=master&interval=1h&partial=1&project=3&query=event.type:transaction&referrer=api.performance.generic-widget-chart.apdex-area&statsPeriod=28d&yAxis=apdex()&yAxis=tpm()&yAxis=failure_rate() returns correct? data (triggered in performance view)

Interestingly GET https://sentry.hostname.tld/api/0/organizations/<org>/events-meta/?environment=master&project=3&query=event.type:transaction&statsPeriod=14d&utc=true returns count > 0 (triggered in discover)

If i remove the utc=true parameter i still get count > 0, but if I add dataset=metricsEnhanced I get count = 0.

[gggritso]

@SCjona this is interesting!

Could you check whether these features are enabled in your Sentry config?

• 'organizations:metrics-extraction'
• 'organizations:transaction-metrics-extraction'
• 'organizations:mep-rollout-flag'

If those features are not enabled, please turn them on. This will turn on metrics extraction from transactions, which will start collecting transaction metrics and start populating the charts. You should start getting data from that point.

Let me know if that works! In the meantime, I will talk to the team about doing a frontend fallback (which the Performance tab does automatically).

@ivacom do you mind checking those features on your instance, too?

[ivacom]

yes, included

[SCjona]

@gggritso Those features were missing from my sentry.conf.py. I hadn't updated that one in a while, just did cp sentry.conf.example.py sentry.conf.py. I now get transactions in the project overview as new data is coming in. Thank you!

[ivacom]

@gggritso Hi! I updated from the very beginning 23.3.1 => 23.6.2 everything worked and transactions were displayed. I updated to version 23.12.1 and everything broke. Maybe it's worth adding a version between 23.6.2 and 23.12.1?

[gggritso]

@SCjona glad to hear it! Sounds like the issue is resolve for you, yes?

@ivacom if can try a version between 23.6.2 and 23.12.1 that would help narrow down the issue! There are 4-5 releases in that range you could try. If you can specify which exact release first breaks the feature, that would help a lot

[ivacom]

@gggritso updated 23.6.2-> 23.10.1 there is a problem. updated 23.6.2-> 23.8.0 no problem. It remains to check 23.6.2-> 23.9.1.

[ivacom]

@gggritso Hi! Updated 23.6.2 -> 23.8.0 -> 23.9.1 no problem. Updated 23.9.1->23.10.1 there is a problem. It turns out that after 23.9.1 it will not be possible to update.

[gggritso]

@ivacom thanks for narrowing it down! I'm going to see if I can figure out what broke between those versions

[gggritso]

@ivacom I took a look at the diff between those versions, but didn't spot anything obvious. Unfortunately, I'm running out of ways to help you! Without access to your environment, there's not much else I can do. One of the downsides of self-hosted over SAAS.

At this point I need more information. If you're able to take a lead on debugging, that would help move this forward. Sentry's front end code has source maps, so you should be able to step through the browser's debugger and check a few points:

• the render method of ProjectCard is in charge of calculating the total transaction count. If you can add a breakpoint there, it should be possible to see if transactionStats has any non-zero data points in it at any point
• the _debouncedLoadStats method is in charge of fetching the data from the endpoint. If you add a breakpoint there, you should be able to confirm/deny whether it's able to successfully fetch the results for your projects

Also overall, watch out for any errors in the Network pane (maybe CORS or CSRF errors, 500s, or anything similar) or in the browser console (errors parsing responses, errors with undefined values, or anything similar)

[ivacom]

@gggritso hi! In general, we partially figured out the problem, but we found a problem point. The problem point is Relay. We have Relay in static mode, only redis and kafka are connected. On sentry version 23.3.1 everything works fine, starting with version 23.9.1 relay in static mode does not transfer data for display in projects or the main stack does not process this data. If relay is in managed mode with upstream: web:9000 specified, everything works and is displayed in projects.

Relay is in a closed network and it cannot be connected to upstream: web:9000, so relay is in static mode with kafka and redis connection. Kafka and redis use the same

please help.

[jjbayer]

@ivacom could you please check whether you have the feature flag for transaction metrics extraction enabled, as it is in the linked example?

https://github.com/getsentry/self-hosted/blob/3d120727d5118e6d9ec6932f88756de4d3cbaaba/sentry/sentry.conf.example.py#L288

[ivacom]

yes, it is enabled as you can see from the message above https://github.com/getsentry/sentry/issues/70473#issuecomment-2244319673

as I said above, the problem is in static mode, in managed mode there is no problem with displaying the number of transactions in projects

[jjbayer]

@ivacom sorry, I had not read the entire conversation. What you're describing does sound like a Relay bug: static mode should forward all data unless configured differently. Would you be able to share your local .relay/config.yml (feel free to redact any fields you don't want to share) ?

[ivacom]

Sure! there is also a projects directory with a json file created according to the documentation https://docs.sentry.io/product/relay/projects/ and for clarification, the data in Issues and Discover are displayed correctly in this operating mode, only the total number of transactions in Projects is not displayed.

[jjbayer]

@ivacom thanks! mode: static and processing.enabled: true are actually mutually exclusive (think of managed+processing as a fourth relay mode). We added an explicit validation step for proxy mode, but apparently forgot about static mode: https://github.com/getsentry/relay/pull/3146.

Relay is in a closed network and it cannot be connected to upstream: web:9000, so relay is in static mode with kafka and redis connection.

Would it be possible for you to switch to Managed mode? If Relay can connect to Kafka and Redis, why not allow it to the upstream as well?

[ivacom]

Unfortunately, this is not possible, Kafka and redis are located in the same place as relay. Sentry can connect to kafka and redis, but reverse connection from a closed network from Relay to Sentry is not possible for security reasons. What config should I use for testing?

Let me remind you that on version 23.3.1 everything works without problems. We want to update to the latest working version, but after version 23.8.0 this is not possible due to display problems.

[jjbayer]

Let me remind you that on version 23.3.1 everything works without problems.

I'm surprised this even worked. What you could try to restore previous behavior is adding {"config": {"transactionMetrics": {"version": 6}}} to your project config(s) in .relay/projects/*.json.

I will discuss with the team whether we want to support processing in static mode in the future.

[ivacom]

Added your config, the result is the same, zero transactions in Projects

Here is the full config

***The current version of Sentry and Relay is 24.8.0

[ivacom]

Hi! Is there any solution?

And by the way, transactions in projects do not work in proxy mode either.

We really need your help, we cannot update.

[jjbayer]

@ivacom your use case is tricky because it's a setup we do not recommend. Essentially you need to create a project config to resemble closely a a project config that would get sent from sentry. I'm looking at this code in particular: https://github.com/getsentry/relay/blob/a1713f11ff1840023e81b15145166eada08c3b14/relay-server/src/services/processor.rs#L1376-L1384

You might also need to add a {"metricExtraction": {"version": 4}} to your config. Do the relay logs contain anything that might point you to the problem?

[ivacom]

@jjbayer added to the config, it did not give a result, previously in the projects there are zero transactions. there are no errors in the log.

maybe something needs to be added to the relay source code so that it works in our complex configuration?

[SuperTommy007]

Every time I run it for a while (about 1 or 2 days) the transactions in the projects page goes to 0, then I use docker-compose down and docker-compose up -d and the transactions value goes back to normal, then after a while it goes back to 0 again.

[SuperTommy007]

Every time I run it for a while (about 1 or 2 days) the transactions in the projects page goes to 0, then I use docker-compose down and docker-compose up -d and the transactions value goes back to normal, then after a while it goes back to 0 again.

[jjbayer]

@ivacom I discussed the matter with the team, we are going to explicitly forbid combining static mode + processing enabled because we cannot support it and it will become even harder to maintain in the future. May I ask why the HTTP connection in your case is a security concern, but the redis & kafka connections are not?

[jjbayer]

@SuperTommy007 what you are describing sounds like a dynamic sampling issue, is the transaction count actually zero or just a very low number?

[ivacom]

@jjbayer Hi! where we have kafka and relay, this is a closed cloud server from which there is no access to the main sentry server, but there is access from the main sentry server to the cloud server. The restrictions are determined by the information and financial security service, and we cannot bypass them. we accept more than 1M transactions per hour, we really need the function of static mode + enabled processing, help us build a relay version at least up to 24.4.x with the required function, and then we will try to solve the issue with access from the cloud server. We really need to update