search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.81k stars 1.76k forks source link

/recording_segments returns empty arrays of segments (it was working before) #2497

Closed 2snEM6 closed 11 months ago

2snEM6 commented 12 months ago

Self-Hosted Version

23.9.1

CPU Architecture

x86_64

Docker Version

23.0.6

Docker Compose Version

2.17.3

Steps to Reproduce

  1. A replay is created via a web app with a working Sentry Replays integration
  2. The replay appears in the replay list
  3. The replay UI screen is loaded, but the replay is empty.

Expected Result

The Replay should load and play properly

Actual Result

When trying to access a specific replay, it is not loading. See:

image

When inspecting the network tab, the API call responsible for fetching recording segments (https://{our-domain}/api/0/projects/{team}/{project}/replays/3fbf89b004e042f6aa7e78a6125796f1/recording-segments/?cursor=0%3A0%3A0&download=true&per_page=100)returns the following:

[
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    [],
    []
]

Logs for sentry-web container return the following when the above request is executed

/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=36, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 58914), raddr=('54.231.160.48', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=43, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 42914), raddr=('52.92.33.80', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=38, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 39684), raddr=('54.231.200.128', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=45, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 40826), raddr=('52.218.100.187', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=31, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 34068), raddr=('54.231.230.208', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=46, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 36248), raddr=('52.218.97.154', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=39, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 34082), raddr=('54.231.230.208', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=43, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 54378), raddr=('52.218.44.16', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=41, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 39694), raddr=('54.231.200.128', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=38, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 45468), raddr=('52.218.37.26', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=40, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 52168), raddr=('52.216.177.253', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=48, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 42466), raddr=('52.92.33.80', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=42, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 39720), raddr=('54.231.200.128', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=46, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 35414), raddr=('52.218.100.187', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=37, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 39706), raddr=('54.231.200.128', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=47, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 49932), raddr=('52.218.106.187', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=44, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 46230), raddr=('16.182.98.72', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=43, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 43838), raddr=('52.218.45.144', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=35, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 43034), raddr=('52.217.196.224', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=41, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 43824), raddr=('52.218.45.144', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=36, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 43044), raddr=('52.217.196.224', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=37, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 42468), raddr=('52.92.33.80', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=31, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 40114), raddr=('54.231.200.128', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=35, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 35416), raddr=('52.218.100.187', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:53 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=45, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 33250), raddr=('52.217.196.224', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=36, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 42482), raddr=('52.92.33.80', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:54 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=38, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 46982), raddr=('16.182.98.72', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=31, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 49918), raddr=('52.218.97.154', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:54 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=39, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 39520), raddr=('54.231.230.208', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=35, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 49942), raddr=('52.218.106.187', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:54 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=40, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 40122), raddr=('54.231.200.128', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=36, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 59824), raddr=('52.92.21.0', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
15:19:54 [WARNING] root: Storage GET error.
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=42, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 34388), raddr=('54.231.133.16', 443)>
  return f(*args, **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/site-packages/sentry/utils/metrics.py:199: ResourceWarning: unclosed <ssl.SSLSocket fd=37, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.18.0.16', 59830), raddr=('52.92.21.0', 443)>
  return f(*args, **kwargs)

Event ID

No response

2snEM6 commented 12 months ago

So, I was just manually inspecting a Replay that did not load on Postgres. The first time I performed this query:

select replay_id, file_id, project_id, sequence_id, size from replays_replayrecordingsegment where replay_id = '1c7f4a71afbf4824a1cb06b479dcd6e6';

It returned just one row.

The second time I did it (after about 15 minutes), the same query returned:

            replay_id             | file_id | project_id | sequence_id | size  
----------------------------------+---------+------------+-------------+-------
 1c7f4a71afbf4824a1cb06b479dcd6e6 |  598669 |          5 |           2 |  2932
 1c7f4a71afbf4824a1cb06b479dcd6e6 |  598605 |          5 |           0 | 44931
 1c7f4a71afbf4824a1cb06b479dcd6e6 |  598643 |          5 |           1 | 44762
 1c7f4a71afbf4824a1cb06b479dcd6e6 |  598720 |          5 |           3 |   527

Could this indicate that there is a significant amount of delay for processing replays?. If so is there anything I can do to fix this? Cpu load is about 60% and memory about 40% so I doubt hardware is the bottleneck?

EDIT: After some minutes, the same query now returns 27 rows!

hubertdeng123 commented 11 months ago

Interesting, have you checked the resource limits of docker for your machine? 🤔

getsantry[bot] commented 11 months ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀