search

getsentry / sentry

Developer-first error tracking and performance monitoring
https://sentry.io
Other
38.62k stars 4.13k forks source link

Upgrade from 8.4 to 8.19: There was an error loading data. #5948

Closed mikeshield13 closed 7 years ago

mikeshield13 commented 7 years ago

kepernyokep errol 2017-08-22 21-32-08

These are the command I used:

rm -rf /www/
pip install --upgrade pip
pip install --upgrade virtualenv
virtualenv /www/sentry/
source /www/sentry/bin/activate
pip install sentry
sentry init /tmp/new-sentry-conf
mkdir /etc/sentry && mv /tmp/new-sentry-conf/* /etc/sentry/ 
pip install redis==2.10.5
SENTRY_CONF=/etc/sentry sentry upgrade
systemctl start sentry-web.service

Logs:

● sentry-cron.service - Sentry Beat Service Loaded: loaded (/etc/systemd/system/sentry-cron.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2017-08-22 20:30:21 BST; 58s ago Main PID: 3195 ([celery beat] r) CGroup: /system.slice/sentry-cron.service └─3195 [celery beat] run cron

Aug 22 20:30:21 sentrytest systemd[1]: Started Sentry Beat Service. Aug 22 20:30:21 sentrytest systemd[1]: Starting Sentry Beat Service...

● sentry-worker.service - Sentry Background Worker Loaded: loaded (/etc/systemd/system/sentry-worker.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2017-08-22 20:30:21 BST; 9min ago Main PID: 3193 ([celeryd: celer) CGroup: /system.slice/sentry-worker.service ├─3193 [celeryd: celery@sentrytest:MainProcess] -active- (run worker) ├─3232 [celeryd: celery@sentrytest:Worker-1] └─3233 [celeryd: celery@sentrytest:Worker-2]

Aug 22 20:30:26 sentrytest sentry[3193]: .> events.save_event exchange=default(direct) key=events.save_event Aug 22 20:30:26 sentrytest sentry[3193]: .> merge exchange=default(direct) key=merge Aug 22 20:30:26 sentrytest sentry[3193]: .> options exchange=default(direct) key=options Aug 22 20:30:26 sentrytest sentry[3193]: .> reports.deliver exchange=default(direct) key=reports.deliver Aug 22 20:30:26 sentrytest sentry[3193]: .> reports.prepare exchange=default(direct) key=reports.prepare Aug 22 20:30:26 sentrytest sentry[3193]: .> search exchange=default(direct) key=search Aug 22 20:30:26 sentrytest sentry[3193]: .> stats exchange=default(direct) key=stats Aug 22 20:30:26 sentrytest sentry[3193]: .> triggers-0 exchange=triggers(direct) key= Aug 22 20:30:26 sentrytest sentry[3193]: .> unmerge exchange=default(direct) key=unmerge Aug 22 20:30:26 sentrytest sentry[3193]: .> update exchange=default(direct) key=update

● sentry-web.service - Sentry Main Service Loaded: loaded (/etc/systemd/system/sentry-web.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2017-08-22 20:30:21 BST; 9min ago Main PID: 3194 (uwsgi) CGroup: /system.slice/sentry-web.service ├─3194 [Sentry] uWSGI master ├─3228 [Sentry] uWSGI worker 1 ├─3229 [Sentry] uWSGI worker 2 └─3230 [Sentry] uWSGI worker 3

Aug 22 20:30:55 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:30:55 +0000] "GET /api/0/projects/MS/msmain/events/379798/committers/ HTTP/1.0" 404 6...efox/55.0" Aug 22 20:31:51 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:31:51 +0000] "GET /api/0/broadcasts/ HTTP/1.0" 200 621 "http://192.168.1.100/MS/msma...efox/55.0" Aug 22 20:32:51 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:32:51 +0000] "GET /api/0/broadcasts/ HTTP/1.0" 200 621 "http://192.168.1.100/MS/msma...efox/55.0" Aug 22 20:33:51 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:33:51 +0000] "GET /api/0/broadcasts/ HTTP/1.0" 200 621 "http://192.168.1.100/MS/msma...efox/55.0" Aug 22 20:34:51 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:34:51 +0000] "GET /api/0/broadcasts/ HTTP/1.0" 200 621 "http://192.168.1.100/MS/msma...efox/55.0" Aug 22 20:35:51 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:35:51 +0000] "GET /api/0/broadcasts/ HTTP/1.0" 200 621 "http://192.168.1.100/MS/msma...efox/55.0" Aug 22 20:36:52 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:36:52 +0000] "GET /api/0/broadcasts/ HTTP/1.0" 200 621 "http://192.168.1.100/MS/msma...efox/55.0" Aug 22 20:37:52 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:37:52 +0000] "GET /api/0/broadcasts/ HTTP/1.0" 200 621 "http://192.168.1.100/MS/msma...efox/55.0" Aug 22 20:38:52 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:38:52 +0000] "GET /api/0/broadcasts/ HTTP/1.0" 200 621 "http://192.168.1.100/MS/msma...efox/55.0" Aug 22 20:39:52 sentrytest sentry[3194]: 127.0.0.1 - - [22/Aug/2017:19:39:52 +0000] "GET /api/0/broadcasts/ HTTP/1.0" 200 621 "http://192.168.1.100/MS/msma...efox/55.0" Hint: Some lines were ellipsized, use -l to show in full.

mattrobenolt commented 7 years ago

That graph is driven by redis. So it looks like you don't have that going. But if it's erroring, you should see errors from this within Sentry.

mikeshield13 commented 7 years ago

Maybe this is relevant:

Aug 22 20:49:48 sentrytest sentry[978]: 127.0.0.1 - - [22/Aug/2017:19:49:48 +0000] "GET /api/0/issues/4556/environments/none/?until=1503428401 HTTP/1.0" 404 633 "http://sentrytest.mikeshield.com/MS/msmain/issues/4556/" "Mozilla/5.0 (X11; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0"

mikeshield13 commented 7 years ago

@mattrobenolt I've just created a new virtual server from an image with Sentry 8.4, then upgraded version by version. The problem first occurred when I was upgrading to 8.7 from 8.6.

I've found this in relase notes, maybe these are the problematic schema changes?

Added Environment model. Added ReleaseEnvironment model.

sentry upgrade log:

Syncing...                                     
Creating tables ...                       
Installing custom SQL ...                                                            
Installing indexes ...                                                               
Installed 0 object(s) from 0 fixture(s)

Synced:                                
 > django.contrib.admin          
 > django.contrib.auth
 > django.contrib.contenttypes        
 > django.contrib.messages             
 > django.contrib.sessions    
 > django.contrib.sites
 > django.contrib.staticfiles      
 > captcha                             
 > crispy_forms                    
 > debug_toolbar     
 > raven.contrib.django.raven_compat    
 > rest_framework                      
 > sentry.plugins.sentry_interface_types
 > sentry.plugins.sentry_mail        
 > sentry.plugins.sentry_urls                                       
 > sentry.plugins.sentry_useragents                                   
 > sentry.plugins.sentry_webhooks                                  
 > south                             
 > sudo

Not synced (use migrations):                                                                                                                     
 - sentry                                                                                                                                        
 - sentry.nodestore
 - sentry.search
 - social_auth
(use ./manage.py migrate to migrate these)
Running migrations for sentry:
 - Migrating forwards to 0268_fill_environment.
 > sentry:0263_remove_default_regression_rule
Rules: 100% |#####################################################################################################################| Time: 0:00:00
 > sentry:0264_drop_grouptagvalue_project_index
 > sentry:0265_auto__add_field_rule_status
 > sentry:0266_auto__add_grouprelease__add_unique_grouprelease_group_id_release_id_en
 > sentry:0267_auto__add_environment__add_unique_environment_project_id_name__add_rel
 > sentry:0268_fill_environment
 - Loading initial data for sentry.
Installed 0 object(s) from 0 fixture(s)
Running migrations for nodestore:
- Nothing to migrate.
 - Loading initial data for nodestore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
- Nothing to migrate.
 - Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for social_auth:
- Nothing to migrate.
 - Loading initial data for social_auth.
Installed 0 object(s) from 0 fixture(s)
Creating missing DSNs
Correcting Group.num_comments counter

Any idea?

mikeshield13 commented 7 years ago

Screenshot of Sentry 8.6: 8 6

Screenshot of Sentry 8.7: 8 7

Could you give me some advices?

dcramer commented 7 years ago

@mikeshield13 look at stderr/internal Sentry. Either way this issue is not an appropriate support forum given its almost certainly a configuration issue.

mikeshield13 commented 7 years ago

I checked the logs and stderr/internal, but no luck. This is the only thing I've found:

127.0.0.1 - - [23/Aug/2017:19:46:29 +0000] "GET /api/0/issues/3509/environments/none/?since=1503428400&until=1503514801 HTTP/1.0" 404 624 "http://sentrytest.mikeshield.c om/MS/mssheld/issues/3509/" "Mozilla/5.0 (X11; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0"

So there's a 404 error, when Sentry tries to get the environment details.

mattrobenolt commented 7 years ago

@mikeshield13 Can you try manually running:

from sentry.models import GroupRelease, Environment, ReleaseEnvironment
from django.db import IntegrityError, transaction
from django.db.models import Min, Max
from sentry.utils.query import RangeQuerySetWrapperWithProgressBar
from django.utils import timezone
from datetime import timedelta
queryset = GroupRelease.objects.filter(
    # limit the scope of data, even though that means we might not
    # capture all historical values
    last_seen__gte=timezone.now() - timedelta(days=1),
    last_seen__lte=timezone.now(),
)
environments = {}
for gr in RangeQuerySetWrapperWithProgressBar(queryset):
    env, _ = Environment.objects.get_or_create(
        project_id=gr.project_id,
        name=gr.environment,
    )
    environments[(env.project_id, env.name)] = env.id
for gr in RangeQuerySetWrapperWithProgressBar(queryset):
    env_id = environments.get((gr.project_id, gr.environment))
    if not env_id:
        continue
    try:
        with transaction.atomic():
            renv = ReleaseEnvironment.objects.create(
                project_id=gr.project_id,
                release_id=gr.release_id,
                environment_id=env_id,
            )
    except IntegrityError:
        pass
    else:
        # we do this after creation to avoid querying it when
        # we dont actually need to create the row (as its not cheap)
        results = GroupRelease.objects.filter(
            project_id=gr.project_id,
            release_id=gr.release_id,
            environment=gr.environment,
        ).aggregate(
            first=Min('first_seen'), last=Max('last_seen')
        )
        ReleaseEnvironment.objects.filter(
            id=renv.id,
        ).update(
            first_seen=results['first'],
            last_seen=results['last'],
        )

Inside of a $ sentry shell session?

This is the data migration that should have run from migration 0268, but for some reason, it seems that maybe you don't have this data?

https://github.com/getsentry/sentry/blob/master/src/sentry/south_migrations/0268_fill_environment.py

mikeshield13 commented 7 years ago 
Python 2.7.5 (default, Nov  6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ImportError: cannot import name Min
  File "<console>", line 7
    for gr in RangeQuerySetWrapperWithProgressBar(queryset):
      ^
SyntaxError: invalid syntax
  File "<console>", line 1
    env_id = environments.get((gr.project_id, gr.environment))
   ^                                      
IndentationError: unexpected indent
  File "<console>", line 1
    if not env_id:
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    continue
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    try:
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    with transaction.atomic():
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    renv = ReleaseEnvironment.objects.create(
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    project_id=gr.project_id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    release_id=gr.release_id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    environment_id=env_id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    )                                     
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    except IntegrityError:
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    pass
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    else:
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    results = GroupRelease.objects.filter(
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    project_id=gr.project_id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    release_id=gr.release_id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    environment=gr.environment,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    ).aggregate(
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    first=Min('first_seen'), last=Max('last_seen')
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    )
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    ReleaseEnvironment.objects.filter(
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    id=renv.id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    ).update(
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    first_seen=results['first'],
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    last_seen=results['last'],
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    )
   ^
IndentationError: unexpected indent
mattrobenolt commented 7 years ago

I think you caught it right before my edit. :) I fixed an import.

mikeshield13 commented 7 years ago

As I see there are some syntax and formatting errors. (Sorry I can't code in Python)

Python 2.7.5 (default, Nov  6 2016, 00:28:07)                                        
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2                                    
Type "help", "copyright", "credits" or "license" for more information.               
(InteractiveConsole)                      
  File "<console>", line 7                
    for gr in RangeQuerySetWrapperWithProgressBar(queryset):                         
      ^                                   
SyntaxError: invalid syntax               
  File "<console>", line 1                
    env_id = environments.get((gr.project_id, gr.environment))                       
   ^                                      
IndentationError: unexpected indent       
  File "<console>", line 1                
    if not env_id:                        
   ^                                      
IndentationError: unexpected indent       
  File "<console>", line 1                
    continue                              
   ^                                      
IndentationError: unexpected indent       
  File "<console>", line 1                
    try:                                  
   ^                                      
IndentationError: unexpected indent       
  File "<console>", line 1                
    with transaction.atomic():            
   ^                                      
IndentationError: unexpected indent       
  File "<console>", line 1                
    renv = ReleaseEnvironment.objects.create(                                        
   ^                                      
IndentationError: unexpected indent       
  File "<console>", line 1                
    project_id=gr.project_id,             
   ^                                      
IndentationError: unexpected indent       
  File "<console>", line 1                
    release_id=gr.release_id,             
   ^                                      
IndentationError: unexpected indent       
  File "<console>", line 1            
  File "<console>", line 1                
    environment_id=env_id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    )
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    except IntegrityError:
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    pass
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    else:
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    results = GroupRelease.objects.filter(
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    project_id=gr.project_id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    release_id=gr.release_id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    environment=gr.environment,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    ).aggregate(
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    first=Min('first_seen'), last=Max('last_seen')
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    )
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    ReleaseEnvironment.objects.filter(
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    id=renv.id,
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    ).update(
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    first_seen=results['first'],
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    last_seen=results['last'],
   ^
IndentationError: unexpected indent
  File "<console>", line 1
    )
   ^
IndentationError: unexpected indent
mattrobenolt commented 7 years ago

I think you need to exit() out of the shell and start over. You're in a wonky state. I just copy/pasted it into a new shell session without any issues.

mattrobenolt commented 7 years ago

Alternatively, this might be a bit easier:

$ curl https://gist.githubusercontent.com/mattrobenolt/3abfd0a86609d062554d26a38bc4233f/raw/fix.py | sentry exec
mikeshield13 commented 7 years ago

It doesn't write anything into output with sentry exec, but I get the same error message If I run this way:

SENTRY_CONF=/etc/sentry sentry shell < fix.py

mattrobenolt commented 7 years ago

Yeah, there wasn't expected to have any output. This was meant to intend to repair your data. Do things work fine now when you restart the web processes?

mikeshield13 commented 7 years ago

Unfortunately it's the same: There was an error loading data.

mattrobenolt commented 7 years ago

I'm not sure. :( Going to have to rope in @macqueen here since she wrote most of this.

macqueen commented 7 years ago

@mikeshield13 is https://github.com/getsentry/sentry/issues/5948#issuecomment-324340378 actually the full output of sentry upgrade? if you're upgrading to 8.19 you should be on migration 0341. can you paste the output of sentry django migrate --list?

macqueen commented 7 years ago

i think this might have to do with our changing the directory of our migrations from migrations to south_migrations

macqueen commented 7 years ago

hmm or maybe not since it seems like you made it past when we switched them, but it looks like the issue is that your migrations didn't finish running

mikeshield13 commented 7 years ago

Nope, that is the output of the upgrading from 8.6 to 8.7. Sentry 8.7 is the first release, where those environment stuffs don't work. I put a screenshot from both releases. Actually I use Sentry 8.7, not 8.19.

root@sentrytest /root] sentry django migrate --list

 sentry
  (*) 0001_initial
  (*) 0002_auto__del_field_groupedmessage_url__chg_field_groupedmessage_view__chg
  (*) 0003_auto__add_field_message_group__del_field_groupedmessage_server_name
  (*) 0004_auto__add_filtervalue__add_unique_filtervalue_key_value
  (*) 0005_auto
  (*) 0006_auto
  (*) 0007_auto__add_field_message_site
  (*) 0008_auto__chg_field_message_view__add_field_groupedmessage_data__chg_field
  (*) 0009_auto__add_field_message_message_id
  (*) 0010_auto__add_messageindex__add_unique_messageindex_column_value_object_id
  (*) 0011_auto__add_field_groupedmessage_score
  (*) 0012_auto
  (*) 0013_auto__add_messagecountbyminute__add_unique_messagecountbyminute_group_
  (*) 0014_auto
  (*) 0014_auto__add_project__add_projectmember__add_unique_projectmember_project
  (*) 0015_auto__add_field_message_project__add_field_messagecountbyminute_projec
  (*) 0016_auto__add_field_projectmember_is_superuser
  (*) 0017_auto__add_field_projectmember_api_key
  (*) 0018_auto__chg_field_project_owner
  (*) 0019_auto__del_field_projectmember_api_key__add_field_projectmember_public_
  (*) 0020_auto__add_projectdomain__add_unique_projectdomain_project_domain
  (*) 0021_auto__del_message__del_groupedmessage__del_unique_groupedmessage_proje
  (*) 0022_auto__del_field_group_class_name__del_field_group_traceback__del_field
  (*) 0023_auto__add_field_event_time_spent
  (*) 0024_auto__add_field_group_time_spent_total__add_field_group_time_spent_cou
  (*) 0025_auto__add_field_messagecountbyminute_time_spent_total__add_field_messa
  (*) 0026_auto__add_field_project_status
  (*) 0027_auto__chg_field_event_server_name
  (*) 0028_auto__add_projectoptions__add_unique_projectoptions_project_key_value
  (*) 0029_auto__del_field_projectmember_is_superuser__del_field_projectmember_pe
  (*) 0030_auto__add_view__chg_field_event_group
  (*) 0031_auto__add_field_view_verbose_name__add_field_view_verbose_name_plural_
  (*) 0032_auto__add_eventmeta
  (*) 0033_auto__add_option__add_unique_option_key_value
  (*) 0034_auto__add_groupbookmark__add_unique_groupbookmark_project_user_group
  (*) 0034_auto__add_unique_option_key__del_unique_option_value_key__del_unique_g
  (*) 0036_auto__chg_field_option_value__chg_field_projectoption_value
  (*) 0037_auto__add_unique_option_key__del_unique_option_value_key__del_unique_g
  (*) 0038_auto__add_searchtoken__add_unique_searchtoken_document_field_token__ad
  (*) 0039_auto__add_field_searchdocument_status
  (*) 0040_auto__del_unique_event_event_id__add_unique_event_project_event_id
  (*) 0041_auto__add_field_messagefiltervalue_last_seen__add_field_messagefilterv
  (*) 0042_auto__add_projectcountbyminute__add_unique_projectcountbyminute_projec
  (*) 0043_auto__chg_field_option_value__chg_field_projectoption_value
  (*) 0044_auto__add_field_projectmember_is_active
  (*) 0045_auto__add_pendingprojectmember__add_unique_pendingprojectmember_projec
  (*) 0046_auto__add_teammember__add_unique_teammember_team_user__add_team__add_p
  (*) 0047_migrate_project_slugs
  (*) 0048_migrate_project_keys
  (*) 0049_create_default_project_keys
  (*) 0050_remove_project_keys_from_members
  (*) 0051_auto__del_pendingprojectmember__del_unique_pendingprojectmember_projec
  (*) 0052_migrate_project_members
  (*) 0053_auto__del_projectmember__del_unique_projectmember_project_user
  (*) 0054_fix_project_keys
  (*) 0055_auto__del_projectdomain__del_unique_projectdomain_project_domain
  (*) 0056_auto__add_field_group_resolved_at
  (*) 0057_auto__add_field_group_active_at
  (*) 0058_auto__add_useroption__add_unique_useroption_user_project_key
  (*) 0059_auto__add_filterkey__add_unique_filterkey_project_key
  (*) 0060_fill_filter_key
  (*) 0061_auto__add_field_group_group_id__add_field_group_is_public
  (*) 0062_correct_del_index_sentry_groupedmessage_logger__view__checksum
  (*) 0063_auto
  (*) 0064_index_checksum
  (*) 0065_create_default_project_key
  (*) 0066_auto__del_view
  (*) 0067_auto__add_field_group_platform__add_field_event_platform
  (*) 0068_auto__add_field_projectkey_user_added__add_field_projectkey_date_added
  (*) 0069_auto__add_lostpasswordhash
  (*) 0070_projectoption_key_length
  (*) 0071_auto__add_field_group_users_seen
  (*) 0072_auto__add_affecteduserbygroup__add_unique_affecteduserbygroup_project_
  (*) 0073_auto__add_field_project_platform
  (*) 0074_correct_filtervalue_index
  (*) 0075_add_groupbookmark_index
  (*) 0076_add_groupmeta_index
  (*) 0077_auto__add_trackeduser__add_unique_trackeduser_project_ident
  (*) 0078_auto__add_field_affecteduserbygroup_tuser
  (*) 0079_auto__del_unique_affecteduserbygroup_project_ident_group__add_unique_a
  (*) 0080_auto__chg_field_affecteduserbygroup_ident
  (*) 0081_fill_trackeduser
  (*) 0082_auto__add_activity__add_field_group_num_comments__add_field_event_num_
  (*) 0083_migrate_dupe_groups
  (*) 0084_auto__del_unique_group_project_checksum_logger_culprit__add_unique_gro
  (*) 0085_auto__del_unique_project_slug__add_unique_project_slug_team
  (*) 0086_auto__add_field_team_date_added
  (*) 0087_auto__del_messagefiltervalue__del_unique_messagefiltervalue_project_ke
  (*) 0088_auto__del_messagecountbyminute__del_unique_messagecountbyminute_projec
  (*) 0089_auto__add_accessgroup__add_unique_accessgroup_team_name
  (*) 0090_auto__add_grouptagkey__add_unique_grouptagkey_project_group_key__add_f
  (*) 0091_auto__add_alert
  (*) 0092_auto__add_alertrelatedgroup__add_unique_alertrelatedgroup_group_alert
  (*) 0093_auto__add_field_alert_status
  (*) 0094_auto__add_eventmapping__add_unique_eventmapping_project_event_id
  (*) 0095_rebase
  (*) 0096_auto__add_field_tagvalue_data
  (*) 0097_auto__del_affecteduserbygroup__del_unique_affecteduserbygroup_project_
  (*) 0098_auto__add_user__chg_field_team_owner__chg_field_activity_user__chg_fie
  (*) 0099_auto__del_field_teammember_is_active
  (*) 0100_auto__add_field_tagkey_label
  (*) 0101_ensure_teams
  (*) 0102_ensure_slugs
  (*) 0103_ensure_non_empty_slugs
  (*) 0104_auto__add_groupseen__add_unique_groupseen_group_user
  (*) 0105_auto__chg_field_projectcountbyminute_time_spent_total__chg_field_group
  (*) 0106_auto__del_searchtoken__del_unique_searchtoken_document_field_token__de
  (*) 0107_expand_user
  (*) 0108_fix_user
  (*) 0109_index_filtervalue_times_seen
  (*) 0110_index_filtervalue_last_seen
  (*) 0111_index_filtervalue_first_seen
  (*) 0112_auto__chg_field_option_value__chg_field_useroption_value__chg_field_pr
  (*) 0113_auto__add_field_team_status
  (*) 0114_auto__add_field_projectkey_roles
  (*) 0115_auto__del_projectcountbyminute__del_unique_projectcountbyminute_projec
  (*) 0116_auto__del_field_event_server_name__del_field_event_culprit__del_field_
  (*) 0117_auto__add_rule
  (*) 0118_create_default_rules
  (*) 0119_auto__add_field_projectkey_label
  (*) 0120_auto__add_grouprulestatus
  (*) 0121_auto__add_unique_grouprulestatus_rule_group
  (*) 0122_add_event_group_id_datetime_index
  (*) 0123_auto__add_groupassignee__add_index_event_group_datetime
  (*) 0124_auto__add_grouphash__add_unique_grouphash_project_hash
  (*) 0125_auto__add_field_user_is_managed
  (*) 0126_auto__add_field_option_last_updated
  (*) 0127_auto__add_release__add_unique_release_project_version
  (*) 0128_auto__add_broadcast
  (*) 0129_auto__chg_field_release_id__chg_field_pendingteammember_id__chg_field_
  (*) 0130_auto__del_field_project_owner
  (*) 0131_auto__add_organizationmember__add_unique_organizationmember_organizati
  (*) 0132_add_default_orgs
  (*) 0133_add_org_members
  (*) 0134_auto__chg_field_team_organization
  (*) 0135_auto__chg_field_project_team
  (*) 0136_auto__add_field_organizationmember_email__chg_field_organizationmember
  (*) 0137_auto__add_field_organizationmember_has_global_access
  (*) 0138_migrate_team_members
  (*) 0139_auto__add_auditlogentry
  (*) 0140_auto__add_field_organization_slug
  (*) 0141_fill_org_slugs
  (*) 0142_auto__add_field_project_organization__add_unique_project_organization_
  (*) 0143_fill_project_orgs
  (*) 0144_auto__chg_field_project_organization
  (*) 0145_auto__chg_field_organization_slug
  (*) 0146_auto__add_field_auditlogentry_ip_address
  (*) 0147_auto__del_unique_team_slug__add_unique_team_organization_slug
  (*) 0148_auto__add_helppage
  (*) 0149_auto__chg_field_groupseen_project__chg_field_groupseen_user__chg_field
  (*) 0150_fix_broken_rules
  (*) 0151_auto__add_file
  (*) 0152_auto__add_field_file_checksum__chg_field_file_name__add_unique_file_na
  (*) 0153_auto__add_field_grouprulestatus_last_active
  (*) 0154_auto__add_field_tagkey_status
  (*) 0155_auto__add_field_projectkey_status
  (*) 0156_auto__add_apikey
  (*) 0157_auto__add_authidentity__add_unique_authidentity_auth_provider_ident__a
  (*) 0158_auto__add_unique_authidentity_auth_provider_user
  (*) 0159_auto__add_field_authidentity_last_verified__add_field_organizationmemb
  (*) 0160_auto__add_field_authprovider_default_global_access
  (*) 0161_auto__chg_field_authprovider_config
  (*) 0162_auto__chg_field_authidentity_data
  (*) 0163_auto__add_field_authidentity_last_synced
  (*) 0164_auto__add_releasefile__add_unique_releasefile_release_ident__add_field
  (*) 0165_auto__del_unique_file_name_checksum
  (*) 0166_auto__chg_field_user_id__add_field_apikey_allowed_origins
  (*) 0167_auto__add_field_authprovider_flags
  (*) 0168_unfill_projectkey_user
  (*) 0169_auto__del_field_projectkey_user
  (*) 0170_auto__add_organizationmemberteam__add_unique_organizationmemberteam_te
  (*) 0171_auto__chg_field_team_owner
  (*) 0172_auto__del_field_team_owner
  (*) 0173_auto__del_teammember__del_unique_teammember_team_user
  (*) 0174_auto__del_field_projectkey_user_added
  (*) 0175_auto__del_pendingteammember__del_unique_pendingteammember_team_email
  (*) 0176_auto__add_field_organizationmember_counter__add_unique_organizationmem
  (*) 0177_fill_member_counters
  (*) 0178_auto__del_unique_organizationmember_organization_counter
  (*) 0179_auto__add_field_release_date_released
  (*) 0180_auto__add_field_release_environment__add_field_release_ref__add_field_
  (*) 0181_auto__del_field_release_environment__del_unique_release_project_versio
  (*) 0182_auto__add_field_auditlogentry_actor_label__add_field_auditlogentry_act
  (*) 0183_auto__del_index_grouphash_hash
  (*) 0184_auto__del_field_group_checksum__del_unique_group_project_checksum__del
  (*) 0185_auto__add_savedsearch__add_unique_savedsearch_project_name
  (*) 0186_auto__add_field_group_first_release
  (*) 0187_auto__add_index_group_project_first_release
  (*) 0188_auto__add_userreport
  (*) 0189_auto__add_index_userreport_project_event_id
  (*) 0190_auto__add_field_release_new_groups
  (*) 0191_auto__del_alert__del_alertrelatedgroup__del_unique_alertrelatedgroup_g
  (*) 0192_add_model_groupemailthread
  (*) 0193_auto__del_unique_groupemailthread_msgid__add_unique_groupemailthread_e
  (*) 0194_auto__del_field_project_platform
  (*) 0195_auto__chg_field_organization_owner
  (*) 0196_auto__del_field_organization_owner
  (*) 0197_auto__del_accessgroup__del_unique_accessgroup_team_name
  (*) 0198_auto__add_field_release_primary_owner
  (*) 0199_auto__add_field_project_first_event
  (*) 0200_backfill_first_event
  (*) 0201_auto__add_eventuser__add_unique_eventuser_project_ident__add_index_eve
  (*) 0202_auto__add_field_eventuser_hash__add_unique_eventuser_project_hash
  (*) 0203_auto__chg_field_eventuser_username__chg_field_eventuser_ident
  (*) 0204_backfill_team_membership
  (*) 0205_auto__add_field_organizationmember_role
  (*) 0206_backfill_member_role
  (*) 0207_auto__add_field_organization_default_role
  (*) 0208_backfill_default_role
  (*) 0209_auto__add_broadcastseen__add_unique_broadcastseen_broadcast_user
  (*) 0210_auto__del_field_broadcast_badge
  (*) 0211_auto__add_field_broadcast_title
  (*) 0212_auto__add_fileblob__add_field_file_blob
  (*) 0212_auto__add_organizationoption__add_unique_organizationoption_organizati
  (*) 0213_migrate_file_blobs
  (*) 0214_auto__add_field_broadcast_upstream_id
  (*) 0215_auto__add_field_broadcast_date_expires
  (*) 0216_auto__add_groupsnooze
  (*) 0217_auto__add_groupresolution
  (*) 0218_auto__add_field_groupresolution_status
  (*) 0219_auto__add_field_groupbookmark_date_added
  (*) 0220_auto__del_field_fileblob_storage_options__del_field_fileblob_storage__
  (*) 0221_auto__chg_field_user_first_name
  (*) 0222_auto__del_field_user_last_name__del_field_user_first_name__add_field_u
  (*) 0223_delete_old_sentry_docs_options
  (*) 0224_auto__add_index_userreport_project_date_added
  (*) 0225_auto__add_fileblobindex__add_unique_fileblobindex_file_blob_offset
  (*) 0226_backfill_file_size
  (*) 0227_auto__del_field_activity_event
  (*) 0228_auto__del_field_event_num_comments
  (*) 0229_drop_event_constraints
  (*) 0230_auto__del_field_eventmapping_group__del_field_eventmapping_project__ad
  (*) 0231_auto__add_field_savedsearch_is_default
  (*) 0232_default_savedsearch
  (*) 0233_add_new_savedsearch
  (*) 0234_auto__add_savedsearchuserdefault__add_unique_savedsearchuserdefault_pr
  (*) 0235_auto__add_projectbookmark__add_unique_projectbookmark_project_id_user_
  (*) 0236_auto__add_organizationonboardingtask__add_unique_organizationonboardin
  (*) 0237_auto__add_eventtag__add_unique_eventtag_event_id_key_id_value_id
  (*) 0238_fill_org_onboarding_tasks
  (*) 0239_auto__add_projectdsymfile__add_unique_projectdsymfile_project_uuid__ad
  (*) 0240_fill_onboarding_option
  (*) 0241_auto__add_counter__add_unique_counter_project_ident__add_field_group_s
  (*) 0242_auto__add_field_project_forced_color
  (*) 0243_remove_inactive_members
  (*) 0244_auto__add_groupredirect
  (*) 0245_auto__del_field_project_callsign__del_unique_project_organization_call
  (*) 0246_auto__add_dsymsymbol__add_unique_dsymsymbol_object_address__add_dsymsd
  (*) 0247_migrate_file_blobs
  (*) 0248_auto__add_projectplatform__add_unique_projectplatform_project_id_platf
  (*) 0249_auto__add_index_eventtag_project_id_key_id_value_id
  (*) 0250_auto__add_unique_userreport_project_event_id
  (*) 0251_auto__add_useravatar
  (*) 0252_default_users_to_gravatar
  (*) 0253_auto__add_field_eventtag_group_id
  (*) 0254_auto__add_index_eventtag_group_id_key_id_value_id
  (*) 0255_auto__add_apitoken
  (*) 0256_auto__add_authenticator
  (*) 0257_repair_activity
  (*) 0258_auto__add_field_user_is_password_expired__add_field_user_last_password
  (*) 0259_auto__add_useremail__add_unique_useremail_user_email
  (*) 0260_populate_email_addresses
  (*) 0261_auto__add_groupsubscription__add_unique_groupsubscription_group_user
  (*) 0262_fix_tag_indexes
  (*) 0263_remove_default_regression_rule
  (*) 0264_drop_grouptagvalue_project_index
  (*) 0265_auto__add_field_rule_status
  (*) 0266_auto__add_grouprelease__add_unique_grouprelease_group_id_release_id_en
  (*) 0267_auto__add_environment__add_unique_environment_project_id_name__add_rel
  (*) 0268_fill_environment

 nodestore
  (*) 0001_initial

 search
  (*) 0001_initial
  (*) 0002_auto__del_searchtoken__del_unique_searchtoken_document_field_token__de

 social_auth
  (*) 0001_initial
  (*) 0002_auto__add_unique_nonce_timestamp_salt_server_url__add_unique_associati
mikeshield13 commented 7 years ago

Any idea? :)

mattrobenolt commented 7 years ago

Version 8.7 is really old. The latest is 8.19. Do you still have this when you upgrade to 8.19? For context, 8.7 is over a year old now.

mikeshield13 commented 7 years ago

Yes. First I upgraded to 8.19 and had this issue. Then I tried to upgrade my Sentry step by step, release by release. And finally I found out the problem first appeared between the upgrade from 8.6 to 8.7.

adamcharnock commented 6 years ago

I've also experienced this issue when deploying to kubernetes using the helm chart. The database init container exits with status 0, with the last log line of Correcting Group.num_comments counter.

Deploying Sentry 8.20.

Googling also turning up the old #3004.