search

18F / 2015-foia-hub

A consolidated FOIA request hub.
Other
48 stars 17 forks source link

Data Loading is failing in production and staging #770

Closed khandelwal closed 9 years ago

khandelwal commented 9 years ago

It looks like data loading is failing in production and staging. 

2015-04-29T16:25:56.05+0000 [App/0]      ERR error: Untracked working tree file '.gitignore' would be overwritten by merge.
2015-04-29T16:25:56.06+0000 [App/0]      ERR fatal: unable to checkout working tree
2015-04-29T16:25:56.06+0000 [App/0]      ERR warning: Clone succeeded, but checkout failed.
2015-04-29T16:25:56.06+0000 [App/0]      ERR You can inspect what was checked out with 'git status'
2015-04-29T16:25:56.06+0000 [App/0]      ERR and retry the checkout with 'git checkout -f HEAD'
2015-04-29T16:25:56.06+0000 [App/0]      ERR Traceback (most recent call last):
2015-04-29T16:25:56.06+0000 [App/0]      ERR   File "manage.py", line 10, in <module>
2015-04-29T16:25:56.06+0000 [App/0]      ERR     execute_from_command_line(sys.argv)
2015-04-29T16:25:56.06+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
2015-04-29T16:25:56.06+0000 [App/0]      ERR     utility.execute()
2015-04-29T16:25:56.06+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
2015-04-29T16:25:56.06+0000 [App/0]      ERR     self.fetch_command(subcommand).run_from_argv(self.argv)
2015-04-29T16:25:56.07+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
2015-04-29T16:25:56.07+0000 [App/0]      ERR     self.execute(*args, **options.__dict__)
2015-04-29T16:25:56.07+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 338, in execute
2015-04-29T16:25:56.07+0000 [App/0]      ERR     output = self.handle(*args, **options)
2015-04-29T16:25:56.07+0000 [App/0]      ERR   File "/home/vcap/app/foia_hub/management/commands/load_agency_contacts.py", line 21, in handle
2015-04-29T16:25:56.07+0000 [App/0]      ERR     yaml_folder = download_data(self)
2015-04-29T16:25:56.07+0000 [App/0]      ERR   File "/home/vcap/app/foia_hub/management/commands/load_agency_contacts.py", line 38, in download_data
2015-04-29T16:25:56.07+0000 [App/0]      ERR     clone_repo(DEFAULT_DATA_REPO, path)
2015-04-29T16:25:56.07+0000 [App/0]      ERR   File "/home/vcap/app/foia_hub/management/commands/load_agency_contacts.py", line 46, in clone_repo
2015-04-29T16:25:56.07+0000 [App/0]      ERR     subprocess.check_call(["git", "clone", repo, directory], shell=False)
2015-04-29T16:25:56.07+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/subprocess.py", line 559, in check_call
2015-04-29T16:25:56.07+0000 [App/0]      ERR     raise CalledProcessError(retcode, cmd)
2015-04-29T16:25:56.08+0000 [App/0]      ERR subprocess.CalledProcessError: Command '['git', 'clone', 'https://github.com/18F/foia.git', '/home/vcap/app/foia_hub/../temp-data']' returned non-zero exit status 128
2015-04-29T16:25:56.14+0000 [App/0]      ERR warning: Duplicated ref: refs/remotes/origin/add_pclob
2015-04-29T16:25:56.14+0000 [App/0]      ERR warning: Duplicated ref: refs/remotes/origin/email_update
2015-04-29T16:25:56.14+0000 [App/0]      ERR warning: Duplicated ref: refs/remotes/origin/extract_contacts
2015-04-29T16:25:56.14+0000 [App/0]      ERR warning: Duplicated ref: refs/remotes/origin/map_more_time_data
2015-04-29T16:25:56.14+0000 [App/0]      ERR warning: Duplicated ref: refs/remotes/origin/master
2015-04-29T16:25:56.14+0000 [App/0]      ERR warning: Duplicated ref: refs/remotes/origin/phone_number_update
2015-04-29T16:25:56.14+0000 [App/0]      ERR warning: Duplicated ref: refs/remotes/origin/project-monitor
2015-04-29T16:25:56.14+0000 [App/0]      ERR warning: Duplicated ref: refs/remotes/origin/revert-95-address_update
2015-04-29T16:25:56.14+0000 [App/0]      ERR warning: Duplicated ref: refs/tags/0.1
2015-04-29T16:25:56.15+0000 [App/0]      ERR error: Untracked working tree file '.gitignore' would be overwritten by merge.
2015-04-29T16:25:56.16+0000 [App/0]      ERR fatal: unable to checkout working tree
2015-04-29T16:25:56.16+0000 [App/0]      ERR warning: Clone succeeded, but checkout failed.
2015-04-29T16:25:56.16+0000 [App/0]      ERR You can inspect what was checked out with 'git status'
2015-04-29T16:25:56.16+0000 [App/0]      ERR and retry the checkout with 'git checkout -f HEAD'
2015-04-29T16:25:56.16+0000 [App/0]      ERR Traceback (most recent call last):
2015-04-29T16:25:56.16+0000 [App/0]      ERR   File "manage.py", line 10, in <module>
2015-04-29T16:25:56.16+0000 [App/0]      ERR     execute_from_command_line(sys.argv)
2015-04-29T16:25:56.16+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
2015-04-29T16:25:56.16+0000 [App/0]      ERR     utility.execute()
2015-04-29T16:25:56.16+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
2015-04-29T16:25:56.16+0000 [App/0]      ERR     self.fetch_command(subcommand).run_from_argv(self.argv)
2015-04-29T16:25:56.16+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
2015-04-29T16:25:56.17+0000 [App/0]      ERR     self.execute(*args, **options.__dict__)
2015-04-29T16:25:56.17+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 338, in execute
2015-04-29T16:25:56.17+0000 [App/0]      ERR     output = self.handle(*args, **options)
2015-04-29T16:25:56.17+0000 [App/0]      ERR   File "/home/vcap/app/foia_hub/management/commands/load_agency_contacts.py", line 21, in handle
2015-04-29T16:25:56.17+0000 [App/0]      ERR     yaml_folder = download_data(self)
2015-04-29T16:25:56.17+0000 [App/0]      ERR   File "/home/vcap/app/foia_hub/management/commands/load_agency_contacts.py", line 38, in download_data
2015-04-29T16:25:56.17+0000 [App/0]      ERR     clone_repo(DEFAULT_DATA_REPO, path)
2015-04-29T16:25:56.17+0000 [App/0]      ERR   File "/home/vcap/app/foia_hub/management/commands/load_agency_contacts.py", line 46, in clone_repo
2015-04-29T16:25:56.17+0000 [App/0]      ERR     subprocess.check_call(["git", "clone", repo, directory], shell=False)
2015-04-29T16:25:56.17+0000 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/subprocess.py", line 559, in check_call
2015-04-29T16:25:56.17+0000 [App/0]      ERR     raise CalledProcessError(retcode, cmd)
2015-04-29T16:25:56.17+0000 [App/0]      ERR subprocess.CalledProcessError: Command '['git', 'clone', 'https://github.com/18F/foia.git', '/home/vcap/app/foia_hub/../temp-data']' returned non-zero exit status 128
geramirez commented 9 years ago

So it looks like the tmp dir was an issue. However, now were are getting a new issue..

2015-04-29T13:41:40.42-0400 [App/0]      ERR Traceback (most recent call last):
2015-04-29T13:41:40.42-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
2015-04-29T13:41:40.42-0400 [App/0]      ERR     return self.cursor.execute(sql, params)
2015-04-29T13:41:40.42-0400 [App/0]      ERR psycopg2.IntegrityError: duplicate key value violates unique constraint "foia_hub_stats_agency_id_6b16dc7079dd4de9_uniq"
2015-04-29T13:41:40.42-0400 [App/0]      ERR DETAIL:  Key (agency_id, office_id, year, stat_type)=(20, 11, 2014, S) already exists.
2015-04-29T13:41:40.42-0400 [App/0]      ERR The above exception was the direct cause of the following exception:
2015-04-29T13:41:40.42-0400 [App/0]      ERR Traceback (most recent call last):
2015-04-29T13:41:40.42-0400 [App/0]      ERR   File "manage.py", line 10, in <module>
2015-04-29T13:41:40.42-0400 [App/0]      ERR     execute_from_command_line(sys.argv)
2015-04-29T13:41:40.42-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
2015-04-29T13:41:40.43-0400 [App/0]      ERR     utility.execute()
2015-04-29T13:41:40.43-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
2015-04-29T13:41:40.43-0400 [App/0]      ERR     self.fetch_command(subcommand).run_from_argv(self.argv)
2015-04-29T13:41:40.43-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
2015-04-29T13:41:40.44-0400 [App/0]      ERR     self.execute(*args, **options.__dict__)
2015-04-29T13:41:40.44-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 338, in execute
2015-04-29T13:41:40.44-0400 [App/0]      ERR     output = self.handle(*args, **options)
2015-04-29T13:41:40.44-0400 [App/0]      ERR   File "/home/vcap/app/foia_hub/management/commands/load_agency_contacts.py", line 26, in handle
2015-04-29T13:41:40.44-0400 [App/0]      ERR     process_yamls(yaml_folder)
2015-04-29T13:41:40.44-0400 [App/0]      ERR   File "/home/vcap/app/foia_hub/scripts/load_agency_contacts.py", line 209, in process_yamls
2015-04-29T13:41:40.44-0400 [App/0]      ERR     load_data(data)
2015-04-29T13:41:40.44-0400 [App/0]      ERR   File "/home/vcap/app/foia_hub/scripts/load_agency_contacts.py", line 200, in load_data
2015-04-29T13:41:40.44-0400 [App/0]      ERR     add_request_time_statistics(dept_rec, a, o)
2015-04-29T13:41:40.44-0400 [App/0]      ERR   File "/home/vcap/app/foia_hub/scripts/load_agency_contacts.py", line 111, in add_request_time_statistics
2015-04-29T13:41:40.44-0400 [App/0]      ERR     stat.save()
2015-04-29T13:41:40.44-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/models/base.py", line 590, in save
2015-04-29T13:41:40.44-0400 [App/0]      ERR     force_update=force_update, update_fields=update_fields)
2015-04-29T13:41:40.44-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/models/base.py", line 618, in save_base
2015-04-29T13:41:40.44-0400 [App/0]      ERR     updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
2015-04-29T13:41:40.44-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/models/base.py", line 699, in _save_table
2015-04-29T13:41:40.45-0400 [App/0]      ERR     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
2015-04-29T13:41:40.45-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/models/base.py", line 732, in _do_insert
2015-04-29T13:41:40.45-0400 [App/0]      ERR     using=using, raw=raw)
2015-04-29T13:41:40.45-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/models/manager.py", line 92, in manager_method
2015-04-29T13:41:40.45-0400 [App/0]      ERR     return getattr(self.get_queryset(), name)(*args, **kwargs)
2015-04-29T13:41:40.45-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/models/query.py", line 921, in _insert
2015-04-29T13:41:40.45-0400 [App/0]      ERR     return query.get_compiler(using=using).execute_sql(return_id)
2015-04-29T13:41:40.45-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 920, in execute_sql
2015-04-29T13:41:40.45-0400 [App/0]      ERR     cursor.execute(sql, params)
2015-04-29T13:41:40.45-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
2015-04-29T13:41:40.45-0400 [App/0]      ERR     return self.cursor.execute(sql, params)
2015-04-29T13:41:40.45-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/utils.py", line 94, in __exit__
2015-04-29T13:41:40.45-0400 [App/0]      ERR     six.reraise(dj_exc_type, dj_exc_value, traceback)
2015-04-29T13:41:40.45-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/utils/six.py", line 549, in reraise
2015-04-29T13:41:40.45-0400 [App/0]      ERR     raise value.with_traceback(tb)
2015-04-29T13:41:40.45-0400 [App/0]      ERR   File "/app/.heroku/python/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
2015-04-29T13:41:40.46-0400 [App/0]      ERR     return self.cursor.execute(sql, params)
2015-04-29T13:41:40.46-0400 [App/0]      ERR django.db.utils.IntegrityError: duplicate key value violates unique constraint "foia_hub_stats_agency_id_6b16dc7079dd4de9_uniq"
2015-04-29T13:41:40.46-0400 [App/0]      ERR DETAIL:  Key (agency_id, office_id, year, stat_type)=(20, 11, 2014, S) already exists.
geramirez commented 9 years ago

It appears that Stats.objects.filter(agency=agency, office=office).delete() is not actually deleting the old object.

khandelwal commented 9 years ago

I think it's worth figuring out what's going on here.

geramirez commented 9 years ago

This should be fixed now.