(psycopg2.IntegrityError) null value in column "storage_backend" violates not-null constraint

[mokhinetc]

Good Morning,

I am getting the following error when trying to migrate fom 1.2 to 2.0.:

(psycopg2.IntegrityError) null value in column "storage_backend" violates not-null constraint
DETAIL:  Failing row contains (1, 109, 1, 0, null, application/pdf, , null, senekane_et_al_dp_student_t_mechanism.pdf).
 [SQL: 'INSERT INTO event_paper_reviewing.files (contribution_id, revision_id, size, storage_backend, content_type, md5, storage_file_id, filename) VALUES (%(contribution_id)s, %(revision_id)s, %(size)s, %(storage_backend)s, %(content_type)s, %(md5)s, %(storage_file_id)s, %(filename)s) RETURNING event_paper_reviewing.files.id'] [parameters: {'storage_file_id': None, 'filename': 'senekane_et_al_dp_student_t_mechanism.pdf', 'contribution_id': 109, 'storage_backend': None, 'content_type': 'application/pdf', 'revision_id': 1, 'md5': u'', 'size': 0}]

[ThiefMaster]

Hi, can you confirm this is the same problem as in the other issue? In any case, please include the traceback here.

[mokhinetc]

Yes it is the same issue.

[mokhinetc]

Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 [<module>]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/cli.py:126 [main]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:722 [__call__]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:697 [main]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:895 [invoke]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:535 [invoke]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/cli.py:122 [cli]
 debug=debug, **kwargs)
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:183 [migrate]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:211 [migrate_event_data]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:137 [run_step]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/__init__.py:83 [run]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/papers.py:116 [migrate]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/papers.py:344 [_migrate_papers]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/papers.py:326 [_migrate_revisions]
...
 |/opt/indico/.venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:470 [do_execute]
 cursor.execute(statement, parameters)

(psycopg2.IntegrityError) null value in column "storage_backend" violates not-null constraint
DETAIL:  Failing row contains (1, 109, 1, 0, null, application/pdf, , null, senekane_et_al_dp_student_t_mechanism.pdf).
 [SQL: 'INSERT INTO event_paper_reviewing.files (contribution_id, revision_id, size, storage_backend, content_type, md5, storage_file_id, filename) VALUES (%(contribution_id)s, %(revision_id)s, %(size)s, %(storage_backend)s, %(content_type)s, %(md5)s, %(storage_file_id)s, %(filename)s) RETURNING event_paper_reviewing.files.id'] [parameters: {'storage_file_id': None, 'filename': 'senekane_et_al_dp_student_t_mechanism.pdf', 'contribution_id': 109, 'storage_backend': None, 'content_type': 'application/pdf', 'revision_id': 1, 'md5': u'', 'size': 0}]

[ThiefMaster]

please try applying this patch:

diff --git a/indico_migrate/steps/events/papers.py b/indico_migrate/steps/events/papers.py
index 7179766..92a9eb8 100644
--- a/indico_migrate/steps/events/papers.py
+++ b/indico_migrate/steps/events/papers.py
@@ -383,6 +383,10 @@ class EventPaperReviewingImporter(LocalFileImporterMixin, EventMigrationStep):
         storage_backend, storage_path, size, md5 = self._get_local_file_info(resource)
         content_type = mimetypes.guess_type(resource.fileName)[0] or 'application/octet-stream'

+        if not storage_path:
+            self.print_error("%[red!]File not accessible [{}]".format(convert_to_unicode(resource.fileName)))
+            return
+
         paper_file = PaperFile(filename=resource.fileName, content_type=content_type,
                                size=size, md5=md5, storage_backend=storage_backend,
                                storage_file_id=storage_path, created_dt=created_dt)

[mokhinetc]

Hi @ThiefMaster , I've tried the patch and now i am getting the following error:

Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 [<module>]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/cli.py:126 [main]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:722 [__call__]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:697 [main]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:895 [invoke]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:535 [invoke]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/cli.py:122 [cli]
 debug=debug, **kwargs)
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:183 [migrate]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:194 [migrate_event_data]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:49 [_get_all_steps]
 from indico_migrate.steps.events.papers import EventPaperReviewingImporter

unindent does not match any outer indentation level (papers.py, line 95)

[ThiefMaster]

How did you apply the patch? This sounds a lot like the indentation is wrong, e.g. by adding it manually and using tabs instead of spaces to indent.

[mokhinetc]

Sorry about that, not much of a python programmer. below is how i inserted the code:

[ThiefMaster]

You need to insert it at the correct place. Please undo your change and add the snippet

        if not storage_path:
            self.print_error("%[red!]File not accessible [{}]".format(convert_to_unicode(resource.fileName)))
            return

below this line:

content_type = mimetypes.guess_type(resource.fileName)[0] or 'application/octet-stream'

[mokhinetc]

I don't know what I am doing wrong, can you please attach the updated file and will upload and run it on the server.

[mokhinetc]

I am getting the following error:

Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 [] |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/cli.py:126 [main] |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:722 [call] |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:697 [main] |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:895 [invoke] |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:535 [invoke] |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/cli.py:122 [cli] debug=debug, **kwargs) |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate] |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run] |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f] |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:183 [migrate] |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:194 [migrate_event_data] |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:49 [_get_all_steps] from indico_migrate.steps.events.papers import EventPaperReviewingImporter

'return' outside function (papers.py, line 97)

[ThiefMaster]

My last comment specified exactly where you had to put the code... Anyway, here's the modified file: https://bpaste.net/raw/ef2e8eedf772