search

OCA / OpenUpgrade

Open source upgrade path for Odoo/OpenERP
https://oca.github.io/OpenUpgrade/
GNU Affero General Public License v3.0
728 stars 693 forks source link

[12.0] module project: initial_value must be str or None, not bytes - error #1909

Closed jolentes closed 5 years ago

jolentes commented 5 years ago

Impacted versions: 11.0 -> 12.0

Steps to reproduce: I have a database that was create with 11.0 and use a year for productive use.

Current behavior: The migration stops with TypeError: initial_value must be str or None, not bytes

2019-07-01 11:49:20,151 3849 INFO postgres odoo.modules.migration: module project: Running migration [>12.0.1.1] pre-migration 2019-07-01 11:49:20,154 3849 INFO postgres OpenUpgrade: project: pre-migration script called with version 11.0.1.1 2019-07-01 11:49:20,175 3849 DEBUG postgres OpenUpgrade: Running ALTER TABLE project_project ADD COLUMN openupgrade_legacy_12_0_analytic_account_id integer; UPDATE project_project SET openupgrade_legacy_12_0_analytic_account_id=analytic_account_id;

2019-07-01 11:49:20,176 3849 DEBUG postgres OpenUpgrade: 12 rows affected 2019-07-01 11:49:20,180 3849 DEBUG postgres OpenUpgrade: Running ALTER TABLE project_task ADD COLUMN rating_last_value DOUBLE PRECISION 2019-07-01 11:49:20,180 3849 DEBUG postgres OpenUpgrade: -1 rows affected 2019-07-01 11:49:20,183 3849 DEBUG postgres OpenUpgrade: Running UPDATE project_task pt SET rating_last_value = rr.rating FROM rating_rating rr WHERE rr.res_model = 'project.task' AND rr.res_id = pt.id

2019-07-01 11:49:20,183 3849 DEBUG postgres OpenUpgrade: 0 rows affected 2019-07-01 11:49:20,398 3849 INFO postgres odoo.fields: Field stock.picking.move_ids_without_package depends on non-stored field stock.move.picking_type_entire_packs 2019-07-01 11:49:20,398 3849 INFO postgres odoo.fields: Field stock.picking.move_line_ids_without_package depends on non-stored field stock.move.line.picking_type_entire_packs 2019-07-01 11:49:20,408 3849 INFO postgres odoo.fields: Field stock.production.lot.product_uom_id depends on non-stored field product.product.uom_id 2019-07-01 11:49:20,491 3849 INFO postgres odoo.modules.registry: module project: creating or updating database tables 2019-07-01 11:49:20,552 3849 ERROR postgres odoo.sql_db: bad query: b'ALTER TABLE "project_project" ALTER COLUMN "name" SET NOT NULL' ERROR: column "name" contains null values

2019-07-01 11:49:20,553 3849 WARNING postgres odoo.schema: Table 'project_project': unable to set NOT NULL on column 'name'! If you want to have it, you should update the records and execute manually: ALTER TABLE "project_project" ALTER COLUMN "name" SET NOT NULL Traceback (most recent call last): File "/home/jole/Development/OpenUpgrade/odoo/tools/sql.py", line 109, in set_not_null cr.execute(query) File "/home/jole/Development/OpenUpgrade/odoo/sql_db.py", line 148, in wrapper return f(self, *args, **kwargs) File "/home/jole/Development/OpenUpgrade/odoo/sql_db.py", line 225, in execute res = self._obj.execute(query, params) psycopg2.IntegrityError: column "name" contains null values

2019-07-01 11:49:21,183 3849 INFO postgres odoo.models: Storing computed values of project.project.percentage_satisfaction_task 2019-07-01 11:49:21,184 3849 INFO postgres odoo.models: Storing computed values of project.project.percentage_satisfaction_project 2019-07-01 11:49:21,185 3849 INFO postgres odoo.models: Storing computed values of project.project.rating_request_deadline 2019-07-01 11:49:22,941 3849 INFO postgres odoo.models: Actual recompute of field ir.model.fields.related_field_id for 36 recs. 2019-07-01 11:49:22,951 3849 INFO postgres odoo.models: Actual recompute of field project.project.rating_request_deadline for 12 recs. 2019-07-01 11:49:22,985 3849 INFO postgres odoo.models: Actual recompute of field ir.model.fields.relation_field_id for 31 recs. 2019-07-01 11:49:22,996 3849 INFO postgres odoo.models: Actual recompute of field project.project.percentage_satisfaction_task for 12 recs. 2019-07-01 11:49:23,015 3849 INFO postgres odoo.models: Actual recompute of field project.project.percentage_satisfaction_project for 12 recs. 2019-07-01 11:49:23,058 3849 INFO postgres odoo.modules.loading: loading project/security/project_security.xml 2019-07-01 11:49:23,365 3849 INFO postgres odoo.models: Actual recompute of field res.users.share for 14 recs. 2019-07-01 11:49:23,412 3849 INFO postgres odoo.models: Actual recompute of field res.partner.partner_share for 14 recs. 2019-07-01 11:49:24,076 3849 INFO postgres odoo.modules.loading: loading project/security/ir.model.access.csv 2019-07-01 11:49:24,243 3849 INFO postgres odoo.modules.loading: loading project/report/project_report_views.xml 2019-07-01 11:49:24,385 3849 INFO postgres odoo.modules.loading: loading project/views/digest_views.xml 2019-07-01 11:49:24,483 3849 INFO postgres odoo.modules.loading: loading project/views/rating_views.xml 2019-07-01 11:49:24,576 3849 INFO postgres odoo.modules.loading: loading project/views/project_views.xml 2019-07-01 11:49:24,893 3849 INFO postgres odoo.models: Actual recompute of field ir.actions.server.model_name for 1 recs. 2019-07-01 11:49:24,971 3849 INFO postgres odoo.models.unlink: User #1 deleted ir.actions.act_window.view records with IDs: [20, 21] 2019-07-01 11:49:25,542 3849 INFO postgres odoo.modules.loading: loading project/views/res_partner_views.xml 2019-07-01 11:49:25,712 3849 INFO postgres odoo.modules.loading: loading project/views/res_config_settings_views.xml 2019-07-01 11:49:25,796 3849 INFO postgres odoo.addons.base.models.ir_ui_view: Element '' cannot be located in parent view

Error context: View res.config.settings.view.form.inherit.account.invoicing [view_id: 682, xml_id: account.res_config_settings_view_form_inherit_account_invoicing, model: res.config.settings, parent_id: 623] 2019-07-01 11:49:26,198 3849 INFO postgres odoo.modules.loading: loading project/views/mail_activity_views.xml 2019-07-01 11:49:26,223 3849 INFO postgres odoo.modules.loading: loading project/views/project_assets.xml 2019-07-01 11:49:26,564 3849 INFO postgres odoo.modules.loading: loading project/views/project_portal_templates.xml 2019-07-01 11:49:26,783 3849 INFO postgres odoo.modules.loading: loading project/views/project_rating_templates.xml 2019-07-01 11:49:26,928 3849 INFO postgres odoo.modules.loading: loading project/data/digest_data.xml 2019-07-01 11:49:26,943 3849 INFO postgres odoo.modules.loading: loading project/data/project_mail_template_data.xml 2019-07-01 11:49:26,960 3849 INFO postgres odoo.models: Actual recompute of field mail.template.model for 1 recs. 2019-07-01 11:49:26,981 3849 INFO postgres odoo.models: Actual recompute of field mail.template.model for 1 recs. 2019-07-01 11:49:26,985 3849 INFO postgres odoo.modules.loading: loading project/data/project_data.xml 2019-07-01 11:49:27,020 3849 INFO postgres odoo.models: Actual recompute of field ir.actions.server.model_name for 1 recs. 2019-07-01 11:49:27,041 3849 INFO postgres odoo.models: Actual recompute of field ir.cron.cron_name for 1 recs. 2019-07-01 11:49:27,171 3849 INFO postgres odoo.modules.migration: module project: Running migration [12.0.1.1>] post-migration 2019-07-01 11:49:27,172 3849 INFO postgres OpenUpgrade: project: post-migration script called with version 11.0.1.1 2019-07-01 11:49:27,177 3849 DEBUG postgres OpenUpgrade: Running UPDATE project_project pp SET name = aaa.name, partner_id = aaa.partner_id, company_id = aaa.company_id FROM account_analytic_account aaa WHERE pp.openupgrade_legacy_12_0_analytic_account_id = aaa.id

2019-07-01 11:49:27,177 3849 DEBUG postgres OpenUpgrade: 12 rows affected 2019-07-01 11:49:27,177 3849 INFO postgres OpenUpgrade: project: loading migrations/12.0.1.1/noupdate_changes.xml 2019-07-01 11:49:27,182 3849 ERROR postgres OpenUpgrade: project: error in migration script project/migrations/12.0.1.1/post-migration.py: initial_value must be str or None, not bytes 2019-07-01 11:49:27,182 3849 ERROR postgres OpenUpgrade: initial_value must be str or None, not bytes Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 1445, in wrapped_function if use_env2 else cr, version) File "/home/jole/Development/OpenUpgrade/addons/project/migrations/12.0.1.1/post-migration.py", line 24, in migrate mode='init_no_create') File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 291, in load_data for fp2 in _get_existing_records(cr, fp, module_name): File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 324, in yield_element if path else etree.Element(node.tag, node.attrib) File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 318, in yield_element yield StringIO(etree.tostring(path)) TypeError: initial_value must be str or None, not bytes 2019-07-01 11:49:27,183 3849 ERROR postgres odoo.modules.loading: Error executing post migration script for module project : initial_value must be str or None, not bytes 2019-07-01 11:49:27,200 3849 WARNING postgres odoo.modules.loading: Transient module states were reset 2019-07-01 11:49:27,204 3849 ERROR postgres odoo.modules.registry: Failed to load registry Traceback (most recent call last): File "/home/jole/Development/OpenUpgrade/odoo/modules/registry.py", line 86, in new odoo.modules.load_modules(registry._db, force_demo, status, update_module) File "/home/jole/Development/OpenUpgrade/odoo/modules/loading.py", line 476, in load_modules force, status, report, loaded_modules, update_module, models_to_check, upg_registry) File "/home/jole/Development/OpenUpgrade/odoo/modules/loading.py", line 360, in load_marked_modules upg_registry=upg_registry, File "/home/jole/Development/OpenUpgrade/odoo/modules/loading.py", line 257, in load_module_graph migrations.migrate_module(package, 'post') File "/home/jole/Development/OpenUpgrade/odoo/modules/migration.py", line 199, in migrate_module migrate(self.cr, installed_version) File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 1445, in wrapped_function if use_env2 else cr, version) File "/home/jole/Development/OpenUpgrade/addons/project/migrations/12.0.1.1/post-migration.py", line 24, in migrate mode='init_no_create') File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 291, in load_data for fp2 in _get_existing_records(cr, fp, module_name): File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 324, in yield_element if path else etree.Element(node.tag, node.attrib) File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 318, in yield_element yield StringIO(etree.tostring(path)) TypeError: initial_value must be str or None, not bytes 2019-07-01 11:49:27,207 3849 CRITICAL postgres odoo.service.server: Failed to initialize database postgres. Traceback (most recent call last): File "/home/jole/Development/OpenUpgrade/odoo/service/server.py", line 1116, in preload_registries registry = Registry.new(dbname, update_module=update_module) File "/home/jole/Development/OpenUpgrade/odoo/modules/registry.py", line 86, in new odoo.modules.load_modules(registry._db, force_demo, status, update_module) File "/home/jole/Development/OpenUpgrade/odoo/modules/loading.py", line 476, in load_modules force, status, report, loaded_modules, update_module, models_to_check, upg_registry) File "/home/jole/Development/OpenUpgrade/odoo/modules/loading.py", line 360, in load_marked_modules upg_registry=upg_registry, File "/home/jole/Development/OpenUpgrade/odoo/modules/loading.py", line 257, in load_module_graph migrations.migrate_module(package, 'post') File "/home/jole/Development/OpenUpgrade/odoo/modules/migration.py", line 199, in migrate_module migrate(self.cr, installed_version) File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 1445, in wrapped_function if use_env2 else cr, version) File "/home/jole/Development/OpenUpgrade/addons/project/migrations/12.0.1.1/post-migration.py", line 24, in migrate mode='init_no_create') File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 291, in load_data for fp2 in _get_existing_records(cr, fp, module_name): File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 324, in yield_element if path else etree.Element(node.tag, node.attrib) File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 318, in yield_element yield StringIO(etree.tostring(path)) TypeError: initial_value must be str or None, not bytes 2019-07-01 11:49:27,208 3849 INFO postgres odoo.service.server: Initiating shutdown 2019-07-01 11:49:27,210 3849 INFO postgres odoo.service.server: Hit CTRL-C again or send a second signal to force the shutdown.

Environment: $ python -V Python 2.7.12 $ python3 -V Python 3.5.2 $ cat /etc/issue Linux Mint 18.3 Sylvia \n \l

Database: Docker bitnami/postgresql:latest

Expected behavior: Project to be migrated in version 12.0

jolentes commented 5 years ago

Any idea what the problem is? What are further steps to analyze the issue?

pedrobaeza commented 5 years ago

Have you got latest openupgradelib from source (not from pypi) and latest OpenUpgrade sources?

jolentes commented 5 years ago

I just got back to work after vacation updated via git pull and still experience the same problem.

pedrobaeza commented 5 years ago

An that's because nothing has changed :wink:

And nothing should change except the data in that DB. There seems to be some missing data that provokes the problem. A previous error about a not null not being able to be setup can give you a hint. Any way, it seems something in your side.

Closing until further information that gives a trace of a possible problem in the script is provided.

jolentes commented 5 years ago

I still cannot get this sorted out. I tried to find out which data in the DB needs to be corrected but without luck so far.

The pre-migration.py script complains about not being able to set the column:

2019-09-16 12:26:02,673 14203 INFO odoo_ams odoo.modules.migration: module project: Running migration [>12.0.1.1] pre-migration 
2019-09-16 12:26:02,675 14203 INFO odoo_ams OpenUpgrade: project: pre-migration script called with version 11.0.1.1 
2019-09-16 12:26:02,694 14203 DEBUG odoo_ams OpenUpgrade: Running 
                ALTER TABLE project_project
                ADD COLUMN openupgrade_legacy_12_0_analytic_account_id integer;
                UPDATE project_project SET openupgrade_legacy_12_0_analytic_account_id=analytic_account_id;

2019-09-16 12:26:02,694 14203 DEBUG odoo_ams OpenUpgrade: 18 rows affected 
2019-09-16 12:26:02,696 14203 DEBUG odoo_ams OpenUpgrade: Running 
            ALTER TABLE project_task
            ADD COLUMN rating_last_value DOUBLE PRECISION 
2019-09-16 12:26:02,696 14203 DEBUG odoo_ams OpenUpgrade: -1 rows affected 
2019-09-16 12:26:02,700 14203 DEBUG odoo_ams OpenUpgrade: Running UPDATE project_task pt
            SET rating_last_value = rr.rating
            FROM rating_rating rr
            WHERE rr.res_model = 'project.task' AND rr.res_id = pt.id

2019-09-16 12:26:02,700 14203 DEBUG odoo_ams OpenUpgrade: 0 rows affected 
2019-09-16 12:26:02,946 14203 INFO odoo_ams odoo.fields: Field stock.production.lot.product_uom_id depends on non-stored field product.product.uom_id 
2019-09-16 12:26:02,989 14203 INFO odoo_ams odoo.fields: Field stock.picking.move_ids_without_package depends on non-stored field stock.move.picking_type_entire_packs 
2019-09-16 12:26:02,989 14203 INFO odoo_ams odoo.fields: Field stock.picking.move_line_ids_without_package depends on non-stored field stock.move.line.picking_type_entire_packs 
2019-09-16 12:26:03,096 14203 INFO odoo_ams odoo.modules.registry: module project: creating or updating database tables 
2019-09-16 12:26:03,158 14203 ERROR odoo_ams odoo.sql_db: bad query: b'ALTER TABLE "project_project" ALTER COLUMN "name" SET NOT NULL'
ERROR: column "name" contains null values

2019-09-16 12:26:03,159 14203 WARNING odoo_ams odoo.schema: Table 'project_project': unable to set NOT NULL on column 'name'!
If you want to have it, you should update the records and execute manually:
ALTER TABLE "project_project" ALTER COLUMN "name" SET NOT NULL 
Traceback (most recent call last):
  File "/home/jole/Development/OpenUpgrade/odoo/tools/sql.py", line 109, in set_not_null
    cr.execute(query)
  File "/home/jole/Development/OpenUpgrade/odoo/sql_db.py", line 148, in wrapper
    return f(self, *args, **kwargs)
  File "/home/jole/Development/OpenUpgrade/odoo/sql_db.py", line 225, in execute
    res = self._obj.execute(query, params)
psycopg2.IntegrityError: column "name" contains null values

This column does not exist before the migration. It was located in account_analytic_account before. It is filled by a query in the post-migration.py.

2019-09-16 12:26:10,410 14203 INFO odoo_ams odoo.modules.migration: module project: Running migration [12.0.1.1>] post-migration 
2019-09-16 12:26:10,412 14203 INFO odoo_ams OpenUpgrade: project: post-migration script called with version 11.0.1.1 
2019-09-16 12:26:10,419 14203 DEBUG odoo_ams OpenUpgrade: Running UPDATE project_project pp
        SET name = aaa.name, partner_id = aaa.partner_id,
            company_id = aaa.company_id
        FROM account_analytic_account aaa
        WHERE pp.openupgrade_legacy_12_0_analytic_account_id = aaa.id

2019-09-16 12:26:10,419 14203 DEBUG odoo_ams OpenUpgrade: 18 rows affected 
2019-09-16 12:26:10,419 14203 INFO odoo_ams OpenUpgrade: project: loading migrations/12.0.1.1/noupdate_changes.xml 
2019-09-16 12:26:10,421 14203 ERROR odoo_ams OpenUpgrade: project: error in migration script project/migrations/12.0.1.1/post-migration.py: initial_value must be str or None, not bytes 
2019-09-16 12:26:10,421 14203 ERROR odoo_ams OpenUpgrade: initial_value must be str or None, not bytes

I don't understand how it should be set to "NOT NULL" before the column is filled. What do I miss? I am still trying to understand how the migration works.

Am I looking for the wrong stuff? Can you give me some pointers?

pedrobaeza commented 5 years ago

The complaint about not null was a non critical thing, but I have solved it in #2019. Check with that code.

jolentes commented 5 years ago

Now the error changed. 

2019-09-16 14:31:39,325 7639 ERROR odoo_ams OpenUpgrade: project: error in migration script project/migrations/12.0.1.1/pre-migration.py: unsupported operand type(s) for %: 'Composed' and 'tuple' 
2019-09-16 14:31:39,325 7639 ERROR odoo_ams OpenUpgrade: unsupported operand type(s) for %: 'Composed' and 'tuple' 
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 1445, in wrapped_function
    if use_env2 else cr, version)
  File "/home/jole/Development/OpenUpgrade/addons/project/migrations/12.0.1.1/pre-migration.py", line 56, in migrate
    fill_project_project_inherits_values(env)
  File "/home/jole/Development/OpenUpgrade/addons/project/migrations/12.0.1.1/pre-migration.py", line 49, in fill_project_project_inherits_values
    sql.Identifier('analytic_account_id'),
  File "/usr/local/lib/python3.5/dist-packages/openupgradelib/openupgrade.py", line 934, in logged_query
    logger.debug('Running %s', query % args)
TypeError: unsupported operand type(s) for %: 'Composed' and 'tuple'
pedrobaeza commented 5 years ago

You definitively have something wrong on your side, as this is something outside OpenUpgrade. Maybe you have an outdated Postgres version or similar. This is working perfectly in Travis tests: https://travis-ci.org/OCA/OpenUpgrade/builds/585304671#L4431

jolentes commented 5 years ago

I did a clean re-setup of the environment and the migration worked. Thanks for your patience and the great work.