Upgrade to CKAN 2.10

[jbrown-xentity]

Turning this into an epic

User Story

In order to utilize latest CKAN patches, data.gov admins want CKAN updated to 2.10.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• [x] GIVEN catalog is upgraded \ WHEN I go to https://catalog.data.gov/api/action/status_show \ THEN ckan version shows 2.10.0 (or equivalent)

Background

https://github.com/GSA/catalog.data.gov/blob/main/ckan/requirements.txt#L11 https://github.com/GSA/data.gov/issues/4113 See email on CKAN mailing list:

Hi all,

If all goes according to plan, next Wednesday 15th February around 13:00 UTC (14:00 CET, 08:00 EST) we will be releasing the next stable version of CKAN, 2.10, as well as patch releases for the 2.9 line (2.9.8).

Users are strongly encouraged to always run on the latest patch release for their version, as these include important security and
stability fixes. The latest patch release is the only one supported by the CKAN team (patch releases don't contain backwards incompatible changes).

More information about CKAN releases can be found here:

http://docs.ckan.org/en/latest/maintaining/upgrading/index.html#ckan-releases

Best,

Adrià 

Security Considerations (required)

None

Sketch

This will be a major lift, and will require picking up the work again here: https://github.com/GSA/data.gov/issues/4113

Ideally every extension should test against 2.10, and verify things work. If they don't work, a new issue may be created to investigate further. Final step will be upgrading catalog, and may require docker changes (among others).

[nickumia-reisys]

Tenative sketch:

• [x] Add `"2.10" to the ckan-test.yml template.
• [x] Test all of our extensions
  • [x] CKAN 2.10 Support - ckanext-datajson
  • [x] CKAN 2.10 Support - ckanext-datagovcatalog
  • [x] CKAN 2.10 Support - ckanext-datagovtheme
  • [x] CKAN 2.10 Support - ckanext-geodatagov
  • [x] CKAN 2.10 Support - ckanext-googleanalyticsbasic
  • [x] CKAN 2.10 Support - ckanext-metrics-dashboard
  • [x] CKAN 2.10 Support - ckanext-usmetadata
  • [x] CKAN 2.10 Support - ckanext-dcat_usmetadata
• [ ] Test all upstream extensions
  • [x] ckanext-envvars
  • [x] ckanext-s3filestore (inventory)
  • [x] ckanext-xloader
  • [x] ckanext-saml2auth
  • [x] ckanext-harvest
  • [x] ckanext-dcat
  • [x] ckanext-spatial
  • [ ] ckanext-qa (catalog)
  • [x] ckanext-report
  • [x] ckanext-archiver
• [x] Make an issue for each extension that takes more than 1-2 hours to get working with CKAN 2.10
• [x] Upgrade and get all tests passing
• [x] Test main applications
  • [x] catalog
  • [x] inventory
• [x] Perform manual user inspection on development

[nickumia-reisys]

General notes on CKAN 2.10 Upgrade:

• CKAN 2.10 supports Python 3.7 to 3.10
• This version requires a requirements upgrade on source installations
• This version requires a database upgrade
• This version does not require a Solr schema upgrade if you are already using the 2.9 schema, but it is recommended to upgrade to the 2.10 Solr schema.
• Make sure to check the Migration notes

[nickumia-reisys]

Current status:

• All data.gov extensions (except geodatagov and dcat_usmetadata) have been minimally upgraded.
• There is work to do to review each and every template to consolidate changed from upstream and our customizations.
• https://github.com/GSA/data.gov/issues/4285
• https://github.com/GSA/data.gov/issues/4293
• The database migration is relatively minor. No data.gov-specific breaking elements.
  • 101_d111f446733b_add_last_active_column_in_user_table.py
  • 102_ff13667243ed_create_conditinal_index_on_user.py
  • 103_353aaf2701f0_add_plugin_data_to_package_table.py
  • 104_9f33a0280c51_resource_view_resource_id_index.py
• geodatagov is proving to be quite a headache. Will continue to make issues for results from investigations (we don't know why some things are broken, hence the headaches..)

[nickumia-reisys]

Proof of upstream extensions:

• saml2auth: https://github.com/keitaroinc/ckanext-saml2auth/actions/runs/4187125230/jobs/7256548158
• xloader: https://github.com/ckan/ckanext-xloader/actions/runs/4607581034/jobs/8142285074
• envvars: https://github.com/okfn/ckanext-envvars/actions/runs/4545468306/jobs/8012854002
• harvest: https://github.com/ckan/ckanext-harvest/actions/runs/4598848073/jobs/8123463947
• dcat: https://github.com/ckan/ckanext-dcat/actions/runs/4869207000/jobs/8683456177
• spatial: https://github.com/ckan/ckanext-spatial/actions/runs/4666446086/jobs/8261053880
• qa: https://github.com/ckan/ckanext-qa/pull/73
• report: https://github.com/ckan/ckanext-report/actions/runs/4112223762/jobs/7096894210
• archiver: https://github.com/ckan/ckanext-archiver/actions/runs/4530177313/jobs/7978779426

[nickumia-reisys]

CKAN 2.10.1 released EIGHT HOURS AGO!!

• Verified that is has the license_list fix

[nickumia-reisys]

Current issue being investigated with the /harvest/new route:

ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 199, in after_dataset_show
ckan_1                  |     source = HarvestSource.get(data_dict["id"])
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/model/__init__.py", line 121, in get
ckan_1                  |     o = cls.filter(**kwds).first()
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/model/__init__.py", line 130, in filter
ckan_1                  |     return query.filter_by(**kwds)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1806, in filter_by
ckan_1                  |     clauses = [
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1807, in <listcomp>
ckan_1                  |     _entity_namespace_key(from_entity, key) == value
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 366, in __eq__
ckan_1                  |     return self.operate(eq, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 317, in operate
ckan_1                  |     return op(self.comparator, *other, **kwargs)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 366, in __eq__
ckan_1                  |     return self.operate(eq, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/properties.py", line 423, in operate
ckan_1                  |     return op(self.__clause_element__(), *other, **kwargs)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/annotation.py", line 221, in __eq__
ckan_1                  |     return self.__element.__class__.__eq__(self, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 366, in __eq__
ckan_1                  |     return self.operate(eq, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 869, in operate
ckan_1                  |     return op(self.comparator, *other, **kwargs)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 366, in __eq__
ckan_1                  |     return self.operate(eq, other)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py", line 77, in operate
ckan_1                  |     return o[0](self.expr, op, *(other + o[1:]), **kwargs)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/default_comparator.py", line 101, in _boolean_compare
ckan_1                  |     obj = coercions.expect(
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py", line 188, in expect
ckan_1                  |     resolved = impl._literal_coercion(
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py", line 523, in _literal_coercion
ckan_1                  |     return expr._bind_param(operator, element, type_=bindparam_type)
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 4708, in _bind_param
ckan_1                  |     return BindParameter(
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 1512, in __init__
ckan_1                  |     self.key = _anonymous_label.safe_construct(
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 5375, in safe_construct
ckan_1                  |     body = re.sub(r"[%\(\) \$]+", "_", body).strip("_")
ckan_1                  |   File "/usr/local/lib/python3.8/re.py", line 210, in sub
ckan_1                  |     return _compile(pattern, flags).sub(repl, string, count)
ckan_1                  |   File "/usr/local/lib/python3.8/re.py", line 291, in _compile
ckan_1                  |     if isinstance(flags, RegexFlag):
ckan_1                  | RecursionError: maximum recursion depth exceeded while calling a Python object
ckan_1                  | Fatal Python error: Cannot recover from stack overflow.
ckan_1                  | Python runtime state: initialized
ckan_1                  |
ckan_1                  | Current thread 0x00007f5be02ec740 (most recent call first):
ckan_1                  |   File "/usr/local/lib/python3.8/uuid.py", line 782 in uuid4
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/types.py", line 17 in make_uuid
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 2776 in <lambda>
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1735 in _exec_default
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1871 in get_insert_default
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py", line 3226 in get_insert_default
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1927 in _process_executesingle_defaults
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1025 in _init_compiled
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1800 in _execute_context
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572 in _execute_clauseelement
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 333 in _execute_on_connection
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705 in _execute_20
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1238 in _emit_insert_statements
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 245 in save_obj
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 630 in execute
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 456 in execute
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3486 in _flush
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3386 in flush
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 28 in notify_observers
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 25 in before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/meta.py", line 70 in ckan_before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 247 in __call__
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 797 in _prepare_impl
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1451 in commit
ckan_1                  |   File "<string>", line 2 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 152 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 136 in save
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/log.py", line 15 in emit
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 954 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1475 in error
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 201 in after_dataset_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/action/get.py", line 1090 in package_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/__init__.py", line 551 in wrapped
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/lib/search/__init__.py", line 167 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 71 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 39 in notify_observers
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 25 in before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/meta.py", line 70 in ckan_before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 247 in __call__
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 797 in _prepare_impl
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1451 in commit
ckan_1                  |   File "<string>", line 2 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 152 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 136 in save
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/log.py", line 15 in emit
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 954 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1475 in error
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 201 in after_dataset_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/action/get.py", line 1090 in package_show
db_1                    | 2023-06-01 12:34:19.208 UTC [226] LOG:  unexpected EOF on client connection with an open transaction
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/__init__.py", line 551 in wrapped
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/lib/search/__init__.py", line 167 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 71 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 39 in notify_observers
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 25 in before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/meta.py", line 70 in ckan_before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 247 in __call__
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 797 in _prepare_impl
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1451 in commit
ckan_1                  |   File "<string>", line 2 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 152 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 136 in save
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/log.py", line 15 in emit
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 954 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1475 in error
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 201 in after_dataset_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/action/get.py", line 1090 in package_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/__init__.py", line 551 in wrapped
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/lib/search/__init__.py", line 167 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 71 in notify
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 65 in notify_observers
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/modification.py", line 25 in before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/meta.py", line 70 in ckan_before_commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 247 in __call__
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 797 in _prepare_impl
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1451 in commit
ckan_1                  |   File "<string>", line 2 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 152 in commit
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/model/base.py", line 136 in save
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/log.py", line 15 in emit
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 954 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
ckan_1                  |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1475 in error
ckan_1                  |   File "/src/ckanext-harvest/ckanext/harvest/plugin.py", line 201 in after_dataset_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/action/get.py", line 1090 in package_show
ckan_1                  |   File "/usr/local/lib/python3.8/site-packages/ckan/logic/__init__.py", line 551 in wrapped
ckan_1                  |   ...
ckan_1                  | [2023-06-01 12:34:19 +0000] [1] [WARNING] Worker with pid 496 was terminated due to signal 6

[nickumia-reisys]

Catalog and Inventory successfully running CKAN 2.10 now.