Upgrade Django to 4.2.11

[jsjiang]

Upgrade Django

• to address security issues
• to keep software up to date

Tasks:

• [x] Review release notes
• [x] Decide Django version
• [x] Make code change
• [x] API Tests (verify_ezid_after_patching.py)
• [x] UI Tests
• [x] Test background jobs
• [x] Test management commands
• [x] Test emailing functionality
• [x] Perfromance tests
• [x] Make migration files
• [x] Apply migration files
• [x] Deploy and test on ezid-stg
• [x] Deploy on ezid-prd
• [x] Consolidate migration files after production deployment

[jsjiang]

Django security releases issued: 4.2.7, 4.1.13, and 3.2.23 - severity "moderate" Django security releases issued: 5.0.3, 4.2.11, and 3.2.25 - severity "moderate"

[jsjiang]

Our current Django version 3.2.7 has reached the end of extended support The latest official version is 5.0.4 (as of today April 19, 2024). Version 4.2.11 (in the 4.2 Long Term Support series) might be the better solution.

Django releases

Release Series	Latest Release	End of mainstream support	End of extended support
5.0	5.04	August 2024	April, 2025
4.2 LTS	4.2.11	December 4, 2023	April, 2026
3.2 LTS	3.2.25	December 7, 2021	April 1, 2024

[jsjiang]

Django 3.2 release notes Django 4.2 release notes

• Django 3.2 supports Python 3.6, 3.7, 3.8, 3.9, and 3.10
• Django 4.2 supports Python 3.8, 3.9, 3.10, 3.11, and 3.12

As EZID current Python version is 3.8.5, we should consider upgrade Django to 4.2.11 first, then upgrade Python to 3.11 or 3.12.

[jsjiang]

Django 4.0 release notes

[jsjiang]

Upgrade checklist: review release notes, make code change accordingly, test EZID

• [x] Backwards incompatible changes in 4.0
• [x] Features removed in 4.0
• [x] Backwards incompatible changes in 4.1
• [x] Features removed in 4.1
• [x] Backwards incompatible changes in 4.2
• [x] Features removed in 4.2

[jsjiang]

Deprecated/Removed features: 1. is_safe_url()

• deprecated in 3.0 To avoid possible confusion as to effective scope, the private internal utility is_safe_url() is renamed to url_has_allowed_host_and_scheme().
• removed in 4.0

2. ugettext()

• deprecated in 3.0 aliases for: django.utils.translation.gettext()
• removed in 4.0

3. django.utils.http.urlquote()

• deprecated in 3.0 django.utils.http.urlquote(), urlquote_plus(), urlunquote(), and urlunquote_plus() are deprecated in favor of the functions that they’re aliases for: urllib.parse.quote(), quote_plus(), unquote(), and unquote_plus().

• removed in 4.0

[jsjiang]

4. index_together option is deprecated in favor of indexes in 4.2

Running the makemigrations command will generate a migration containing a RenameIndex operation which will rename the existing index. Next, consider squashing migrations to remove index_together from historical migrations.

[jsjiang]

Failed creating new user and group in UI on Dev. Error message:

Forbidden (403), CSRF verification failed. Request aborted.

2024-04-24 15:52:42,188  WARNING Forbidden (Origin checking failed - https://ezid-dev.cdlib.org does not match any trusted origins.): /admin/ezidapp/user/add/

Solution: add trusted origins for unsafe requests to variable CSRF_TRUSTED_ORIGINS to the settings file settings.py.j2.

ref: https://docs.djangoproject.com/en/4.0/ref/settings/#csrf-trusted-origins

5. add trusted origins for unsafe requests to variable CSRF_TRUSTED_ORIGINS

CSRF_TRUSTED_ORIGINS = ['https://*.cdlib.org']

[jsjiang]

Make migration files

python manage.py makemigrations ezidapp
created new migration file: 0005_rename_linkchecker_owner_id_isBad_lastCheckTime_ezidapp_lin_owner_i_866e15_idx_and_more.py
renamed file to: 0005_rename_index

(ezid-py38) CDL-jjiang-9m:ezid jjiang$ python manage.py squashmigrations ezidapp 0005
Will squash the following migrations:
- 0001_initial
- 0002_auto_20221026_1139
- 0003_auto_20230809_1154
- 0004_minter
- 0005_rename_index
Do you wish to proceed? [yN] y
Optimizing...
  Optimized from 65 operations to 25 operations.
Created new squashed migration /Users/jjiang/my_dev_space/ezid-dev/ezid/ezidapp/migrations/0001_squashed_0005_rename_index.py
  You should commit this migration but leave the old ones in place;
  the new migration will be used for new installs. Once you are sure
  all instances of the codebase have applied the migrations you squashed,
  you can delete them.

Apply migration files:

ezid@uc3-ezidui-dev01:17:16:18:~/ezid$ python manage.py migrate
System check identified some issues:

WARNINGS:
?: (mysql.W002) MySQL Strict Mode is not set for database connection 'default'
       HINT: MySQL's Strict Mode fixes many data integrity problems in MySQL, such as data truncation upon insertion, by escalating warnings into errors. It is strongly recommended you activate it. See: https://docs.djangoproject.com/en/4.2/ref/databases/#mysql-sql-mode
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, ezidapp, sessions
Running migrations:
  Applying ezidapp.0005_rename_index... OK

[jsjiang]

6. Added USE_TZ to the settings The default value of USE_TZ will change from False to True in Django 5.0. Set USE_TZ to False in your project settings if you want to keep the current default behavior.

USE_TZ = False

[jsjiang]

7. The HttpRequest.is_ajax() method

• removed in 4.0
• deprecated 3.1

The HttpRequest.is_ajax() method is deprecated as it relied on a jQuery-specific way of signifying AJAX calls, while current usage tends to use the JavaScript Fetch API. Depending on your use case, you can either write your own AJAX detection method, or use the new HttpRequest.accepts() method if your code depends on the client Accept HTTP header.

If you are writing your own AJAX detection method, request.is_ajax() can be reproduced exactly as request.headers.get('x-requested-with') == 'XMLHttpRequest'.

Note: The ajax_dashboard_table() function in ui_admin.py uses is_ajax():

def ajax_dashboard_table(request):
    # noinspection PyDictCreation
    if request.is_ajax():
        user = impl.userauth.getUser(request)
        G = request.GET
        d = {
            'owner_selected': (G['owner_selected'] if 'owner_selected' in G else user.username),
            'p': G.get('p'),
        }
        if 'name' in G and d['p'] is not None and d['p'].isdigit():
            d['ajax'] = True
            d['s_type'] = G['name']
            d = impl.ui_search.search(d, request, NO_CONSTRAINTS, G['name'])
            return impl.ui_common.render(request, "dashboard/_" + G['name'], d)

Find out how to test the ajax_dashboard_table() function.

• Go to the dashboard page
• Perform page navigation for the "Identifier Issues" will trigger loading the dashboard/ajax_table? page which calls the ajax_dashboard_table() function.

javascript in _dashboard_js.html page

<script type="text/javascript">
$(document).ready(function() {
  $('#p_{{name}}').on('submit', function(e){
      e.preventDefault();
      var newpage = $('#page-directselect-{{name}}').val()
      $('#ajax_{{name}}').html('&nbsp;').load('dashboard/ajax_table?owner_selected={{owner_selected}}&name={{name}}&p=' + newpage, function( response, status, xhr ) {
          if ( status == "error" ) {
              $('#error_{{name}}').html("<div class='alert alert-error' role='alert'>" +
                "<div class='alert-text'>Oops! We have encountered an error: " +
                xhr.status + " " + xhr.statusText ); 
          } 
      });
  });

Corresponding code in view-source:https://ezid-stg.cdlib.org/dashboard:

<script type="text/javascript">
  $(document).ready(function() {
  $('#p_issues').on('submit', function(e){
  e.preventDefault();
  var newpage = $('#page-directselect-issues').val()
  $('#ajax_issues').html('&nbsp;').load('dashboard/ajax_table?owner_selected=user_eschol_harvester&name=issues&p=' + newpage, function( response, status, xhr ) {
  if ( status == "error" ) {
  $('#error_issues').html("<div class='alert alert-error' role='alert'>" +
  "<div class='alert-text'>Oops! We have encountered an error: " +
  xhr.status + " " + xhr.statusText );
  }
  });
  });

Log entry:

172.31.57.58 - - [01/May/2024:15:36:27 -0700] "GET /dashboard/ajax_table?owner_selected=user_eschol_harvester&name=issues&p=8 HTTP/1.1" 200 6633 "https://ezid-stg.cdlib.org/dashboard" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

[jsjiang]

• Got page navigation error and following error message before replacing request.is_ajax:

  2024-05-01 16:12:30,944    ERROR Internal Server Error: /dashboard/ajax_table
  File "/ezid/ezid/impl/ui_admin.py", line 81, in ajax_dashboard_table
  if request.is_ajax():
  AttributeError: 'WSGIRequest' object has no attribute 'is_ajax'

• Navigation error was resolved after replacing request.is_ajax with request.headers.get('x-requested-with') == 'XMLHttpRequest'.

[jsjiang]

Recent 4.2 new releases:

• Django 4.2.12 release notes¶

  • May 6, 2024
  • Django 4.2.12 fixes a compatibility issue with Python 3.11.9+ and 3.12.3+.

• Django 4.2.13 release notes¶

  • May 7, 2024
  • Django 4.2.13 fixes a packaging error in 4.2.12.

[jsjiang]

Test notes:

• This upgrade fixed an existing EZID bug: the radio buttons for title types "Main title/Alternative title/Subtitle/Translated title/Other" were not clickable/selectable. They are clickable now. The radio buttons for the additional titles (added by clicking on the plus + icon) also work as expected now. Selected values are saved properly and can be edited using the edit form.
• Crossref registration form: still not working (known issue)
• run the python manage.py check-ezid --test-level 3 command to cover minter and shoulder related commands (create, update, delete)

[jsjiang]

Deployment procedure:

• Note the ezidapp_linkchecker and ezidapp_searchidentifier table status
  • total rows:
  • size:
  • index names
• Run the uc3_pupapply.sh --exec command to deploy the code change
• Update thesettings/settings.py file to use the eziddba account for database schema change (rename indexes)
• Stop ezid
• run the migration command
• restore the settings.py file
• restart ezid

ezid@uc3-ezidui-stg01:13:20:46:~/ezid$ sudo cdlsysctl stop ezid
ezid@uc3-ezidui-stg01:13:21:00:~/ezid$ python manage.py migrate
System check identified some issues:

WARNINGS:
?: (mysql.W002) MySQL Strict Mode is not set for database connection 'default'
    HINT: MySQL's Strict Mode fixes many data integrity problems in MySQL, such as data truncation upon insertion, by escalating warnings into errors. It is strongly recommended you activate it. See: https://docs.djangoproject.com/en/4.2/ref/databases/#mysql-sql-mode
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, ezidapp, sessions
Running migrations:
  Applying ezidapp.0005_rename_index... OK
ezid@uc3-ezidui-stg01:13:21:25:~/ezid$ sudo cdlsysctl start ezid

[jsjiang]

Regarding database index name changes:

• This Django upgrade will result in index name change
• The changes are captured in the migration file 0005_rename_index.py produced by the makemigration command
• two tables are affected: linkchecker and searchidentifier
• new index names are listed below
• applying index name change does not require downtime.

ezid@uc3-ezidui-stg01:13:31:18:~/ezid/ezidapp/migrations$ more 0005_rename_index.py | grep model_name | sort -u
            model_name='linkchecker',
            model_name='searchidentifier',
ezid@uc3-ezidui-stg01:13:35:19:~/ezid/ezidapp/migrations$ more 0005_rename_index.py | grep new_name
            new_name='ezidapp_lin_owner_i_866e15_idx',
            new_name='ezidapp_sea_ownergr_c3cf9a_idx',
            new_name='ezidapp_sea_owner_i_f5f078_idx',
            new_name='ezidapp_sea_owner_i_e25844_idx',
            new_name='ezidapp_sea_owner_i_155d5f_idx',
            new_name='ezidapp_sea_ownergr_89f1d2_idx',
            new_name='ezidapp_sea_owner_i_b09cbd_idx',
            new_name='ezidapp_sea_owner_i_1c8373_idx',
            new_name='ezidapp_sea_owner_i_f36098_idx',
            new_name='ezidapp_sea_ownergr_b5c787_idx',
            new_name='ezidapp_sea_publicS_76c2e5_idx',
            new_name='ezidapp_sea_owner_i_c4d7aa_idx',
            new_name='ezidapp_sea_publicS_44eb10_idx',
            new_name='ezidapp_sea_owner_i_845796_idx',
            new_name='ezidapp_sea_ownergr_3808dc_idx',
            new_name='ezidapp_sea_owner_i_940df1_idx',
            new_name='ezidapp_sea_oaiVisi_95a198_idx',
            new_name='ezidapp_sea_owner_i_a9c320_idx',
            new_name='ezidapp_sea_ownergr_36ea7c_idx',
            new_name='ezidapp_sea_ownergr_df06b0_idx',
            new_name='ezidapp_sea_publicS_a5777f_idx',
            new_name='ezidapp_sea_ownergr_04c9f2_idx',
            new_name='ezidapp_sea_owner_i_59a541_idx',
            new_name='ezidapp_sea_ownergr_19a761_idx',
            new_name='ezidapp_sea_ownergr_165cbb_idx',
            new_name='ezidapp_sea_owner_i_cfdb17_idx',
            new_name='ezidapp_sea_owner_i_f854de_idx',
            new_name='ezidapp_sea_ownergr_de548b_idx',
            new_name='ezidapp_sea_publicS_edadba_idx',

[jsjiang]

EZID-PRD DB table status before deployment:

SHOW TABLE STATUS LIKE 'ezidapp_linkchecker';

show index from ezidapp_linkchecker; (7 indexes)

SHOW TABLE STATUS LIKE 'ezidapp_searchidentifier';

show index from ezidapp_searchidentifier; (69 indexes)

ezid-prd_linkchecker-table-info.before.csv

ezid-prd_linkchecker-indexes.before.csv

ezid-prd_searchidentifier-table-info.before.csv

ezid-prd_searchidentifier-indexes.before.csv

ezid-prd_linkchecker-table-info.after.csv

ezid-prd_linkchecker-indexes.after.csv

ezid-prd_searchidentifier-table-info.after.csv

ezid-prd_searchidentifier-indexes.after.csv

Note: attached are TSV files. Since github does not allow upload .tsv files, the file extensions were changed to .csv as a workaround. Change extension to .tsv when opening the file using Excel or other tsv file editors.

[jsjiang]

• Consolidate migration files into 0001_squashed_0005_rename_index.py
• Saved old migration files in the ezidapp/migrations/squashed_migrations folder

To do:

• create a new release tag v3.2.8 - done
• deploy new release on Tuesday May 21 - done