Unit tests on latest signbank (Related to Senses)

[susanodd]

One test fails:

FAIL: test_crud_Senses (signbank.dictionary.tests.SensesCRUDTests.test_crud_Senses)

The test creates multiple senses (using a form), then does a search on "Has Multiple Senses". But it finds nothing instead of the gloss it just added the senses to. (See output when running this test.)

[susanodd]

I didn't get this error on signbank-dev after I got it working again.

I'm wondering if this could be a setting with the utf-8 or return characters or something? (What would be different on the Apache server versus PyCharm that involves "text" in a form in the Django interface?)

See if it works on your local server.

[susanodd]

This seems to have resolved itself.

[susanodd]

This is the error now when the deploy script wants to create a test database.

django.db.utils.IntegrityError: The row in table 'dictionary_translation' with primary key '8' has an invalid foreign key: dictionary_translation.translation_id contains a value '7' that does not have a corresponding value in dictionary_keyword.id.

[susanodd]

These are the senses with keywords that are missing on global (this is run on a local database that is older):

keyword with key 8:  <QuerySet [<Keyword: verzekering>]>
keyword with key 7:  <QuerySet [<Keyword: laatste>]>

Missing keywords found for dataset  NGT
LAATSTE-B (laatste)
LAATSTE-A (laatste)
LAATSTE-C (laatste)
VERZEKERING (verzekering)
VERZEKERING-B (verzekering)
VERZEKERING-C (verzekering)
LAAT-B (laatste)

[susanodd]

It's this gloss it's complaining about:

https://signbank.cls.ru.nl/dictionary/gloss/6/

translation with pk 8:  {'_state': <django.db.models.base.ModelState object at 0x7f985fc62990>, 'id': 8, 'gloss_id': 6, 'language_id': 2, 'translation_id': 7, 'index': 1, 'orderIndex': 1}
keyword with key 7:  {'_state': <django.db.models.base.ModelState object at 0x7f985fc62900>, 'id': 7, 'text': 'laatste'}

[susanodd]

@Jetske here's another version:

        lookup_translation_8 = Translation.objects.filter(pk=8).first()
        print('translation with pk 8: ', lookup_translation_8.__dict__)
        lookup_keyword_7 = Keyword.objects.filter(pk=7).first()
        print('keyword with pk 7: ', lookup_keyword_7.__dict__)
        translations_refer_to_keyword = SenseTranslation.objects.filter(
            translations__translation_id__in=[8])
        print('SenseTranslation has translation that refers to translation with pk 8: ',
              translations_refer_to_keyword)

Output:

translation with pk 8:  {'_state': <django.db.models.base.ModelState object at 0x7f6f1ad827e0>, 'id': 8, 'gloss_id': 6, 'language_id': 2, 'translation_id': 7, 'index': 1, 'orderIndex': 1}
keyword with pk 7:  {'_state': <django.db.models.base.ModelState object at 0x7f6f1ad82780>, 'id': 7, 'text': 'laatste'}
SenseTranslation has translation that refers to translation with pk 8:  <QuerySet [<SenseTranslation: verzekering>, <SenseTranslation: verzekering>, <SenseTranslation: verzekering>]>

[susanodd]

Then adding to that:

        senses_that_refer_to_sensetranslations = Sense.objects.filter(
            senseTranslations__in=translations_refer_to_keyword)
        print(senses_that_refer_to_sensetranslations)

<QuerySet [<Sense: verzekering>, <Sense: verzekering>, <Sense: insurance | verzekering>]>

[susanodd]

I don't see how those can be referring to "laatste".

[susanodd]

Maybe the language that is empty has somehow caught the wrong translation id? Maybe it used to be empty. Or a bound variable inside a loop that still refers to something instead of nothing?

[susanodd]

[Jetske]

I think it should be something like sensetranslations_refer_to_keyword = SenseTranslation.objects.filter(translations__translation_id__in=[7]) to get the SenseTranslations that contain a translation for which a keyword has id 7

or otherwise sensetranslations_refer_to_translation = SenseTranslation.objects.filter(translations__pk=8) to get the SenseTranslations that contain a translation with pk 8

then it shouldn't find 'verzekering'

[susanodd]

Okay, I'll try those next.

On my walk home I started wondering if some of the really old glosses (it found gloss ID 6!) If it's possible they have ints instead of foreign keys stored in them.

Before we had Field Choice objects, there were literally character string representations of the "machine value" stored.

It's possible that when the original migration from Translation Keywords to Senses was done that they were just copied to the same "int" instead of a foreign key to the object.

Maybe there's some code that expects a foreign key but is getting an int.

(It seems some of the Query operations are very lax about this and it works to just use the string ID from the template instead of an actual int. It still looks up stuff, because the interface with Sqlite allows this.)

I'll try to write something to test the type of the field to make sure it's a foreign key. (This would only be glosses that are really old.)

The "machine values" of the field choices used to be CharFields. (!!) That was because the choice lists in the templates needed strings in the domain field, not integers. Now there is some cleverness that adds "_" before the machine values.

[susanodd]

@Jetske I put the newest migrated database in the usual location.

[susanodd]

The consistency problem shown above on that Keyword / Translation is still causing the unit tests to fail on Global. (Just now during deployment of #1212.) It can't create the test database. (It fails.)

  File "/var/www/repo/signbank/dictionary/management/commands/create_test_db.py", line 55, in handle
    make_db_small(test_db_filename)
  File "/var/www/repo/signbank/dictionary/management/commands/create_test_db.py", line 29, in make_db_small
    c.execute('DELETE FROM ' + table[0] + ';')

  File "/var/www/env/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py", line 275, in check_constraints
    raise IntegrityError(
django.db.utils.IntegrityError: The row in table 'dictionary_translation' with primary key '8' has an invalid foreign key: dictionary_translation.translation_id contains a value '7' that does not have a corresponding value in dictionary_keyword.id.

[susanodd]

I added the Bug tag because we need to figure out what is causing this to avoid more severe problems in future. It's a bug, but no idea what the bug is. There is a constraint violation somewhere in the database.

It could be that an additional table needs to be emptied during creation of the test database.

On a second deployment, there weren't any errors. So this is also unexplained.