Error accessing the encrypt fields.

[kamaleshtangudu]

Data got encrypted when I created a field wrapping with encrypt method. But when I access any obj.field a Value Error is raised. ValueError: Invalid padding bytes

Which in turn raises django_cryptography.utils.crypto.InvalidToken

Django==3.0.7 djangorestframework==3.11.0 cryptography==3.1 django-cryptography==1.0

[thismatters]

Existing data is not encrypted automatically when you wrap an existing field (with existing data) with the encrypt method.

Did you do your own migration to get any existing plaintext data into the encrypted field?

[kamaleshtangudu]

Yes, I wrote the migration. The data is encrypted inside the DB as plain text.

[kamaleshtangudu]

@thismatters anything you can help with here?

[thismatters]

Could there have been an issue with your token? Are you certain that you're using the same token (and salt) as that you used when doing the encrypting migration?

[kamaleshtangudu]

No, I have generated a token with os.urandom(32) and added it as CRYPTOGRAPHY_KEY in django settings. Did not make any changes to it.

[thismatters]

To be sure, you have generated a string literal and set that as CRYPTOGRAPHY_KEY, you're not setting CRYPTOGRAPHY_KEY = os.urandom(32), right?

[kamaleshtangudu]

Yup I have generated a value from my python shell and was using it.

[kamaleshtangudu]

Values in my settings look like this, from cryptography.hazmat.backends.openssl.backend import backend CRYPTOGRAPHY_BACKEND = backend CRYPTOGRAPHY_KEY=b'v\xce\x0e\xe2\xbc\xdb&\x05q\xfb\xa9\xff\x95\xb2\x9d9Cc\x81\xf9\xb0_\x8a\xbeG\xaf\x05\xf0oDt\xa8'

The data migration looks like this, this is third migration. The first two migrations are auto generated for renaming the field and creating encrypt field.

# Generated by Django 3.0.7 on 2021-04-22 15:16

from django.db import migrations

def forwards_encrypted_char(apps, schema_editor):
    User = apps.get_model("users", "User")

    for row in User.objects.all():
        row.last_name = row.old_last_name
        row.save(update_fields=["last_name"])

def reverse_encrypted_char(apps, schema_editor):
    User = apps.get_model("users", "User")

    for row in User.objects.all():
        row.old_last_name = row.last_name
        row.save(update_fields=["old_last_name"])

class Migration(migrations.Migration):

    dependencies = [
        ('users', '0012_user_last_name'),
    ]

    operations = [
        migrations.RunPython(forwards_encrypted_char, reverse_encrypted_char),
    ]

[kamaleshtangudu]

My python version is 3.7.5 along with these pip packages. Django==3.0.7 djangorestframework==3.11.0 cryptography==3.1 django-cryptography==1.0

[thismatters]

I would try again with a different key. I've only ever used an alphanumeric key.

You've confirmed that the migration works and that the data does appear encrypted in the database?

[kamaleshtangudu]

Alphanumeric keys did not work for me actually. Can you tell me how you have created an alphanumeric key that worked? I was always getting AES unsupported no. of bits error since it only supported 128, 192 and 256bits.

Yes and the data did encrypt, and example value for encrypted last_name looks like this, \x113b2574b9bd2e278c06d076323256f4

[kamaleshtangudu]

@thismatters could you find anything?

[thismatters]

I had misremembered the details about my key. It has punctuation as well, but no unicode characters or \x## sequences.

[thismatters]

I just noticed the bit about your backend:

CRYPTOGRAPHY_BACKEND = backend

I've only ever used the default backend. Is there some motivating factor for using the openssl backend?

[kamaleshtangudu]

This is something where I was confused. The docs said CRYPTOGRAPHY_BACKEND is not compulsory but whereas as the migrations were not working without this being provided.

[thismatters]

This is the extent of configuration I had to do:

CRYPTOGRAPHY_KEY = os.environ.get("CRYPTOGRAPHY_KEY", "nothing")
CRYPTOGRAPHY_SALT = os.environ.get("CRYPTOGRAPHY_SALT", "nothing")

INSTALLED_APPS = [
    ...
    "django_cryptography",
    ....
]

Try that while using an ascii only key (and salt)!

[kamaleshtangudu]

Okay let me try with them. However the backend I was using is the very backend that comes as a response from the default_backend function.

[kamaleshtangudu]

The docs does not seem to suggest that app needs to be added in the installed apps. Also going with these settings I am getting AttributeError: 'Settings' object has no attribute 'CRYPTOGRAPHY_BACKEND'

[thismatters]

Can you post the full traceback for this error AttributeError: 'Settings' object has no attribute 'CRYPTOGRAPHY_BACKEND'?

[kamaleshtangudu]

  File "manage.py", line 24, in <module>
    execute_from_command_line(sys.argv)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/commands/showmigrations.py", line 52, in handle
    return self.show_list(connection, options['app_label'])
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/commands/showmigrations.py", line 71, in show_list
    loader = MigrationLoader(connection, ignore_no_migrations=True)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/db/migrations/loader.py", line 49, in __init__
    self.build_graph()
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/db/migrations/loader.py", line 206, in build_graph
    self.load_disk()
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/db/migrations/loader.py", line 108, in load_disk
    migration_module = import_module(migration_path)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/i0501/Documents/work/patient-profile/patient_profile/apps/users/migrations/0012_user_last_name.py", line 7, in <module>
    class Migration(migrations.Migration):
  File "/home/i0501/Documents/work/patient-profile/patient_profile/apps/users/migrations/0012_user_last_name.py", line 17, in Migration
    field=django_cryptography.fields.encrypt(models.CharField(max_length=100, null=True)),
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django_cryptography/fields.py", line 218, in encrypt
    return get_encrypted_field(type(base_field))(*args, **kwargs)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django_cryptography/fields.py", line 103, in __init__
    self._fernet = FernetBytes(self.key)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django_cryptography/utils/crypto.py", line 109, in __init__
    self._backend = settings.CRYPTOGRAPHY_BACKEND
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/appconf/base.py", line 126, in __getattr__
    return getattr(self._meta.holder, name)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/conf/__init__.py", line 77, in __getattr__
    val = getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'CRYPTOGRAPHY_BACKEND'

[thismatters]

and can you paste the full text of the migration called 0012_user_last_name.py?

[thismatters]

when you got that last traceback had you put django_cryptography in the INSTALLED_APPS?

[kamaleshtangudu]

Got the same traceback with and without adding django_cryptography to installed apps. 0012 is auto generated migration for encrypt field that is created by django itself.

[thismatters]

I would like to see the text of that migration since that is where the exception is being raised.

[kamaleshtangudu]

Can you use this traceback, as the same error is getting returned even when I am retrying to generate the very 0012 migration. Initially when I was testing, I changed this setting after this creating this migration.

  File "manage.py", line 24, in <module>
    execute_from_command_line(sys.argv)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/core/management/commands/makemigrations.py", line 142, in handle
    ProjectState.from_apps(apps),
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/db/migrations/state.py", line 221, in from_apps
    model_state = ModelState.from_model(model)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/db/migrations/state.py", line 413, in from_model
    fields.append((name, field.clone()))
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django_cryptography/fields.py", line 135, in clone
    self.base_class(*args, **kwargs), self.key, self.ttl)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django_cryptography/fields.py", line 218, in encrypt
    return get_encrypted_field(type(base_field))(*args, **kwargs)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django_cryptography/fields.py", line 103, in __init__
    self._fernet = FernetBytes(self.key)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django_cryptography/utils/crypto.py", line 109, in __init__
    self._backend = settings.CRYPTOGRAPHY_BACKEND
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/appconf/base.py", line 126, in __getattr__
    return getattr(self._meta.holder, name)
  File "/home/i0501/Documents/work/patient-profile/venv/lib/python3.7/site-packages/django/conf/__init__.py", line 77, in __getattr__
    val = getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'CRYPTOGRAPHY_BACKEND'

THis is the field I was trying to add here, last_name = encrypt(models.CharField(max_length=100, null=True))

[thismatters]

it looks to me like django_cryptography is not being configured for some reason. I'm not sure why, but it could be related to your settings and how they are being specified to the wsgi server. I'm still learning how this works in django, so I don't quite know how to troubleshoot this further.

[thismatters]

This package no longer requires being added to INSTALLED_APPS, so this isn't a failure of django to do the app config stuff. In fact utils/crypto.py ensures that it has the config that it needs to operate:

from ..conf import CryptographyConf

settings = CryptographyConf()

So I'm very confused about why you're getting the error you're seeing.

I did just notice that you're using a fairly old version of Django. Can you try using a more recent version (3.2.x) to see if this problem persists?

[thismatters]

Before upgrading version, just as a sanity check can you open a django shell (within your project) and run the following three lines:

from django_cryptography.conf import CryptographyConf
settings = CryptographyConf()
settings.CRYPTOGRAPHY_BACKEND

And see if the AttributeError is raised.

[kamaleshtangudu]

Yes, the error is raised. Sadly migrating to django 3.2.x at this time is not possible because of backwards incompatible changes in django 3.1 on JSONField.

[kamaleshtangudu]

Upgrading to django-3.2.x and running this on the shell still causes and issue. I think I might have got the issue partially, I use django-configurations to achieve class based settings rather than the classic django settings(file based settings). AppConf class might be something that is only compatible with classic settings and not with django-configurations.

[thismatters]

I don't see how that could be possible.

I've just re-created your environment in a docker container and the commands executed as expected:

/test # python manage.py shell
Python 3.7.10 (default, Apr 15 2021, 05:25:07) 
[GCC 10.2.1 20201203] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django_cryptography.conf import CryptographyConf
>>> settings = CryptographyConf()
>>> settings.CRYPTOGRAPHY_BACKEND
<cryptography.hazmat.backends.openssl.backend.Backend object at 0x7fd6cb3d4750>
>>> 
now exiting InteractiveConsole...
/test # pip freeze | grep cryptography
cryptography==3.1
django-cryptography==1.0
/test # pip freeze | grep Django
Django==3.0.7

Have you altered any files the django_cryptography package?

[thismatters]

I realized that I was using a newer version of python (3.7.10) while you're using 3.7.5.

I was unable to get my test to run in 3.7.5. I recommend upgrading your python. At a minimum I would say you should be using the most recent minor version (3.7.10), but you should consider getting to the most recent major version as well.

[kamaleshtangudu]

The problem does not seem to be with these versions. I have run the same version of django-cryptography with this python version in another setup without django-configurations and it is working fine. The issue seems to be the compatibility between django-configurations based settings and django-appconf.

[thismatters]

Oh, I hadn't seen that you were using django-configurations.