pydantic.errors.ConfigError: duplicate validator function

[sbwhitney]

• fhir.resources version: 6.0.0b9
• Python version: 3.7
• Operating System: AWS Lambda/Jenkins/Docker/localstack/Linux

Description

Tried to redeploy an AWS Lambda function to localstack from a Jenkins container. Deploy works the first time, but subsequent deploys throw the error below after updating the Python lambda function code.

What I Did

The error seems to be thrown when importing CapabilityStatement resource from custom Validator module.

From util.validator.Validator:
from fhir.resources.capabilitystatement import (
    CapabilityStatement, CapabilityStatementRestResource)

Result:
File "/tmp/localstack/lambda_script_l_79309ee6.py", line 29, in <module>
from util.validator import Validator
File "/tmp/localstack/zipfile.e257216d/util/validator.py", line 2, in <module>
from fhir.resources.capabilitystatement import (
File "/tmp/localstack/zipfile.9e07ed84/fhir/resources/capabilitystatement.py", line 18, in <module>
class CapabilityStatement(domainresource.DomainResource):
File "/tmp/localstack/zipfile.9e07ed84/fhir/resources/capabilitystatement.py", line 453, in CapabilityStatement
def validate_required_primitive_elements(
File "/tmp/localstack/zipfile.9e07ed84/pydantic/class_validators.py", line 126, in dec
f_cls = _prepare_validator(f, allow_reuse)
File "/tmp/localstack/zipfile.9e07ed84/pydantic/class_validators.py", line 144, in _prepare_validator
raise ConfigError(f'duplicate validator function "{ref}"; if this is intended, set allow_reuse=True')
pydantic.errors.ConfigError: duplicate validator function "fhir.resources.capabilitystatement.CapabilityStatement.validate_required_primitive_elements"; if this is intended, set allow_reuse=True

[sbwhitney]

Perhaps we would need to set the boolean to True for allow_reuse wherever @root_validator is called:

https://github.com/samuelcolvin/pydantic/blob/31bc2435d7968fdf8d1f0c5c67f0f851c1bef54e/pydantic/class_validators.py#L106

[nazrulworld]

@sbwhitney thanks a lot for reporting this issue, do you add more root validator on CapabilityStatement or any custom root validator added named validate_required_primitive_elements Just trying to understand to reproduce this error :)

[sbwhitney]

We don't add any additional root validator.

[nazrulworld]

@sbwhitney I have spent some time on this error, which seems interesting to me and also a bit unusual (I cannot re-produce in my local).

As far as I understand the reason might be cache or thread or Jenkin container issue (I really don't have the full idea) if you look at here https://github.com/samuelcolvin/pydantic/blob/31bc2435d7968fdf8d1f0c5c67f0f851c1bef54e/pydantic/class_validators.py#L44 This _FUNCS container might not empty! (first time when you imports, container got values) (even you redeploy a second time) but normally when the python interpreter restarted that container be empty. Not sure how AWS Jenkin docker is working. I am really curious to find the exact reason.

Is it possible somehow manually empty that container _FUNCS, before running your Lambda functional running (maybe a stupid question :))

As per your suggestion by enabling reuse true, might fully solve this problem, but we have to compromise some integrities. I think we can do that as the last option after analyzing all perspectives, before that let's try to find the real issue.

[sbwhitney]

@nazrulworld How would we compromise integrity if we set root_validator to allow_reuse=True?

[nazrulworld]

@sbwhitney ´compromise integrity´ meant by me, for example, if two root validators func are created with the same name for a class now we are getting this ConfigError (which is good) and if we reuse true, silently second validator function will work by nature. [honestly speaking this could be a very rare case]

I see you opened an issue at pydantic, which is so nice 👌 , let's see their reaction after then we will make it allow_reuse=True for all our root validators

[sbwhitney]

Good news. I figured out what the root cause is. It turns out that the way we're importing is causing the issue.

@root_validator would fail when we do this:

from fhir.resources.capabilitystatement import CapabilityStatement

I resolved the conflict by doing this:

from fhir.resources import capabilitystatement

This seems to be a very narrow edge case when we deploy lambdas to localstack.

[sbwhitney]

Turns out the issue just moves downstream.

This works:

def create_hin_cap_statement(data, event):
    f_date = datetime.datetime.now()

    metadata = {
        "status": "active",
        "kind": "instance",
        "fhirVersion": "4.0.1",
        "date": f_date,
        "format": ["application/fhir+json"],
        "publisher": "Acme, Inc.",
    }
    cap_statement = capabilitystatement.CapabilityStatement(**metadata)

    return cap_statement.json()

while this doesn't:

def create_hin_cap_statement(data, event):
    f_date = datetime.datetime.now()

    metadata = {
        "status": "active",
        "kind": "instance",
        "fhirVersion": "4.0.1",
        "date": f_date,
        "format": ["application/fhir+json"],
        "publisher": "Acme, Inc.",
    }
    cap_statement = capabilitystatement.CapabilityStatement(**metadata)

    issues_list = []
    required_data = {
        "description": event["product"] + " API",
        "url": os.environ["BASE_URL"]
        + "/"
        + event["product"]
        + "/"
        + event["version"],
    }
    impl = capabilitystatement.CapabilityStatementImplementation(**required_data)
    cap_statement.implementation = impl.dict()

    if len(issues_list) == 0:
        issues_list.append(
            create_issue("information", "No issues creating CapabilityStatement")
        )
    required_data = {"issue": issues_list}
    op_out = operationoutcome.OperationOutcome(**required_data)

    return cap_statement.json(), op_out.dict()

[sbwhitney]

Not sure why it works locally and in AWS environment but not localstack. There are 12 methods with the same name in capabilitystatement.py which seems to be the issue:

validate_required_primitive_elements

[nazrulworld]

It is working in Travis CI as well without complaining. f_cls.__func__.__module__ + '.' + f_cls.__func__.__qualname__ that should ensure 12 unique name of validate_required_primitive_elements. as those same name 12 methods are individually assigned to 12 classes.

Is this only for capabilitystatement, you are facing this kind of problem or for all?

I am also working on avoiding same name method.

[sbwhitney]

Here's an example where the same behavior occurs in ipython and they fixed it specifically for ipython:

https://github.com/samuelcolvin/pydantic/issues/312

[sbwhitney]

It is working in Travis CI as well without complaining. f_cls.__func__.__module__ + '.' + f_cls.__func__.__qualname__ that should ensure 12 unique name of validate_required_primitive_elements. as those same name 12 methods are individually assigned to 12 classes.

Is this only for capabilitystatement, you are facing this kind of problem or for all?

I am also working on avoiding same name method.

I'm not certain if it occurs for all but I would assume so.

Thank you so much.

[nazrulworld]

@sbwhitney https://pypi.org/project/fhir.resources/6.0.0b10/