pydantic.errors.ConfigError: field not yet prepared so type is still a ForwardRef

[irv]

Hi,

using 1.0.0a with python 3.9, I can't get the example to work

from datetime import datetime
from pydantic_schemaorg.ScholarlyArticle import ScholarlyArticle

scholarly_article = ScholarlyArticle(url='https://github.com/lexiq-legal/pydantic_schemaorg',
                                    sameAs='https://github.com/lexiq-legal/pydantic_schemaorg',
                                    copyrightNotice='Free to use under the MIT license',
                                    dateCreated=datetime.now())
print(scholarly_article.json(exclude_none=True))

  File "pydantic/main.py", line 329, in pydantic.main.BaseModel.__init__
  File "pydantic/main.py", line 1022, in pydantic.main.validate_model
  File "pydantic/fields.py", line 830, in pydantic.fields.ModelField.validate
pydantic.errors.ConfigError: field "url" not yet prepared so type is still a ForwardRef, you might need to call ScholarlyArticle.update_forward_refs().

Which may be a pydantic bug/feature, there's possibly a workaround listed here https://github.com/samuelcolvin/pydantic/issues/2411

[crbaker89]

@irv Thank you for the bug report, i am currently working to resolve this issue.

The problem with implementing schema.org in pydantic is that there are a lot of (circular) references. Importing the all the models in the package's init.py is a solution makes the package initial import very slow..

I will keep you updated!

[crbaker89]

@irv I have updated the BaseModel to resolve the forward references on model creation. To keep this updating of forward references efficient, only those fields that are initiated are updated.

I also tested the library to all the JSON-LD examples present in this file: https://schema.org/version/latest/schemaorg-all-examples.txt. The model creations now all succeed.

@irv Can you try to use the release 1.0.0b?

See SchemaOrgBase to see how the references are resolved/updated.

class SchemaOrgBase(BaseModel):
    #JSON-LD fields
    reverse_ : Optional[Any] = Field(None,alias='@reverse')
    id_ : Optional[Any] = Field(None,alias='@id')
    context_ : Optional[Any] = Field(None,alias='@context')
    graph_ : Optional[Any] = Field(None,alias='@graph')

    def dict(self, *args, **kwargs):
        defaults = {
            "exclude_none": True,
            "by_alias": True
        }
        return super().dict(*args, **dict(defaults, **kwargs))

    def json(self, *args, **kwargs):
        defaults = {
            "exclude_none": True,
            "by_alias": True
        }
        return super().json(*args, **dict(defaults, **kwargs))

    class Config:
        allow_population_by_field_name = True

    @classmethod
    def get_classes_for_forward_ref(cls, field: Any) -> List[tuple]:
        classes = []
        if type(field.type_) == ForwardRef:
            t = field.type_
            u = t.__forward_code__
            v = u.co_consts
            for w in v:
                pydanticschema_org_type = types[w]
                mod = __import__(pydanticschema_org_type[1], fromlist=[pydanticschema_org_type[0]])
                class_ = getattr(mod, pydanticschema_org_type[0])
                classes.append((w, class_))
        return classes

    @classmethod
    def get_local_ns(cls):
        global updated_models
        localns = {}
        for k, v in cls.__fields__.items():
            classes = cls.get_classes_for_forward_ref(v)
            for class_name, class_ in classes:
                localns.update({class_name: class_})
        return localns

    @classmethod
    def update_forward_refs(cls, **localns: Any) -> None:
        """
        Try to update ForwardRefs on fields based on this Model, globalns and localns.
        """
        locals = {'Optional': Optional, 'List': List, 'Union': Union, 'StrictBool': StrictBool, 'AnyUrl': AnyUrl,
                  'Decimal': Decimal, 'time': time,'ISO8601Date':ISO8601Date}
        for cls_ in cls.mro():
            if hasattr(cls_, 'get_local_ns'):
                locals.update(cls_.get_local_ns())
        update_model_forward_refs(cls, cls.__fields__.values(), cls.__config__.json_encoders, locals)

    @classmethod
    def _update_all_fields(cls):
        for cls_ in cls.mro():
            if hasattr(cls_, 'get_classes_for_forward_ref'):
                for k in cls_.__fields__.keys():
                    if k not in updated_models:
                        field = cls_.__fields__[k]
                        classes = cls_.get_classes_for_forward_ref(field)
                        for class_name, class_ in classes:
                            class_.update_forward_refs()

    def __init__(__pydantic_self__, **data: Any) -> None:
        __pydantic_self__.update_forward_refs()
        for k, v in data.items():
            if k in __pydantic_self__.__fields__.keys():
                if k not in updated_models:
                    field = __pydantic_self__.__fields__[k]
                    classes = __pydantic_self__.get_classes_for_forward_ref(field)
                    for class_name, class_ in classes:
                        class_.update_forward_refs()
        super().__init__(**data)

[irv]

Hi, thanks for looking at this so quickly! And also thanks for taking the project on, you're absolutelyright that it's a nightmare for circular references 😭

OK, if i clone the repo and run src/example.py it works 🎉

However, if I add 1.0.0b as a dep in a new project, I get:

python schema_test.py
Traceback (most recent call last):
  File "/Users/airving/Source/schema_test/schema_test.py", line 2, in <module>
    from pydantic_schemaorg.ScholarlyArticle import ScholarlyArticle
  File "/Users/airving/Library/Caches/pypoetry/virtualenvs/schema-test-nsFkZVx8-py3.9/lib/python3.9/site-packages/pydantic_schemaorg/ScholarlyArticle.py", line 5, in <module>
    from pydantic_schemaorg.Article import Article
  File "/Users/airving/Library/Caches/pypoetry/virtualenvs/schema-test-nsFkZVx8-py3.9/lib/python3.9/site-packages/pydantic_schemaorg/Article.py", line 9, in <module>
    from pydantic_schemaorg.CreativeWork import CreativeWork
  File "/Users/airving/Library/Caches/pypoetry/virtualenvs/schema-test-nsFkZVx8-py3.9/lib/python3.9/site-packages/pydantic_schemaorg/CreativeWork.py", line 6, in <module>
    from pydantic_schemaorg.ISO8601.ISO8601Date import ISO8601Date
  File "/Users/airving/Library/Caches/pypoetry/virtualenvs/schema-test-nsFkZVx8-py3.9/lib/python3.9/site-packages/pydantic_schemaorg/ISO8601/ISO8601Date.py", line 9, in <module>
    from ISO8601 import errors
ModuleNotFoundError: No module named 'ISO8601'

So, I think just this would make it work?

diff --git a/src/ISO8601/ISO8601Date.py b/src/ISO8601/ISO8601Date.py
index 3685211..f6ee764 100644
--- a/src/ISO8601/ISO8601Date.py
+++ b/src/ISO8601/ISO8601Date.py
@@ -6,7 +6,7 @@ from pydantic.fields import ModelField
 from pydantic.utils import update_not_none
 from pydantic.validators import str_validator, constr_length_validator

-from ISO8601 import errors
+from pydantic_schemaorg.ISO8601 import errors

 if TYPE_CHECKING:
     from pydantic.typing import AnyCallable
diff --git a/src/templates/schema_org_base.py.tpl b/src/templates/schema_org_base.py.tpl
index d0d8495..c7333cf 100644
--- a/src/templates/schema_org_base.py.tpl
+++ b/src/templates/schema_org_base.py.tpl
@@ -5,7 +5,7 @@ from typing import Any, Optional, ForwardRef, List, Union
 from pydantic import BaseModel, Field, StrictBool, AnyUrl
 from pydantic.typing import update_model_forward_refs

-from ISO8601.ISO8601Date import ISO8601Date
+from pydantic_schemaorg.ISO8601.ISO8601Date import ISO8601Date
 from pydantic_schemaorg.__types__ import types

 updated_models=set()

[crbaker89]

Hi @irv, and thank you for using this library and reporting the bugs :D :+1:

I merged your suggestion in the library. It was indeed importing the ISO8601Date from the wrong directory. (OMG so many files)

I made a new pre-release (v1.0.0c) with this fix.

Please let me know :dancers:

[irv]

Awesome! I think the path in pydantic_schemaorg/ISO8601/ISO8601Date.py still needs changing

diff --git a/src/ISO8601/ISO8601Date.py b/src/ISO8601/ISO8601Date.py
index 3685211..f6ee764 100644
--- a/src/ISO8601/ISO8601Date.py
+++ b/src/ISO8601/ISO8601Date.py
@@ -6,7 +6,7 @@ from pydantic.fields import ModelField
 from pydantic.utils import update_not_none
 from pydantic.validators import str_validator, constr_length_validator

-from ISO8601 import errors
+from pydantic_schemaorg.ISO8601 import errors

 if TYPE_CHECKING:
     from pydantic.typing import AnyCallable

But I think it should all work after that 🤞

[crbaker89]

Sorry was a bit too quick there. Lets try with v1.0.0c2! :clown_face:

[irv]

It works! Thank you

[crbaker89]

Great, thanks for the input.

I also tested the package against the tests i created. I will release this version (v1.0.0c2) as the first stable release v1.0.0