Closed irv closed 2 years ago
@irv Thank you for the bug report, i am currently working to resolve this issue.
The problem with implementing schema.org in pydantic is that there are a lot of (circular) references. Importing the all the models in the package's init.py is a solution makes the package initial import very slow..
I will keep you updated!
@irv I have updated the BaseModel to resolve the forward references on model creation. To keep this updating of forward references efficient, only those fields that are initiated are updated.
I also tested the library to all the JSON-LD examples present in this file: https://schema.org/version/latest/schemaorg-all-examples.txt. The model creations now all succeed.
@irv Can you try to use the release 1.0.0b?
See SchemaOrgBase
to see how the references are resolved/updated.
class SchemaOrgBase(BaseModel):
#JSON-LD fields
reverse_ : Optional[Any] = Field(None,alias='@reverse')
id_ : Optional[Any] = Field(None,alias='@id')
context_ : Optional[Any] = Field(None,alias='@context')
graph_ : Optional[Any] = Field(None,alias='@graph')
def dict(self, *args, **kwargs):
defaults = {
"exclude_none": True,
"by_alias": True
}
return super().dict(*args, **dict(defaults, **kwargs))
def json(self, *args, **kwargs):
defaults = {
"exclude_none": True,
"by_alias": True
}
return super().json(*args, **dict(defaults, **kwargs))
class Config:
allow_population_by_field_name = True
@classmethod
def get_classes_for_forward_ref(cls, field: Any) -> List[tuple]:
classes = []
if type(field.type_) == ForwardRef:
t = field.type_
u = t.__forward_code__
v = u.co_consts
for w in v:
pydanticschema_org_type = types[w]
mod = __import__(pydanticschema_org_type[1], fromlist=[pydanticschema_org_type[0]])
class_ = getattr(mod, pydanticschema_org_type[0])
classes.append((w, class_))
return classes
@classmethod
def get_local_ns(cls):
global updated_models
localns = {}
for k, v in cls.__fields__.items():
classes = cls.get_classes_for_forward_ref(v)
for class_name, class_ in classes:
localns.update({class_name: class_})
return localns
@classmethod
def update_forward_refs(cls, **localns: Any) -> None:
"""
Try to update ForwardRefs on fields based on this Model, globalns and localns.
"""
locals = {'Optional': Optional, 'List': List, 'Union': Union, 'StrictBool': StrictBool, 'AnyUrl': AnyUrl,
'Decimal': Decimal, 'time': time,'ISO8601Date':ISO8601Date}
for cls_ in cls.mro():
if hasattr(cls_, 'get_local_ns'):
locals.update(cls_.get_local_ns())
update_model_forward_refs(cls, cls.__fields__.values(), cls.__config__.json_encoders, locals)
@classmethod
def _update_all_fields(cls):
for cls_ in cls.mro():
if hasattr(cls_, 'get_classes_for_forward_ref'):
for k in cls_.__fields__.keys():
if k not in updated_models:
field = cls_.__fields__[k]
classes = cls_.get_classes_for_forward_ref(field)
for class_name, class_ in classes:
class_.update_forward_refs()
def __init__(__pydantic_self__, **data: Any) -> None:
__pydantic_self__.update_forward_refs()
for k, v in data.items():
if k in __pydantic_self__.__fields__.keys():
if k not in updated_models:
field = __pydantic_self__.__fields__[k]
classes = __pydantic_self__.get_classes_for_forward_ref(field)
for class_name, class_ in classes:
class_.update_forward_refs()
super().__init__(**data)
Hi, thanks for looking at this so quickly! And also thanks for taking the project on, you're absolutelyright that it's a nightmare for circular references 😭
OK, if i clone the repo and run src/example.py it works 🎉
However, if I add 1.0.0b as a dep in a new project, I get:
python schema_test.py
Traceback (most recent call last):
File "/Users/airving/Source/schema_test/schema_test.py", line 2, in <module>
from pydantic_schemaorg.ScholarlyArticle import ScholarlyArticle
File "/Users/airving/Library/Caches/pypoetry/virtualenvs/schema-test-nsFkZVx8-py3.9/lib/python3.9/site-packages/pydantic_schemaorg/ScholarlyArticle.py", line 5, in <module>
from pydantic_schemaorg.Article import Article
File "/Users/airving/Library/Caches/pypoetry/virtualenvs/schema-test-nsFkZVx8-py3.9/lib/python3.9/site-packages/pydantic_schemaorg/Article.py", line 9, in <module>
from pydantic_schemaorg.CreativeWork import CreativeWork
File "/Users/airving/Library/Caches/pypoetry/virtualenvs/schema-test-nsFkZVx8-py3.9/lib/python3.9/site-packages/pydantic_schemaorg/CreativeWork.py", line 6, in <module>
from pydantic_schemaorg.ISO8601.ISO8601Date import ISO8601Date
File "/Users/airving/Library/Caches/pypoetry/virtualenvs/schema-test-nsFkZVx8-py3.9/lib/python3.9/site-packages/pydantic_schemaorg/ISO8601/ISO8601Date.py", line 9, in <module>
from ISO8601 import errors
ModuleNotFoundError: No module named 'ISO8601'
So, I think just this would make it work?
diff --git a/src/ISO8601/ISO8601Date.py b/src/ISO8601/ISO8601Date.py
index 3685211..f6ee764 100644
--- a/src/ISO8601/ISO8601Date.py
+++ b/src/ISO8601/ISO8601Date.py
@@ -6,7 +6,7 @@ from pydantic.fields import ModelField
from pydantic.utils import update_not_none
from pydantic.validators import str_validator, constr_length_validator
-from ISO8601 import errors
+from pydantic_schemaorg.ISO8601 import errors
if TYPE_CHECKING:
from pydantic.typing import AnyCallable
diff --git a/src/templates/schema_org_base.py.tpl b/src/templates/schema_org_base.py.tpl
index d0d8495..c7333cf 100644
--- a/src/templates/schema_org_base.py.tpl
+++ b/src/templates/schema_org_base.py.tpl
@@ -5,7 +5,7 @@ from typing import Any, Optional, ForwardRef, List, Union
from pydantic import BaseModel, Field, StrictBool, AnyUrl
from pydantic.typing import update_model_forward_refs
-from ISO8601.ISO8601Date import ISO8601Date
+from pydantic_schemaorg.ISO8601.ISO8601Date import ISO8601Date
from pydantic_schemaorg.__types__ import types
updated_models=set()
Hi @irv, and thank you for using this library and reporting the bugs :D :+1:
I merged your suggestion in the library. It was indeed importing the ISO8601Date from the wrong directory. (OMG so many files)
I made a new pre-release (v1.0.0c) with this fix.
Please let me know :dancers:
Awesome! I think the path in pydantic_schemaorg/ISO8601/ISO8601Date.py still needs changing
diff --git a/src/ISO8601/ISO8601Date.py b/src/ISO8601/ISO8601Date.py
index 3685211..f6ee764 100644
--- a/src/ISO8601/ISO8601Date.py
+++ b/src/ISO8601/ISO8601Date.py
@@ -6,7 +6,7 @@ from pydantic.fields import ModelField
from pydantic.utils import update_not_none
from pydantic.validators import str_validator, constr_length_validator
-from ISO8601 import errors
+from pydantic_schemaorg.ISO8601 import errors
if TYPE_CHECKING:
from pydantic.typing import AnyCallable
But I think it should all work after that 🤞
Sorry was a bit too quick there. Lets try with v1.0.0c2! :clown_face:
It works! Thank you
Great, thanks for the input.
I also tested the package against the tests i created. I will release this version (v1.0.0c2) as the first stable release v1.0.0
Hi,
using 1.0.0a with python 3.9, I can't get the example to work
Which may be a pydantic bug/feature, there's possibly a workaround listed here https://github.com/samuelcolvin/pydantic/issues/2411