Circular dependencies in schemas causing AttributeError

CausticYarn commented 6 years ago

I am getting an AttributeError in _nsgroups.py when trying to parse a document from the bindings generated for the attached schema. It seems to be related to circular dependencies in the source schemas (which are auto-generated).

I am just looking for some guidance on whether PyXB supports handling schemas with circular dependencies and/or if there is some trick to generating the bindings, or if I need to completely refactor the source schema to get it to work.

I have attached a zip of the source schema. Thanks in advance for any guidance.

regxml.zip

CausticYarn commented 6 years ago

FWIW, Altova XMLSpy is able to validate a sample file using the include.xsd schema.

pabigot commented 6 years ago

I don't know how you generate the bindings; my second attempt was with the following, which worked with some warnings about lack of information for imported namespaces that might be eliminated by changing the order of generation.

Theoretically PyXB sorts the dependencies for anything generated in the same invocation and performs resolution bottom up, so it should handle anything that isn't a true circular dependency (which I believe would be invalid).

Without a document to parse I can't tell if this works.

pyxbgen \
-u reg.xsd -m reg \
-u www-ebu-ch-metadata-schemas-ebucore-smpte-class13-element.xsd -m www-ebu-ch-metadata-schemas-ebucore-smpte-class13-element \
-u www-ebu-ch-metadata-schemas-ebucore-smpte-class13-group.xsd -m www-ebu-ch-metadata-schemas-ebucore-smpte-class13-group \
-u www-ebu-ch-metadata-schemas-ebucore-smpte-class13-type.xsd -m www-ebu-ch-metadata-schemas-ebucore-smpte-class13-type \
-u www-smpte-ra-org-reg-2003-2012-13-1-amwa-as11.xsd -m www-smpte-ra-org-reg-2003-2012-13-1-amwa-as11 \
-u www-smpte-ra-org-reg-2003-2012-13-1-amwa-as12.xsd -m www-smpte-ra-org-reg-2003-2012-13-1-amwa-as12 \
-u www-smpte-ra-org-reg-2003-2012-13-12-as11.xsd -m www-smpte-ra-org-reg-2003-2012-13-12-as11 \
-u www-smpte-ra-org-reg-2003-2012-13-4-archive.xsd -m www-smpte-ra-org-reg-2003-2012-13-4-archive \
-u www-smpte-ra-org-reg-2003-2012.xsd -m www-smpte-ra-org-reg-2003-2012 \
-u www-smpte-ra-org-reg-335-2012-13-1-aaf.xsd -m www-smpte-ra-org-reg-335-2012-13-1-aaf \
-u www-smpte-ra-org-reg-335-2012-13-1-amwa-as10.xsd -m www-smpte-ra-org-reg-335-2012-13-1-amwa-as10 \
-u www-smpte-ra-org-reg-335-2012-13-1-amwa-as11.xsd -m www-smpte-ra-org-reg-335-2012-13-1-amwa-as11 \
-u www-smpte-ra-org-reg-335-2012-13-1-amwa-as12.xsd -m www-smpte-ra-org-reg-335-2012-13-1-amwa-as12 \
-u www-smpte-ra-org-reg-335-2012-13-1-amwa-rules.xsd -m www-smpte-ra-org-reg-335-2012-13-1-amwa-rules \
-u www-smpte-ra-org-reg-335-2012-13-12-as11.xsd -m www-smpte-ra-org-reg-335-2012-13-12-as11 \
-u www-smpte-ra-org-reg-335-2012-13-13.xsd -m www-smpte-ra-org-reg-335-2012-13-13 \
-u www-smpte-ra-org-reg-335-2012-13-4-archive.xsd -m www-smpte-ra-org-reg-335-2012-13-4-archive \
-u www-smpte-ra-org-reg-335-2012.xsd -m www-smpte-ra-org-reg-335-2012 \
-u www-smpte-ra-org-reg-395-2014-13-1-aaf.xsd -m www-smpte-ra-org-reg-395-2014-13-1-aaf \
-u www-smpte-ra-org-reg-395-2014-13-1-amwa-as-common.xsd -m www-smpte-ra-org-reg-395-2014-13-1-amwa-as-common \
-u www-smpte-ra-org-reg-395-2014-13-1-amwa-as10.xsd -m www-smpte-ra-org-reg-395-2014-13-1-amwa-as10 \
-u www-smpte-ra-org-reg-395-2014-13-1-amwa-as11.xsd -m www-smpte-ra-org-reg-395-2014-13-1-amwa-as11 \
-u www-smpte-ra-org-reg-395-2014-13-1-amwa-as12.xsd -m www-smpte-ra-org-reg-395-2014-13-1-amwa-as12 \
-u www-smpte-ra-org-reg-395-2014-13-12-as11.xsd -m www-smpte-ra-org-reg-395-2014-13-12-as11 \
-u www-smpte-ra-org-reg-395-2014-13-13.xsd -m www-smpte-ra-org-reg-395-2014-13-13 \
-u www-smpte-ra-org-reg-395-2014-13-4-archive.xsd -m www-smpte-ra-org-reg-395-2014-13-4-archive \
-u www-smpte-ra-org-reg-395-2014.xsd -m www-smpte-ra-org-reg-395-2014

CausticYarn commented 6 years ago

Thanks so much for the quick reply. Sorry if I was unclear, I was able to generate the bindings, but when I try to use www-smpte-ra-org-reg-395-2014-13-1-aaf.CreateFromDocument on the attached XML files, I get the attribute error; it seems like _nsgroups.py doesn't get to finish loading before another module tries to access a type.

instances.zip

pabigot commented 6 years ago

You weren't unclear, just incomplete. You identified a problem parsing a document, but didn't provide a document I could use to reproduce the problem.

It would save time in the future if you would provide:

the schema (which you did);
how you built the bindings (which I guessed);
a small wrapper application that used the bindings to reproduce the problem;
output that shows exactly what error you saw;
a description of what you expected to have happened.

I made a script that imported the modules generated by the script I showed in that same order. Running it produces the following:

lilith[48]$ python test.py 
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    import www_ebu_ch_metadata_schemas_ebucore_smpte_class13_element
  File "/mnt/devel/pyxb/www_ebu_ch_metadata_schemas_ebucore_smpte_class13_element.py", line 30, in <module>
    import _nsgroup as _ImportedBinding__nsgroup
  File "/mnt/devel/pyxb/_nsgroup.py", line 45, in <module>
    import www_smpte_ra_org_reg_335_2012 as _ImportedBinding_www_smpte_ra_org_reg_335_2012
  File "/mnt/devel/pyxb/www_smpte_ra_org_reg_335_2012.py", line 89, in <module>
    InstanceID = pyxb.binding.basis.element(pyxb.namespace.ExpandedName(Namespace, 'InstanceID'), _ImportedBinding__nsgroup.UUID_, location=pyxb.utils.utility.Location('/mnt/devel/pyxb/www-smpte-ra-org-reg-335-2012.xsd', 9, 1))
AttributeError: 'module' object has no attribute 'UUID_'

Is that the sort of error you see?

PyXB does try to impose a strict order where types are defined before they're referenced, and puts all those types into a file where the order is satisfied (in this case, _nsgroup). Normally it diagnoses a situation where it finds a cycle in the dependency graph, but it didn't do so in this case. That's probably a bug, but not one likely to be fixed.

With "sane" schema one can often generate the bindings for related namespaces in smaller groups that form a strongly-connected component; see this example. Maybe that can be done based on prefixes of your namespaces. Then you build additional namespaces using PyXB's archive facility to load in the metadata describing the binding content. The opengis bundle as a whole may be a useful example.

However, this is only supposed to be needed to reduce the size of individual generated binding files. PyXB does the graph analysis to identify SCCs, and might have concluded that everything in _nsgroup is a strongly-connected-component (otherwise it would have done _nsgroup2, etc).

So in short, this should work, if you can generate the bindings in sets that have self-contained cross-namespace dependency loops, and no true loops at the individual type level (e.g. group models that contain elements that contain the group, though in some cases that can be made to work). If the loops are inherent in the information structures you'll need to find another solution.

pabigot / pyxb

Circular dependencies in schemas causing AttributeError #101