0.8.3 - TypeError: cannot pickle 'lxml.etree.XMLSchema' object

[gunhanoral]

I can generate openconfig interfaces model with pyangbind 0.8.2 but not with 0.8.3

Directory

``` . ├── Dockerfile-v0.8.2 ├── Dockerfile-v0.8.3 └── public ```

Dockerfile-v0.8.2

``` FROM python:3.9-slim RUN apt update RUN pip install -U pip RUN pip install pyangbind=='0.8.2' COPY public public RUN pyang --plugindir /usr/local/lib/python3.9/site-packages/pyangbind/plugin -f pybind ./public/release/models/openconfig-extensions.yang ./public/release/models/types/openconfig-yang-types.yang ./public/release/models/types/openconfig-types.yang ./public/release/models/interfaces/openconfig-interfaces.yang ./public/release/models/optical-transport/openconfig-transport-types.yang ./public/release/models/platform/openconfig-platform-types.yang ```

Dockerfile-v0.8.3

``` FROM python:3.9-slim RUN apt update RUN pip install -U pip RUN pip install pyangbind=='0.8.3' COPY public public RUN pyang --plugindir /usr/local/lib/python3.9/site-packages/pyangbind/plugin -f pybind ./public/release/models/openconfig-extensions.yang ./public/release/models/types/openconfig-yang-types.yang ./public/release/models/types/openconfig-types.yang ./public/release/models/interfaces/openconfig-interfaces.yang ./public/release/models/optical-transport/openconfig-transport-types.yang ./public/release/models/platform/openconfig-platform-types.yang ```

docker build -t pyangbind:v0.8.2 -f Dockerfile-v0.8.2 . completes successfully but v0.8.3 fails with

TypeError: cannot pickle 'lxml.etree.XMLSchema' object

``` > [6/6] RUN pyang --plugindir /usr/local/lib/python3.9/site-packages/pyangbind/plugin -f pybind ./public/release/models/openconfig-extensions.yang ./public/release/models/types/openconfig-yang-types.yang ./public/release/models/types/openconfig-types.yang ./public/release/models/interfaces/openconfig-interfaces.yang ./public/release/models/optical-transport/openconfig-transport-types.yang ./public/release/models/platform/openconfig-platform-types.yang: 0.382 # -*- coding: utf-8 -*- 0.382 from operator import attrgetter 0.382 from pyangbind.lib.yangtypes import RestrictedPrecisionDecimalType 0.382 from pyangbind.lib.yangtypes import RestrictedClassType 0.382 from pyangbind.lib.yangtypes import TypedListType 0.382 from pyangbind.lib.yangtypes import YANGBool 0.382 from pyangbind.lib.yangtypes import YANGListType 0.382 from pyangbind.lib.yangtypes import YANGDynClass 0.382 from pyangbind.lib.yangtypes import ReferenceType 0.382 from pyangbind.lib.yangtypes import YANGBinary 0.382 from pyangbind.lib.base import PybindBase 0.382 from collections import OrderedDict 0.382 from decimal import Decimal 0.382 import six 0.382 0.382 # PY3 support of some PY2 keywords (needs improved) 0.382 if six.PY3: 0.382 import builtins as __builtin__ 0.382 long = int 0.382 elif six.PY2: 0.382 import __builtin__ 0.382 0.382 Traceback (most recent call last): 0.382 File "/usr/local/bin/pyang", line 580, in 0.382 run() 0.382 File "/usr/local/bin/pyang", line 549, in run 0.382 emit_obj.emit(ctx, modules, fd) 0.382 File "/usr/local/lib/python3.9/site-packages/pyangbind/plugin/pybind.py", line 201, in emit 0.383 build_pybind(ctx, modules, fd) 0.383 File "/usr/local/lib/python3.9/site-packages/pyangbind/plugin/pybind.py", line 403, in build_pybind 0.383 build_identities(ctx, defn["identity"]) 0.383 File "/usr/local/lib/python3.9/site-packages/pyangbind/plugin/pybind.py", line 448, in build_identities 0.383 idstore.build_store_from_definitions(ctx, defnd) 0.383 File "/usr/local/lib/python3.9/site-packages/pyangbind/helpers/identity.py", line 75, in build_store_from_definitions 0.383 mod_ref_prefixes = module_import_prefixes(ctx) 0.383 File "/usr/local/lib/python3.9/site-packages/pyangbind/helpers/misc.py", line 24, in module_import_prefixes 0.383 for mod in copy.deepcopy(ctx.modules): 0.383 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.383 y = copier(x, memo) 0.383 File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict 0.383 y[deepcopy(key, memo)] = deepcopy(value, memo) 0.383 File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy 0.383 y = _reconstruct(x, memo, *rv) 0.383 File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct 0.383 state = deepcopy(state, memo) 0.383 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.383 y = copier(x, memo) 0.383 File "/usr/local/lib/python3.9/copy.py", line 210, in _deepcopy_tuple 0.384 y = [deepcopy(a, memo) for a in x] 0.384 File "/usr/local/lib/python3.9/copy.py", line 210, in 0.384 y = [deepcopy(a, memo) for a in x] 0.384 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.384 y = copier(x, memo) 0.384 File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict 0.384 y[deepcopy(key, memo)] = deepcopy(value, memo) 0.384 File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy 0.384 y = _reconstruct(x, memo, *rv) 0.384 File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct 0.384 state = deepcopy(state, memo) 0.384 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.384 y = copier(x, memo) 0.384 File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict 0.384 y[deepcopy(key, memo)] = deepcopy(value, memo) 0.384 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.384 y = copier(x, memo) 0.384 File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict 0.384 y[deepcopy(key, memo)] = deepcopy(value, memo) 0.384 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.385 y = copier(x, memo) 0.385 File "/usr/local/lib/python3.9/copy.py", line 205, in _deepcopy_list 0.385 append(deepcopy(a, memo)) 0.385 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.385 y = copier(x, memo) 0.385 File "/usr/local/lib/python3.9/copy.py", line 210, in _deepcopy_tuple 0.385 y = [deepcopy(a, memo) for a in x] 0.385 File "/usr/local/lib/python3.9/copy.py", line 210, in 0.385 y = [deepcopy(a, memo) for a in x] 0.385 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.385 y = copier(x, memo) 0.385 File "/usr/local/lib/python3.9/copy.py", line 210, in _deepcopy_tuple 0.385 y = [deepcopy(a, memo) for a in x] 0.385 File "/usr/local/lib/python3.9/copy.py", line 210, in 0.385 y = [deepcopy(a, memo) for a in x] 0.385 File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy 0.385 y = _reconstruct(x, memo, *rv) 0.385 File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct 0.385 state = deepcopy(state, memo) 0.385 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.385 y = copier(x, memo) 0.385 File "/usr/local/lib/python3.9/copy.py", line 210, in _deepcopy_tuple 0.386 y = [deepcopy(a, memo) for a in x] 0.386 File "/usr/local/lib/python3.9/copy.py", line 210, in 0.386 y = [deepcopy(a, memo) for a in x] 0.386 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.386 y = copier(x, memo) 0.386 File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict 0.386 y[deepcopy(key, memo)] = deepcopy(value, memo) 0.386 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.386 y = copier(x, memo) 0.386 File "/usr/local/lib/python3.9/copy.py", line 205, in _deepcopy_list 0.386 append(deepcopy(a, memo)) 0.386 File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy 0.386 y = _reconstruct(x, memo, *rv) 0.386 File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct 0.386 state = deepcopy(state, memo) 0.386 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.386 y = copier(x, memo) 0.386 File "/usr/local/lib/python3.9/copy.py", line 210, in _deepcopy_tuple 0.386 y = [deepcopy(a, memo) for a in x] 0.387 File "/usr/local/lib/python3.9/copy.py", line 210, in 0.387 y = [deepcopy(a, memo) for a in x] 0.387 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.387 y = copier(x, memo) 0.387 File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict 0.387 y[deepcopy(key, memo)] = deepcopy(value, memo) 0.387 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.387 y = copier(x, memo) 0.387 File "/usr/local/lib/python3.9/copy.py", line 205, in _deepcopy_list 0.387 append(deepcopy(a, memo)) 0.387 File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy 0.387 y = _reconstruct(x, memo, *rv) 0.387 File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct 0.387 state = deepcopy(state, memo) 0.387 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.387 y = copier(x, memo) 0.387 File "/usr/local/lib/python3.9/copy.py", line 210, in _deepcopy_tuple 0.387 y = [deepcopy(a, memo) for a in x] 0.387 File "/usr/local/lib/python3.9/copy.py", line 210, in 0.387 y = [deepcopy(a, memo) for a in x] 0.387 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.387 y = copier(x, memo) 0.388 File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict 0.388 y[deepcopy(key, memo)] = deepcopy(value, memo) 0.388 File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy 0.388 y = _reconstruct(x, memo, *rv) 0.388 File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct 0.388 state = deepcopy(state, memo) 0.388 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.388 y = copier(x, memo) 0.388 File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict 0.388 y[deepcopy(key, memo)] = deepcopy(value, memo) 0.388 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.388 y = copier(x, memo) 0.388 File "/usr/local/lib/python3.9/copy.py", line 205, in _deepcopy_list 0.388 append(deepcopy(a, memo)) 0.388 File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy 0.388 y = _reconstruct(x, memo, *rv) 0.388 File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct 0.388 state = deepcopy(state, memo) 0.388 File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy 0.388 y = copier(x, memo) 0.388 File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict 0.388 y[deepcopy(key, memo)] = deepcopy(value, memo) 0.388 File "/usr/local/lib/python3.9/copy.py", line 161, in deepcopy 0.388 rv = reductor(4) 0.388 TypeError: cannot pickle 'lxml.etree.XMLSchema' object ------ Dockerfile-v0.8.3:7 -------------------- 5 | RUN pip install pyangbind=='0.8.3' 6 | COPY public public 7 | >>> RUN pyang --plugindir /usr/local/lib/python3.9/site-packages/pyangbind/plugin -f pybind ./public/release/models/openconfig-extensions.yang ./public/release/models/types/openconfig-yang-types.yang ./public/release/models/types/openconfig-types.yang ./public/release/models/interfaces/openconfig-interfaces.yang ./public/release/models/optical-transport/openconfig-transport-types.yang ./public/release/models/platform/openconfig-platform-types.yang 8 | -------------------- ERROR: failed to solve: process "/bin/sh -c pyang --plugindir /usr/local/lib/python3.9/site-packages/pyangbind/plugin -f pybind ./public/release/models/openconfig-extensions.yang ./public/release/models/types/openconfig-yang-types.yang ./public/release/models/types/openconfig-types.yang ./public/release/models/interfaces/openconfig-interfaces.yang ./public/release/models/optical-transport/openconfig-transport-types.yang ./public/release/models/platform/openconfig-platform-types.yang" did not complete successfully: exit code: 1 ```

[gunhanoral]

Probably unrelated but when I tried to import from the file generated with docker-v0.8.2 using pyangbind v0.8.3 installed on my computer I got ModuleNotFoundError: No module named 'pyangbind.lib' error. After I downgrade to v0.8.2 I was able to import the same file without a problem.

[xavier-contreras]

Hello @gunhanoral , we are aware of some breakage and had yanked v0.8.3. @JoseIgnacioTamayo I noticed v0.8.3 was unyanked on PyPi, is the error here related to the original problem?

[JoseIgnacioTamayo]

Interesting.

I updated the release notes of v0.8.3. Indeed, files generated with 0.8.2 depend on bitarray while files generated with 0.8.3 depend on pyangbind...YANGBinary. So indeed 0.8.3 is not backwards compatible.

I un-yanked the release because I was able to use and generate files. I did not had the error reported in here. I will try to reproduce it and maybe fix it.

The fact that 0.8.3 is not backwards-compatible, grants it to be yanked again until fixed? Maybe...

[JoseIgnacioTamayo]

Yep, using oc_config_validate, I get the same error as reported. I yanked 0.8.3 again until:

• This error is fixed
• The release is backwards compatible (might be as simple as a importing bitarray library)

[fperrin]

For the second problem:

I tried to import from the file generated with docker-v0.8.2 using pyangbind v0.8.3 installed on my computer I got ModuleNotFoundError: No module named 'pyangbind.lib' error

The bindings are not self-contained, they import some classes from pyangbind. In order to properly support the YANG bytes type, I've added a new class to map the type, and bindings now reference that new class; that's why the older pyangbind version can't be used with bindings generated by the new version.

I'd be very careful before promising that bindings generated by pyangbind version X can be used on a machine running pyangbind version Y. It'll mean you can't add or remove functions or classes referenced from bindings, or rename them, move to a different module, accept extra or fewer arguments, change their behaviour, etc.

To be perfectly honest, I hadn't thought about that sort of compatibility. I always used pyangbind to generate bindings at application startup time.

[JoseIgnacioTamayo]

After some retries, I was able to use pyangbind 0.8.3 from source, and it worked ok. The error presented itself only when using the PyPi package.

I created a Package 0.8.3.post1 with a fresh build from source. @gunhanoral , could you please try again?

[gunhanoral]

Build completes successfully. Thanks! Also there's no problem with importing from the generated file.

[JoseIgnacioTamayo]

Great! So the issue was just the package 0.8.3 in PyPi. Something went wrong when I built and uploaded it. The latest package 0.8.3.post1 works as intended.