Use `pytest` and make GitHub Workflow for tests

smokestacklightnin commented 1 month ago

Changes in this PR:

Use pytest to run tests
Write a workflow to run the tests on the GitHub repository
Remove if __name__ == "__main__" section from tests because it is not necessary, or even run, when using pytest. Nontrivial functionality from those sections was preserved.
Use a matrix to run the tests with multiple Python versions
Remove enable_v2_behavior from tests that previously called it because tensorflow v2 is what is currently used.
Add xfail marks to classes and methods that have failing tests
Add .gitignore file to the project

smokestacklightnin commented 1 month ago

Test workflows are passing now that we xfail and skip classes with failing tests

embr commented 3 days ago

It looks like many of the tests have been annotated as xfail. All of these tests are passing internally, so I'm worried that there is something major broken in the pytest setup. Can you share the stack traces for the failing tests?

peytondmurray commented 2 days ago

@embr So I just ran pytest --runxfail now in a fresh virtual environment but was unable to run anything - I got a bunch of errors collecting tests. Every single failure looked like this:

_______________________________________________________________ ERROR collecting tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py _______________________________________________________________
ImportError while importing test module '/home/pdmurray/Desktop/workspace/model-analysis/tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tensorflow_model_analysis/addons/__init__.py:16: in <module>
    from tensorflow_model_analysis.addons import fairness
tensorflow_model_analysis/addons/fairness/__init__.py:16: in <module>
    from tensorflow_model_analysis.addons.fairness import metrics
tensorflow_model_analysis/addons/fairness/metrics/__init__.py:17: in <module>
    from tensorflow_model_analysis.addons.fairness.metrics import counterfactual_fairness
tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/__init__.py:17: in <module>
    from tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count import FlipCount
tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count.py:20: in <module>
    from tensorflow_model_analysis.metrics import binary_confusion_matrices
tensorflow_model_analysis/metrics/__init__.py:16: in <module>
    from tensorflow_model_analysis.metrics import bleu
tensorflow_model_analysis/metrics/bleu.py:27: in <module>
    from tensorflow_model_analysis.metrics import metric_types
tensorflow_model_analysis/metrics/metric_types.py:23: in <module>
    from tensorflow_model_analysis.api import types
tensorflow_model_analysis/api/types.py:25: in <module>
    from tensorflow_model_analysis.proto import metrics_for_slice_pb2
tensorflow_model_analysis/proto/metrics_for_slice_pb2.py:9: in <module>
    from google.protobuf import runtime_version as _runtime_version
E   ImportError: cannot import name 'runtime_version' from 'google.protobuf' (/home/pdmurray/.pyenv/versions/3.10.12/envs/tfma/lib/python3.10/site-packages/google/protobuf/__init__.py)

Not sure exactly what the problem is here but it seems like a bad dependency pin. At least according to this protobuf>=5 seems to be required for this to work, and that certainly seems to be the case looking at the protobuf git repo. At this point we are many major versions behind the current protobuf release, as are other dependencies - maybe the solution here is to update dependencies?

smokestacklightnin commented 2 days ago

@embr So I just ran pytest --runxfail now in a fresh virtual environment but was unable to run anything - I got a bunch of errors collecting tests. Every single failure looked like this:

_______________________________________________________________ ERROR collecting tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py _______________________________________________________________
ImportError while importing test module '/home/pdmurray/Desktop/workspace/model-analysis/tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tensorflow_model_analysis/addons/__init__.py:16: in <module>
    from tensorflow_model_analysis.addons import fairness
tensorflow_model_analysis/addons/fairness/__init__.py:16: in <module>
    from tensorflow_model_analysis.addons.fairness import metrics
tensorflow_model_analysis/addons/fairness/metrics/__init__.py:17: in <module>
    from tensorflow_model_analysis.addons.fairness.metrics import counterfactual_fairness
tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/__init__.py:17: in <module>
    from tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count import FlipCount
tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count.py:20: in <module>
    from tensorflow_model_analysis.metrics import binary_confusion_matrices
tensorflow_model_analysis/metrics/__init__.py:16: in <module>
    from tensorflow_model_analysis.metrics import bleu
tensorflow_model_analysis/metrics/bleu.py:27: in <module>
    from tensorflow_model_analysis.metrics import metric_types
tensorflow_model_analysis/metrics/metric_types.py:23: in <module>
    from tensorflow_model_analysis.api import types
tensorflow_model_analysis/api/types.py:25: in <module>
    from tensorflow_model_analysis.proto import metrics_for_slice_pb2
tensorflow_model_analysis/proto/metrics_for_slice_pb2.py:9: in <module>
    from google.protobuf import runtime_version as _runtime_version
E   ImportError: cannot import name 'runtime_version' from 'google.protobuf' (/home/pdmurray/.pyenv/versions/3.10.12/envs/tfma/lib/python3.10/site-packages/google/protobuf/__init__.py)

Not sure exactly what the problem is here but it seems like a bad dependency pin. At least according to this protobuf>=5 seems to be required for this to work, and that certainly seems to be the case looking at the protobuf git repo. At this point we are many major versions behind the current protobuf release, as are other dependencies - maybe the solution here is to update dependencies?

@peytondmurray what version of python did you use?

smokestacklightnin commented 2 days ago

@embr There are many many errors, so for space, I am including just the first two. The vast majority of them are TypeError: can't (safely) pickle generator objects that you will see below. I believe that fixing those will get rid of the vast majority of the errors.

This is in a fresh Python 3.9.20 virtual environment.

FF
=================================== FAILURES ===================================
_________________________ FlipCountTest.testFlipCount __________________________

o = CallableWrapperDoFn(<function Map.<locals>.<lambda> at 0x737e3c3feee0>)
enable_trace = True, use_zlib = False

    def dumps(o, enable_trace=True, use_zlib=False) -> bytes:
      """For internal use only; no backwards-compatibility guarantees."""
      with _pickle_lock:
        try:
>         s = dill.dumps(o, byref=settings['dill_byref'])

.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:376: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.venv39/lib/python3.9/site-packages/dill/_dill.py:265: in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
.venv39/lib/python3.9/site-packages/dill/_dill.py:259: in dump
    Pickler(file, protocol, **_kwds).dump(obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:445: in dump
    StockPickler.dump(self, obj)
/usr/lib/python3.9/pickle.py:487: in dump
    self.save(obj)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1410: in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1147: in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1410: in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1147: in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:931: in save_list
    self._batch_appends(obj)
/usr/lib/python3.9/pickle.py:955: in _batch_appends
    save(x)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:1002: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:931: in save_list
    self._batch_appends(obj)
/usr/lib/python3.9/pickle.py:958: in _batch_appends
    save(tmp[0])
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1056: in save_functor
    pickler.save_reduce(_create_ftype, (type(obj), obj.func, obj.args,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

unused_pickler = <dill._dill.Pickler object at 0x737e3c3fa760>
unused_obj = <generator object UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737e3c545b30>

    def _reject_generators(unused_pickler, unused_obj):
>     raise TypeError("can't (safely) pickle generator objects")
E     TypeError: can't (safely) pickle generator objects

.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:287: TypeError

During handling of the above exception, another exception occurred:

self = <tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest testMethod=testFlipCount>

    def testFlipCount(self):
      computations = flip_count.FlipCount(
          thresholds=[0.3],
          counterfactual_prediction_key='counterfactual_pred_key',
          example_id_key='example_id_key',
      ).computations(example_weighted=True)
      binary_confusion_matrix = computations[0]
      matrices = computations[1]
      metrics = computations[2]
      # TODO(b/171180441): Handle absence of ground truth labels in counterfactual
      # examples while computing flip count metrics.
      examples = [
          {
              'labels': None,
              'predictions': np.array([0.5]),
              'example_weights': np.array([1.0]),
              'features': {
                  'counterfactual_pred_key': np.array([0.7]),
                  'example_id_key': np.array(['id_1']),
              },
          },
          {
              'labels': None,
              'predictions': np.array([0.1, 0.7]),  # to test flattening
              'example_weights': np.array([3.0]),
              'features': {
                  'counterfactual_pred_key': np.array([1.0, 0.1]),
                  'example_id_key': np.array(['id_2']),
              },
          },
          {
              'labels': None,
              'predictions': np.array([0.5, 0.2]),
              'example_weights': np.array([2.0]),
              'features': {
                  'counterfactual_pred_key': np.array([0.2, 0.4]),
                  'example_id_key': np.array(['id_3']),
              },
          },
          {
              'labels': None,
              'predictions': np.array([0.2, 0.1]),
              'example_weights': np.array([1.0]),
              'features': {
                  'counterfactual_pred_key': np.array([0.4, 0.5]),
                  'example_id_key': np.array(['id_4']),
              },
          },
      ]

      with beam.Pipeline() as pipeline:
        # pylint: disable=no-value-for-parameter
        result = (
            pipeline
            | 'Create' >> beam.Create(examples)
            | 'Process' >> beam.Map(metric_util.to_standard_metric_inputs, True)
            | 'AddSlice' >> beam.Map(lambda x: ((), x))
            | 'ComputeBinaryConfusionMatrix'
            >> beam.CombinePerKey(binary_confusion_matrix.combiner)
            | 'ComputeMatrices'
            >> beam.Map(
                lambda x: (x[0], matrices.result(x[1]))
            )  # pyformat: ignore
            | 'ComputeMetrics' >> beam.Map(lambda x: (x[0], metrics.result(x[1])))
        )

        # pylint: enable=no-value-for-parameter

        def check_result(got):
          try:
            self.assertLen(got, 1)
            got_slice_key, got_metrics = got[0]
            self.assertEqual(got_slice_key, ())
            self.assertLen(got_metrics, 6)
            self.assertDictElementsAlmostEqual(
                got_metrics,
                {
                    metric_types.MetricKey(
                        name='flip_count/positive_to_negative@0.3',
                        example_weighted=True,
                    ): 5.0,
                    metric_types.MetricKey(
                        name='flip_count/negative_to_positive@0.3',
                        example_weighted=True,
                    ): 7.0,
                    metric_types.MetricKey(
                        name='flip_count/positive_examples_count@0.3',
                        example_weighted=True,
                    ): 6.0,
                    metric_types.MetricKey(
                        name='flip_count/negative_examples_count@0.3',
                        example_weighted=True,
                    ): 7.0,
                },
            )
            self.assertAllEqual(
                got_metrics[
                    metric_types.MetricKey(
                        name='flip_count/positive_to_negative_examples_ids@0.3',
                        example_weighted=True,
                    )
                ],
                np.array([['id_2'], ['id_3']]),
            )
            self.assertAllEqual(
                got_metrics[
                    metric_types.MetricKey(
                        name='flip_count/negative_to_positive_examples_ids@0.3',
                        example_weighted=True,
                    )
                ],
                np.array([['id_2'], ['id_3'], ['id_4']]),
            )
          except AssertionError as err:
            raise util.BeamAssertException(err)

>       util.assert_that(result, check_result, label='result')

tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py:151: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.venv39/lib/python3.9/site-packages/apache_beam/testing/util.py:309: in assert_that
    return actual | AssertThat()
.venv39/lib/python3.9/site-packages/apache_beam/pvalue.py:138: in __or__
    return self.pipeline.apply(ptransform, self)
.venv39/lib/python3.9/site-packages/apache_beam/pipeline.py:748: in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
.venv39/lib/python3.9/site-packages/apache_beam/runners/runner.py:191: in apply
    return self.apply_PTransform(transform, input, options)
.venv39/lib/python3.9/site-packages/apache_beam/runners/runner.py:195: in apply_PTransform
    return transform.expand(input)
.venv39/lib/python3.9/site-packages/apache_beam/testing/util.py:304: in expand
    return plain_actual | 'Match' >> Map(matcher)
.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py:2090: in Map
    pardo = FlatMap(wrapper, *args, **kwargs)
.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py:2033: in FlatMap
    pardo = ParDo(CallableWrapperDoFn(fn), *args, **kwargs)
.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py:1545: in __init__
    super().__init__(fn, *args, **kwargs)
.venv39/lib/python3.9/site-packages/apache_beam/transforms/ptransform.py:870: in __init__
    self.fn = pickler.loads(pickler.dumps(self.fn))
.venv39/lib/python3.9/site-packages/apache_beam/internal/pickler.py:43: in dumps
    return desired_pickle_lib.dumps(
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:380: in dumps
    s = dill.dumps(o, byref=settings['dill_byref'])
.venv39/lib/python3.9/site-packages/dill/_dill.py:265: in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
.venv39/lib/python3.9/site-packages/dill/_dill.py:259: in dump
    Pickler(file, protocol, **_kwds).dump(obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:445: in dump
    StockPickler.dump(self, obj)
/usr/lib/python3.9/pickle.py:487: in dump
    self.save(obj)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1410: in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1147: in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1410: in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1147: in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:931: in save_list
    self._batch_appends(obj)
/usr/lib/python3.9/pickle.py:955: in _batch_appends
    save(x)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:1002: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:931: in save_list
    self._batch_appends(obj)
/usr/lib/python3.9/pickle.py:958: in _batch_appends
    save(tmp[0])
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1056: in save_functor
    pickler.save_reduce(_create_ftype, (type(obj), obj.func, obj.args,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

unused_pickler = <dill._dill.Pickler object at 0x737e3c3e4cd0>
unused_obj = <generator object UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737e3c545b30>

    def _reject_generators(unused_pickler, unused_obj):
>     raise TypeError("can't (safely) pickle generator objects")
E     TypeError: can't (safely) pickle generator objects

.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:287: TypeError
----------------------------- Captured stderr call -----------------------------
T4: <class 'apache_beam.transforms.core.CallableWrapperDoFn'>
# T4
D2: <dict object at 0x737e3c39e380>
F1: <function Map.<locals>.<lambda> at 0x737e3c3feee0>
F2: <function _create_function at 0x737e69dde430>
# F2
T1: <class 'code'>
F2: <function _load_type at 0x737e69dde310>
# F2
# T1
B1: <built-in function getattr>
F2: <function _get_attr at 0x737e69ddee50>
# F2
# B1
M2: <module 'apache_beam.transforms.core' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py'>
F2: <function _import_module at 0x737e69ddef70>
# F2
# M2
Ce: <cell at 0x737e3c3e90a0: function object at 0x737e3c3dea60>
F2: <function _create_cell at 0x737e69dde820>
# F2
F1: <function FlipCountTest.testFlipCount.<locals>.check_result at 0x737e3c3dea60>
M2: <module 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test' from '/home/william/google-tensorflow-ci-cd/model-analysis/tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py'>
# M2
Ce: <cell at 0x737e4294ecd0: FlipCountTest object at 0x737deb136730>
T4: <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>
# T4
D2: <dict object at 0x737debad99c0>
T4: <class 'unittest.case._Outcome'>
# T4
D2: <dict object at 0x737e3c53d180>
T4: <class '_pytest.unittest.TestCaseFunction'>
# T4
D2: <dict object at 0x737deb0c1a80>
T4: <class '_pytest.mark.structures.NodeKeywords'>
# T4
D2: <dict object at 0x737e3c3f1b00>
T4: <class '_pytest.unittest.UnitTestCase'>
# T4
D2: <dict object at 0x737deb11b180>
D2: <dict object at 0x737e3c393c40>
T4: <class '_pytest.python.Module'>
# T4
D2: <dict object at 0x737e69fdd640>
D2: <dict object at 0x737e3c393b40>
T4: <class '_pytest.python.Package'>
# T4
D2: <dict object at 0x737e69fdab00>
D2: <dict object at 0x737e3c388140>
D2: <dict object at 0x737e69fd0240>
D2: <dict object at 0x737e3c403d00>
D2: <dict object at 0x737e6a50a080>
D2: <dict object at 0x737e3c3ea0c0>
D2: <dict object at 0x737e69fc85c0>
D2: <dict object at 0x737e42c89680>
D2: <dict object at 0x737e69fc6c00>
D2: <dict object at 0x737e3c38a8c0>
T4: <class '_pytest.main.Dir'>
# T4
D2: <dict object at 0x737e6a4fd6c0>
D2: <dict object at 0x737e3c38c300>
T4: <class '_pytest.main.Session'>
# T4
D2: <dict object at 0x737e6a80f640>
D2: <dict object at 0x737e3c4a6400>
D2: <dict object at 0x737e6a50a0c0>
# D2
# D2
T4: <class '_pytest.stash.Stash'>
# T4
D2: <dict object at 0x737e3c36c740>
D2: <dict object at 0x737e6a7b4a40>
# D2
# D2
T4: <class 'pluggy._tracing.TagTracerSub'>
# T4
D2: <dict object at 0x737e6a80f8c0>
T4: <class 'pluggy._tracing.TagTracer'>
# T4
D2: <dict object at 0x737e6a8d6940>
D2: <dict object at 0x737e6a8dd200>
# D2
# D2
# D2
T4: <class 'pathlib.PosixPath'>
# T4
D2: <dict object at 0x737e3c53dc40>
# D2
D2: <dict object at 0x737deb0ccbc0>
D2: <dict object at 0x737e3c370280>
D2: <dict object at 0x737deb0ccb80>
# D2
# D2
D2: <dict object at 0x737e3c378900>
D2: <dict object at 0x737deb0ccc00>
# D2
# D2
D2: <dict object at 0x737deb0ccb00>
D2: <dict object at 0x737deb0ccd00>
T1: <class 'dict'>
# T1
T1: <class 'list'>
# T1
T1: <class 'tuple'>
# T1
T1: <class 'set'>
# T1
T1: <class 'frozenset'>
# T1
T1: <class 'str'>
# T1
# D2
# D2
Me: <bound method FlipCountTest.testFlipCountWitEvalConfig of <tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest testMethod=testFlipCountWitEvalConfig>>
T1: <class 'method'>
# T1
F1: <function FlipCountTest.testFlipCountWitEvalConfig at 0x737deb0c5430>
D2: <dict object at 0x737debac5a40>
# D2
# F1
# Me
T4: <class '_pytest.fixtures.FuncFixtureInfo'>
# T4
D2: <dict object at 0x737e3c378940>
D2: <dict object at 0x737deb0cccc0>
T4: <class '_pytest.fixtures.FixtureDef'>
# T4
D2: <dict object at 0x737deb11b2c0>
F1: <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737deb0c5790>
M2: <module '_pytest.unittest' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/_pytest/unittest.py'>
# M2
Ce: <cell at 0x737deb1045b0: method object at 0x737deb0cc780>
Me: <bound method TestCase.doClassCleanups of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
F1: <function TestCase.doClassCleanups at 0x737e6af7de50>
M2: <module 'unittest.case' from '/usr/lib/python3.9/unittest/case.py'>
# M2
D2: <dict object at 0x737e3c3f1580>
# D2
# F1
# Me
# Ce
Ce: <cell at 0x737deb104b80: function object at 0x737deb0c5700>
F1: <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.process_teardown_exceptions at 0x737deb0c5700>
Ce: <cell at 0x737deb104d60: type object at 0x3c6f2630>
# Ce
D2: <dict object at 0x737e3c3f1400>
# D2
# F1
# Ce
Ce: <cell at 0x737deb104580: method object at 0x737debac5440>
Me: <bound method TensorFlowTestCase.setUpClass of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
F1: <function TensorFlowTestCase.setUpClass at 0x737df0173e50>
M2: <module 'tensorflow.python.framework.test_util' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/tensorflow/python/framework/test_util.py'>
# M2
Ce: <cell at 0x737df0232910: type object at 0x3a778e30>
T4: <class 'tensorflow.python.framework.test_util.TensorFlowTestCase'>
# T4
# Ce
D2: <dict object at 0x737e3c3f1ec0>
# D2
# F1
# Me
# Ce
Ce: <cell at 0x737deb1042e0: method object at 0x737debad9700>
Me: <bound method TestCase.tearDownClass of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
F1: <function TestCase.tearDownClass at 0x737e6af7d280>
D2: <dict object at 0x737e3c3f1840>
# D2
# F1
# Me
# Ce
D2: <dict object at 0x737e3c43a640>
# D2
# F1
T4: <enum 'Scope'>
# T4
Fu: functools.partial(<function _teardown_yield_fixture at 0x737e6ab07040>, <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737deb0c5790>, <generator object UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737e3c545b30>)
F2: <function _create_ftype at 0x737e69dde4c0>
# F2
T1: <class 'functools.partial'>
# T1
F2: <function _teardown_yield_fixture at 0x737e6ab07040>
# F2
------------------------------ Captured log call -------------------------------
INFO     dill:_dill.py:1352 T4: <class 'apache_beam.transforms.core.CallableWrapperDoFn'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c39e380>
INFO     dill:_dill.py:1384 F1: <function Map.<locals>.<lambda> at 0x737e3c3feee0>
INFO     dill:_dill.py:1434 F2: <function _create_function at 0x737e69dde430>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1308 T1: <class 'code'>
INFO     dill:_dill.py:1434 F2: <function _load_type at 0x737e69dde310>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1074 B1: <built-in function getattr>
INFO     dill:_dill.py:1434 F2: <function _get_attr at 0x737e69ddee50>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1085 # B1
INFO     dill:dill_pickler.py:303 M2: <module 'apache_beam.transforms.core' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py'>
INFO     dill:_dill.py:1434 F2: <function _import_module at 0x737e69ddef70>
INFO     dill:_dill.py:1436 # F2
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:1145 Ce: <cell at 0x737e3c3e90a0: function object at 0x737e3c3dea60>
INFO     dill:_dill.py:1434 F2: <function _create_cell at 0x737e69dde820>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1384 F1: <function FlipCountTest.testFlipCount.<locals>.check_result at 0x737e3c3dea60>
INFO     dill:dill_pickler.py:303 M2: <module 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test' from '/home/william/google-tensorflow-ci-cd/model-analysis/tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py'>
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:1145 Ce: <cell at 0x737e4294ecd0: FlipCountTest object at 0x737deb136730>
INFO     dill:_dill.py:1352 T4: <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737debad99c0>
INFO     dill:_dill.py:1352 T4: <class 'unittest.case._Outcome'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c53d180>
INFO     dill:_dill.py:1352 T4: <class '_pytest.unittest.TestCaseFunction'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0c1a80>
INFO     dill:_dill.py:1352 T4: <class '_pytest.mark.structures.NodeKeywords'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3f1b00>
INFO     dill:_dill.py:1352 T4: <class '_pytest.unittest.UnitTestCase'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb11b180>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c393c40>
INFO     dill:_dill.py:1352 T4: <class '_pytest.python.Module'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fdd640>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c393b40>
INFO     dill:_dill.py:1352 T4: <class '_pytest.python.Package'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fdab00>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c388140>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fd0240>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c403d00>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a50a080>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3ea0c0>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fc85c0>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e42c89680>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fc6c00>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c38a8c0>
INFO     dill:_dill.py:1352 T4: <class '_pytest.main.Dir'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a4fd6c0>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c38c300>
INFO     dill:_dill.py:1352 T4: <class '_pytest.main.Session'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a80f640>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c4a6400>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a50a0c0>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1352 T4: <class '_pytest.stash.Stash'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c36c740>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a7b4a40>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1352 T4: <class 'pluggy._tracing.TagTracerSub'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a80f8c0>
INFO     dill:_dill.py:1352 T4: <class 'pluggy._tracing.TagTracer'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a8d6940>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a8dd200>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1352 T4: <class 'pathlib.PosixPath'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c53dc40>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0ccbc0>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c370280>
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0ccb80>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c378900>
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0ccc00>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0ccb00>
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0ccd00>
INFO     dill:_dill.py:1308 T1: <class 'dict'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1308 T1: <class 'list'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1308 T1: <class 'tuple'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1308 T1: <class 'set'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1308 T1: <class 'frozenset'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1308 T1: <class 'str'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1094 Me: <bound method FlipCountTest.testFlipCountWitEvalConfig of <tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest testMethod=testFlipCountWitEvalConfig>>
INFO     dill:_dill.py:1308 T1: <class 'method'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1384 F1: <function FlipCountTest.testFlipCountWitEvalConfig at 0x737deb0c5430>
INFO     dill:_dill.py:908 D2: <dict object at 0x737debac5a40>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1100 # Me
INFO     dill:_dill.py:1352 T4: <class '_pytest.fixtures.FuncFixtureInfo'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c378940>
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0cccc0>
INFO     dill:_dill.py:1352 T4: <class '_pytest.fixtures.FixtureDef'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb11b2c0>
INFO     dill:_dill.py:1384 F1: <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737deb0c5790>
INFO     dill:dill_pickler.py:303 M2: <module '_pytest.unittest' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/_pytest/unittest.py'>
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb1045b0: method object at 0x737deb0cc780>
INFO     dill:_dill.py:1094 Me: <bound method TestCase.doClassCleanups of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
INFO     dill:_dill.py:1384 F1: <function TestCase.doClassCleanups at 0x737e6af7de50>
INFO     dill:dill_pickler.py:303 M2: <module 'unittest.case' from '/usr/lib/python3.9/unittest/case.py'>
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3f1580>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1100 # Me
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb104b80: function object at 0x737deb0c5700>
INFO     dill:_dill.py:1384 F1: <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.process_teardown_exceptions at 0x737deb0c5700>
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb104d60: type object at 0x3c6f2630>
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3f1400>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb104580: method object at 0x737debac5440>
INFO     dill:_dill.py:1094 Me: <bound method TensorFlowTestCase.setUpClass of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
INFO     dill:_dill.py:1384 F1: <function TensorFlowTestCase.setUpClass at 0x737df0173e50>
INFO     dill:dill_pickler.py:303 M2: <module 'tensorflow.python.framework.test_util' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/tensorflow/python/framework/test_util.py'>
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:1145 Ce: <cell at 0x737df0232910: type object at 0x3a778e30>
INFO     dill:_dill.py:1352 T4: <class 'tensorflow.python.framework.test_util.TensorFlowTestCase'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3f1ec0>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1100 # Me
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb1042e0: method object at 0x737debad9700>
INFO     dill:_dill.py:1094 Me: <bound method TestCase.tearDownClass of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
INFO     dill:_dill.py:1384 F1: <function TestCase.tearDownClass at 0x737e6af7d280>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3f1840>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1100 # Me
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c43a640>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1352 T4: <enum 'Scope'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:1055 Fu: functools.partial(<function _teardown_yield_fixture at 0x737e6ab07040>, <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737deb0c5790>, <generator object UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737e3c545b30>)
INFO     dill:_dill.py:1434 F2: <function _create_ftype at 0x737e69dde4c0>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1308 T1: <class 'functools.partial'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1434 F2: <function _teardown_yield_fixture at 0x737e6ab07040>
INFO     dill:_dill.py:1436 # F2
___________________ FlipCountTest.testFlipCountWitEvalConfig ___________________

o = CallableWrapperDoFn(<function Map.<locals>.<lambda> at 0x737e3c16e430>)
enable_trace = True, use_zlib = False

    def dumps(o, enable_trace=True, use_zlib=False) -> bytes:
      """For internal use only; no backwards-compatibility guarantees."""
      with _pickle_lock:
        try:
>         s = dill.dumps(o, byref=settings['dill_byref'])

.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:376: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.venv39/lib/python3.9/site-packages/dill/_dill.py:265: in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
.venv39/lib/python3.9/site-packages/dill/_dill.py:259: in dump
    Pickler(file, protocol, **_kwds).dump(obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:445: in dump
    StockPickler.dump(self, obj)
/usr/lib/python3.9/pickle.py:487: in dump
    self.save(obj)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1410: in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1147: in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1410: in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1147: in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:931: in save_list
    self._batch_appends(obj)
/usr/lib/python3.9/pickle.py:955: in _batch_appends
    save(x)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:1002: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:931: in save_list
    self._batch_appends(obj)
/usr/lib/python3.9/pickle.py:958: in _batch_appends
    save(tmp[0])
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1056: in save_functor
    pickler.save_reduce(_create_ftype, (type(obj), obj.func, obj.args,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

unused_pickler = <dill._dill.Pickler object at 0x737e39938d00>
unused_obj = <generator object UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737e3c545b30>

    def _reject_generators(unused_pickler, unused_obj):
>     raise TypeError("can't (safely) pickle generator objects")
E     TypeError: can't (safely) pickle generator objects

.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:287: TypeError

During handling of the above exception, another exception occurred:

self = <tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest testMethod=testFlipCountWitEvalConfig>

    def testFlipCountWitEvalConfig(self):
      eval_config = text_format.Parse(
          """
          model_specs: {
            name: "original"
          }
          model_specs: {
            name: "counterfactual"
            is_baseline: true
          }
          """,
          config_pb2.EvalConfig(),
      )
      computations = flip_count.FlipCount(
          thresholds=[0.3], example_id_key='example_id_key'
      ).computations(
          eval_config=eval_config,
          example_weighted=True,
          model_names=['original', 'counterfactual'],
          output_names=[''],
      )
      binary_confusion_matrix = computations[0]
      matrices = computations[1]
      metrics = computations[2]
      original_model_name = 'original'
      counterfactual_model_name = 'counterfactual'
      examples = [
          {
              'labels': None,
              'predictions': {
                  original_model_name: np.array([0.5]),
                  counterfactual_model_name: np.array([0.7]),
              },
              'example_weights': np.array([1.0]),
              'features': {
                  'example_id_key': np.array(['id_1']),
              },
          },
          {
              'labels': None,
              'predictions': {
                  original_model_name: np.array([0.1, 0.7]),  # to test flattening
                  counterfactual_model_name: np.array([1.0, 0.1]),
              },
              'example_weights': np.array([3.0]),
              'features': {
                  'example_id_key': np.array(['id_2']),
              },
          },
          {
              'labels': None,
              'predictions': {
                  original_model_name: np.array([0.5, 0.2]),
                  counterfactual_model_name: np.array([0.2, 0.4]),
              },
              'example_weights': np.array([2.0]),
              'features': {
                  'example_id_key': np.array(['id_3']),
              },
          },
          {
              'labels': None,
              'predictions': {
                  original_model_name: np.array([0.2, 0.1]),
                  counterfactual_model_name: np.array([0.4, 0.5]),
              },
              'example_weights': np.array([1.0]),
              'features': {
                  'example_id_key': np.array(['id_4']),
              },
          },
      ]

      with beam.Pipeline() as pipeline:
        # pylint: disable=no-value-for-parameter
        result = (
            pipeline
            | 'Create' >> beam.Create(examples)
            | 'Process' >> beam.Map(metric_util.to_standard_metric_inputs, True)
            | 'AddSlice' >> beam.Map(lambda x: ((), x))
            | 'ComputeBinaryConfusionMatrix'
            >> beam.CombinePerKey(binary_confusion_matrix.combiner)
            | 'ComputeMatrices'
            >> beam.Map(lambda x: (x[0], matrices.result(x[1])))
            | 'ComputeMetrics' >> beam.Map(lambda x: (x[0], metrics.result(x[1])))
        )

        # pylint: enable=no-value-for-parameter
        def check_result(got):
          try:
            self.assertLen(got, 1)
            got_slice_key, got_metrics = got[0]
            self.assertEqual(got_slice_key, ())
            self.assertLen(got_metrics, 6)
            self.assertDictElementsAlmostEqual(
                got_metrics,
                {
                    metric_types.MetricKey(
                        name='flip_count/positive_to_negative@0.3',
                        model_name='original',
                        example_weighted=True,
                    ): 5.0,
                    metric_types.MetricKey(
                        name='flip_count/negative_to_positive@0.3',
                        model_name='original',
                        example_weighted=True,
                    ): 7.0,
                    metric_types.MetricKey(
                        name='flip_count/positive_examples_count@0.3',
                        model_name='original',
                        example_weighted=True,
                    ): 6.0,
                    metric_types.MetricKey(
                        name='flip_count/negative_examples_count@0.3',
                        model_name='original',
                        example_weighted=True,
                    ): 7.0,
                },
            )
            self.assertAllEqual(
                got_metrics[
                    metric_types.MetricKey(
                        name='flip_count/positive_to_negative_examples_ids@0.3',
                        model_name='original',
                        example_weighted=True,
                    )
                ],
                np.array([['id_2'], ['id_3']]),
            )
            self.assertAllEqual(
                got_metrics[
                    metric_types.MetricKey(
                        name='flip_count/negative_to_positive_examples_ids@0.3',
                        model_name='original',
                        example_weighted=True,
                    )
                ],
                np.array([['id_2'], ['id_3'], ['id_4']]),
            )
          except AssertionError as err:
            raise util.BeamAssertException(err)

>       util.assert_that(result, check_result, label='result')

tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py:295: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.venv39/lib/python3.9/site-packages/apache_beam/testing/util.py:309: in assert_that
    return actual | AssertThat()
.venv39/lib/python3.9/site-packages/apache_beam/pvalue.py:138: in __or__
    return self.pipeline.apply(ptransform, self)
.venv39/lib/python3.9/site-packages/apache_beam/pipeline.py:748: in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
.venv39/lib/python3.9/site-packages/apache_beam/runners/runner.py:191: in apply
    return self.apply_PTransform(transform, input, options)
.venv39/lib/python3.9/site-packages/apache_beam/runners/runner.py:195: in apply_PTransform
    return transform.expand(input)
.venv39/lib/python3.9/site-packages/apache_beam/testing/util.py:304: in expand
    return plain_actual | 'Match' >> Map(matcher)
.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py:2090: in Map
    pardo = FlatMap(wrapper, *args, **kwargs)
.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py:2033: in FlatMap
    pardo = ParDo(CallableWrapperDoFn(fn), *args, **kwargs)
.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py:1545: in __init__
    super().__init__(fn, *args, **kwargs)
.venv39/lib/python3.9/site-packages/apache_beam/transforms/ptransform.py:870: in __init__
    self.fn = pickler.loads(pickler.dumps(self.fn))
.venv39/lib/python3.9/site-packages/apache_beam/internal/pickler.py:43: in dumps
    return desired_pickle_lib.dumps(
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:380: in dumps
    s = dill.dumps(o, byref=settings['dill_byref'])
.venv39/lib/python3.9/site-packages/dill/_dill.py:265: in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
.venv39/lib/python3.9/site-packages/dill/_dill.py:259: in dump
    Pickler(file, protocol, **_kwds).dump(obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:445: in dump
    StockPickler.dump(self, obj)
/usr/lib/python3.9/pickle.py:487: in dump
    self.save(obj)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1410: in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1147: in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1410: in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1147: in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:931: in save_list
    self._batch_appends(obj)
/usr/lib/python3.9/pickle.py:955: in _batch_appends
    save(x)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:1002: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
/usr/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:348: in new_save_module_dict
    return old_save_module_dict(pickler, obj)
.venv39/lib/python3.9/site-packages/dill/_dill.py:912: in save_module_dict
    StockPickler.save_dict(pickler, obj)
/usr/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
/usr/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:931: in save_list
    self._batch_appends(obj)
/usr/lib/python3.9/pickle.py:958: in _batch_appends
    save(tmp[0])
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
.venv39/lib/python3.9/site-packages/dill/_dill.py:1056: in save_functor
    pickler.save_reduce(_create_ftype, (type(obj), obj.func, obj.args,
/usr/lib/python3.9/pickle.py:692: in save_reduce
    save(args)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:901: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
/usr/lib/python3.9/pickle.py:886: in save_tuple
    save(element)
/usr/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

unused_pickler = <dill._dill.Pickler object at 0x737e3971b670>
unused_obj = <generator object UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737e3c545b30>

    def _reject_generators(unused_pickler, unused_obj):
>     raise TypeError("can't (safely) pickle generator objects")
E     TypeError: can't (safely) pickle generator objects

.venv39/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:287: TypeError
----------------------------- Captured stderr call -----------------------------
T4: <class 'apache_beam.transforms.core.CallableWrapperDoFn'>
# T4
D2: <dict object at 0x737e399dc300>
F1: <function Map.<locals>.<lambda> at 0x737e3c16e430>
F2: <function _create_function at 0x737e69dde430>
# F2
T1: <class 'code'>
F2: <function _load_type at 0x737e69dde310>
# F2
# T1
B1: <built-in function getattr>
F2: <function _get_attr at 0x737e69ddee50>
# F2
# B1
M2: <module 'apache_beam.transforms.core' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py'>
F2: <function _import_module at 0x737e69ddef70>
# F2
# M2
Ce: <cell at 0x737e39938610: function object at 0x737e399894c0>
F2: <function _create_cell at 0x737e69dde820>
# F2
F1: <function FlipCountTest.testFlipCountWitEvalConfig.<locals>.check_result at 0x737e399894c0>
M2: <module 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test' from '/home/william/google-tensorflow-ci-cd/model-analysis/tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py'>
# M2
Ce: <cell at 0x737e3c3f3940: FlipCountTest object at 0x737deb136dc0>
T4: <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>
# T4
D2: <dict object at 0x737deb0ccb00>
T4: <class 'unittest.case._Outcome'>
# T4
D2: <dict object at 0x737e3c1b00c0>
T4: <class '_pytest.unittest.TestCaseFunction'>
# T4
D2: <dict object at 0x737deb0ccbc0>
T4: <class '_pytest.mark.structures.NodeKeywords'>
# T4
D2: <dict object at 0x737e6a4ff2c0>
T4: <class '_pytest.unittest.UnitTestCase'>
# T4
D2: <dict object at 0x737deb11b180>
D2: <dict object at 0x737e42c8c640>
T4: <class '_pytest.python.Module'>
# T4
D2: <dict object at 0x737e69fdd640>
D2: <dict object at 0x737e3c1df980>
T4: <class '_pytest.python.Package'>
# T4
D2: <dict object at 0x737e69fdab00>
D2: <dict object at 0x737e42cff040>
D2: <dict object at 0x737e69fd0240>
D2: <dict object at 0x737e395ba040>
D2: <dict object at 0x737e6a50a080>
D2: <dict object at 0x737e395a9900>
D2: <dict object at 0x737e69fc85c0>
D2: <dict object at 0x737e399472c0>
D2: <dict object at 0x737e69fc6c00>
D2: <dict object at 0x737e39947c00>
T4: <class '_pytest.main.Dir'>
# T4
D2: <dict object at 0x737e6a4fd6c0>
D2: <dict object at 0x737e39976580>
T4: <class '_pytest.main.Session'>
# T4
D2: <dict object at 0x737e6a80f640>
D2: <dict object at 0x737e399721c0>
D2: <dict object at 0x737e6a50a0c0>
# D2
# D2
T4: <class '_pytest.stash.Stash'>
# T4
D2: <dict object at 0x737e39972d40>
D2: <dict object at 0x737e6a7b4a40>
# D2
# D2
T4: <class 'pluggy._tracing.TagTracerSub'>
# T4
D2: <dict object at 0x737e6a80f8c0>
T4: <class 'pluggy._tracing.TagTracer'>
# T4
D2: <dict object at 0x737e6a8d6940>
D2: <dict object at 0x737e6a8dd200>
# D2
# D2
# D2
T4: <class 'pathlib.PosixPath'>
# T4
D2: <dict object at 0x737e3c53dc40>
# D2
D2: <dict object at 0x737deb0c1a80>
D2: <dict object at 0x737e3996c040>
D2: <dict object at 0x737deb0c1a40>
# D2
# D2
D2: <dict object at 0x737e39967b40>
D2: <dict object at 0x737e4ec16c40>
T4: <class '_pytest.stash.StashKey'>
# T4
T4: <class '_pytest.skipping.Xfail'>
# T4
D2: <dict object at 0x737e39967480>
# D2
D2: <dict object at 0x737e3c53d540>
# D2
# D2
# D2
T4: <class '_pytest.fixtures.FuncFixtureInfo'>
# T4
D2: <dict object at 0x737e39967a80>
D2: <dict object at 0x737debac5dc0>
T4: <class '_pytest.fixtures.FixtureDef'>
# T4
D2: <dict object at 0x737deb11b2c0>
F1: <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737deb0c5790>
M2: <module '_pytest.unittest' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/_pytest/unittest.py'>
# M2
Ce: <cell at 0x737deb1045b0: method object at 0x737deb0cc780>
Me: <bound method TestCase.doClassCleanups of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
T1: <class 'method'>
# T1
F1: <function TestCase.doClassCleanups at 0x737e6af7de50>
M2: <module 'unittest.case' from '/usr/lib/python3.9/unittest/case.py'>
# M2
D2: <dict object at 0x737e3c3f1580>
# D2
# F1
# Me
# Ce
Ce: <cell at 0x737deb104b80: function object at 0x737deb0c5700>
F1: <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.process_teardown_exceptions at 0x737deb0c5700>
Ce: <cell at 0x737deb104d60: type object at 0x3c6f2630>
# Ce
D2: <dict object at 0x737e3c3f1400>
# D2
# F1
# Ce
Ce: <cell at 0x737deb104580: method object at 0x737debac5440>
Me: <bound method TensorFlowTestCase.setUpClass of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
F1: <function TensorFlowTestCase.setUpClass at 0x737df0173e50>
M2: <module 'tensorflow.python.framework.test_util' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/tensorflow/python/framework/test_util.py'>
# M2
Ce: <cell at 0x737df0232910: type object at 0x3a778e30>
T4: <class 'tensorflow.python.framework.test_util.TensorFlowTestCase'>
# T4
# Ce
D2: <dict object at 0x737e3c3f1ec0>
# D2
# F1
# Me
# Ce
Ce: <cell at 0x737deb1042e0: method object at 0x737debad9700>
Me: <bound method TestCase.tearDownClass of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
F1: <function TestCase.tearDownClass at 0x737e6af7d280>
D2: <dict object at 0x737e3c3f1840>
# D2
# F1
# Me
# Ce
D2: <dict object at 0x737e3c43a640>
# D2
# F1
T4: <enum 'Scope'>
# T4
Fu: functools.partial(<function _teardown_yield_fixture at 0x737e6ab07040>, <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737deb0c5790>, <generator object UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737e3c545b30>)
F2: <function _create_ftype at 0x737e69dde4c0>
# F2
T1: <class 'functools.partial'>
# T1
F2: <function _teardown_yield_fixture at 0x737e6ab07040>
# F2
------------------------------ Captured log call -------------------------------
INFO     dill:_dill.py:1352 T4: <class 'apache_beam.transforms.core.CallableWrapperDoFn'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e399dc300>
INFO     dill:_dill.py:1384 F1: <function Map.<locals>.<lambda> at 0x737e3c16e430>
INFO     dill:_dill.py:1434 F2: <function _create_function at 0x737e69dde430>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1308 T1: <class 'code'>
INFO     dill:_dill.py:1434 F2: <function _load_type at 0x737e69dde310>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1074 B1: <built-in function getattr>
INFO     dill:_dill.py:1434 F2: <function _get_attr at 0x737e69ddee50>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1085 # B1
INFO     dill:dill_pickler.py:303 M2: <module 'apache_beam.transforms.core' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/apache_beam/transforms/core.py'>
INFO     dill:_dill.py:1434 F2: <function _import_module at 0x737e69ddef70>
INFO     dill:_dill.py:1436 # F2
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:1145 Ce: <cell at 0x737e39938610: function object at 0x737e399894c0>
INFO     dill:_dill.py:1434 F2: <function _create_cell at 0x737e69dde820>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1384 F1: <function FlipCountTest.testFlipCountWitEvalConfig.<locals>.check_result at 0x737e399894c0>
INFO     dill:dill_pickler.py:303 M2: <module 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test' from '/home/william/google-tensorflow-ci-cd/model-analysis/tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py'>
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:1145 Ce: <cell at 0x737e3c3f3940: FlipCountTest object at 0x737deb136dc0>
INFO     dill:_dill.py:1352 T4: <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0ccb00>
INFO     dill:_dill.py:1352 T4: <class 'unittest.case._Outcome'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c1b00c0>
INFO     dill:_dill.py:1352 T4: <class '_pytest.unittest.TestCaseFunction'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0ccbc0>
INFO     dill:_dill.py:1352 T4: <class '_pytest.mark.structures.NodeKeywords'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a4ff2c0>
INFO     dill:_dill.py:1352 T4: <class '_pytest.unittest.UnitTestCase'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb11b180>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e42c8c640>
INFO     dill:_dill.py:1352 T4: <class '_pytest.python.Module'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fdd640>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c1df980>
INFO     dill:_dill.py:1352 T4: <class '_pytest.python.Package'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fdab00>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e42cff040>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fd0240>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e395ba040>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a50a080>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e395a9900>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fc85c0>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e399472c0>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e69fc6c00>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e39947c00>
INFO     dill:_dill.py:1352 T4: <class '_pytest.main.Dir'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a4fd6c0>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e39976580>
INFO     dill:_dill.py:1352 T4: <class '_pytest.main.Session'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a80f640>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e399721c0>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a50a0c0>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1352 T4: <class '_pytest.stash.Stash'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e39972d40>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a7b4a40>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1352 T4: <class 'pluggy._tracing.TagTracerSub'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a80f8c0>
INFO     dill:_dill.py:1352 T4: <class 'pluggy._tracing.TagTracer'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a8d6940>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e6a8dd200>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1352 T4: <class 'pathlib.PosixPath'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c53dc40>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0c1a80>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3996c040>
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb0c1a40>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:908 D2: <dict object at 0x737e39967b40>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e4ec16c40>
INFO     dill:_dill.py:1352 T4: <class '_pytest.stash.StashKey'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:1352 T4: <class '_pytest.skipping.Xfail'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e39967480>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c53d540>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1352 T4: <class '_pytest.fixtures.FuncFixtureInfo'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737e39967a80>
INFO     dill:_dill.py:908 D2: <dict object at 0x737debac5dc0>
INFO     dill:_dill.py:1352 T4: <class '_pytest.fixtures.FixtureDef'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:908 D2: <dict object at 0x737deb11b2c0>
INFO     dill:_dill.py:1384 F1: <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737deb0c5790>
INFO     dill:dill_pickler.py:303 M2: <module '_pytest.unittest' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/_pytest/unittest.py'>
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb1045b0: method object at 0x737deb0cc780>
INFO     dill:_dill.py:1094 Me: <bound method TestCase.doClassCleanups of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
INFO     dill:_dill.py:1308 T1: <class 'method'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1384 F1: <function TestCase.doClassCleanups at 0x737e6af7de50>
INFO     dill:dill_pickler.py:303 M2: <module 'unittest.case' from '/usr/lib/python3.9/unittest/case.py'>
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3f1580>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1100 # Me
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb104b80: function object at 0x737deb0c5700>
INFO     dill:_dill.py:1384 F1: <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.process_teardown_exceptions at 0x737deb0c5700>
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb104d60: type object at 0x3c6f2630>
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3f1400>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb104580: method object at 0x737debac5440>
INFO     dill:_dill.py:1094 Me: <bound method TensorFlowTestCase.setUpClass of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
INFO     dill:_dill.py:1384 F1: <function TensorFlowTestCase.setUpClass at 0x737df0173e50>
INFO     dill:dill_pickler.py:303 M2: <module 'tensorflow.python.framework.test_util' from '/home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/tensorflow/python/framework/test_util.py'>
INFO     dill:dill_pickler.py:307 # M2
INFO     dill:_dill.py:1145 Ce: <cell at 0x737df0232910: type object at 0x3a778e30>
INFO     dill:_dill.py:1352 T4: <class 'tensorflow.python.framework.test_util.TensorFlowTestCase'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3f1ec0>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1100 # Me
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:1145 Ce: <cell at 0x737deb1042e0: method object at 0x737debad9700>
INFO     dill:_dill.py:1094 Me: <bound method TestCase.tearDownClass of <class 'tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count_test.FlipCountTest'>>
INFO     dill:_dill.py:1384 F1: <function TestCase.tearDownClass at 0x737e6af7d280>
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c3f1840>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1100 # Me
INFO     dill:_dill.py:1148 # Ce
INFO     dill:_dill.py:908 D2: <dict object at 0x737e3c43a640>
INFO     dill:_dill.py:913 # D2
INFO     dill:_dill.py:1432 # F1
INFO     dill:_dill.py:1352 T4: <enum 'Scope'>
INFO     dill:_dill.py:1357 # T4
INFO     dill:_dill.py:1055 Fu: functools.partial(<function _teardown_yield_fixture at 0x737e6ab07040>, <function UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737deb0c5790>, <generator object UnitTestCase._register_unittest_setup_class_fixture.<locals>.unittest_setup_class_fixture at 0x737e3c545b30>)
INFO     dill:_dill.py:1434 F2: <function _create_ftype at 0x737e69dde4c0>
INFO     dill:_dill.py:1436 # F2
INFO     dill:_dill.py:1308 T1: <class 'functools.partial'>
INFO     dill:_dill.py:1310 # T1
INFO     dill:_dill.py:1434 F2: <function _teardown_yield_fixture at 0x737e6ab07040>
INFO     dill:_dill.py:1436 # F2
=============================== warnings summary ===============================
.venv39/lib/python3.9/site-packages/jupyter_client/connect.py:22
  /home/william/google-tensorflow-ci-cd/model-analysis/.venv39/lib/python3.9/site-packages/jupyter_client/connect.py:22: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
  given by the platformdirs library.  To remove this warning and
  see the appropriate new directories, set the environment variable
  `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
  The use of platformdirs will be the default in `jupyter_core` v6
    from jupyter_core.paths import jupyter_data_dir, jupyter_runtime_dir, secure_write

tensorflow_model_analysis/experimental/dataframe.py:19
  /home/william/google-tensorflow-ci-cd/model-analysis/tensorflow_model_analysis/experimental/dataframe.py:19: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
    logging.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py::FlipCountTest::testFlipCount
FAILED tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py::FlipCountTest::testFlipCountWitEvalConfig
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 2 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2 failed, 2 warnings in 7.54s

peytondmurray commented 2 days ago

@peytondmurray what version of python did you use?

3.10.12

smokestacklightnin commented 2 days ago

3.10.12

Interesting. I was able to run the tests using a fresh 3.10.12 virtual environment and get basically the same pickle errors on my machine. I get the following errors (again only first two shown) with 3.11.10:


==================================== ERRORS ====================================
_ ERROR collecting tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py _
<frozen importlib._bootstrap_external>:940: in exec_module
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
tensorflow_model_analysis/__init__.py:30: in <module>
    from tensorflow_model_analysis.sdk import *
tensorflow_model_analysis/sdk.py:60: in <module>
    from tensorflow_model_analysis.proto.config_pb2 import AggregationOptions
tensorflow_model_analysis/proto/config_pb2.py:35: in <module>
    _descriptor.EnumValueDescriptor(
.venv311/lib/python3.11/site-packages/google/protobuf/descriptor.py:789: in __new__
    _message.Message._CheckCalledFromGeneratedFile()
E   TypeError: Descriptors cannot be created directly.
E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E   If you cannot immediately regenerate your protos, some other possible workarounds are:
E    1. Downgrade the protobuf package to 3.20.x or lower.
E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E   
E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
_ ERROR collecting tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_rate_test.py _
<frozen importlib._bootstrap_external>:940: in exec_module
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
tensorflow_model_analysis/addons/__init__.py:16: in <module>
    from tensorflow_model_analysis.addons import fairness
tensorflow_model_analysis/addons/fairness/__init__.py:16: in <module>
    from tensorflow_model_analysis.addons.fairness import metrics
tensorflow_model_analysis/addons/fairness/metrics/__init__.py:17: in <module>
    from tensorflow_model_analysis.addons.fairness.metrics import counterfactual_fairness
tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/__init__.py:17: in <module>
    from tensorflow_model_analysis.addons.fairness.metrics.counterfactual_fairness.flip_count import FlipCount
tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count.py:20: in <module>
    from tensorflow_model_analysis.metrics import binary_confusion_matrices
tensorflow_model_analysis/metrics/__init__.py:16: in <module>
    from tensorflow_model_analysis.metrics import bleu
tensorflow_model_analysis/metrics/bleu.py:27: in <module>
    from tensorflow_model_analysis.metrics import metric_types
tensorflow_model_analysis/metrics/metric_types.py:23: in <module>
    from tensorflow_model_analysis.api import types
tensorflow_model_analysis/api/types.py:25: in <module>
    from tensorflow_model_analysis.proto import metrics_for_slice_pb2
tensorflow_model_analysis/proto/metrics_for_slice_pb2.py:36: in <module>
    _descriptor.EnumValueDescriptor(
.venv311/lib/python3.11/site-packages/google/protobuf/descriptor.py:789: in __new__
    _message.Message._CheckCalledFromGeneratedFile()
E   TypeError: Descriptors cannot be created directly.
E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E   If you cannot immediately regenerate your protos, some other possible workarounds are:
E    1. Downgrade the protobuf package to 3.20.x or lower.
E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E   
E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
------------------------------- Captured stderr --------------------------------
2024-10-01 20:29:18.571644: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-10-01 20:29:18.573151: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-10-01 20:29:18.594850: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-10-01 20:29:18.594872: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-10-01 20:29:18.595512: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-10-01 20:29:18.599092: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-10-01 20:29:18.599211: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-01 20:29:19.066551: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
=============================== warnings summary ===============================
.venv311/lib/python3.11/site-packages/jupyter_client/connect.py:22
  /home/william/google-tensorflow-ci-cd/model-analysis/.venv311/lib/python3.11/site-packages/jupyter_client/connect.py:22: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
  given by the platformdirs library.  To remove this warning and
  see the appropriate new directories, set the environment variable
  `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
  The use of platformdirs will be the default in `jupyter_core` v6
    from jupyter_core.paths import jupyter_data_dir, jupyter_runtime_dir, secure_write

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_count_test.py
ERROR tensorflow_model_analysis/addons/fairness/metrics/counterfactual_fairness/flip_rate_test.py
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 2 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
1 warning, 2 errors in 2.08s

smokestacklightnin commented 2 days ago

It seems that there is some issue with pickling and using pytest. The pickling errors do not show up outside of pytest. I'm not sure what it is, and searching online isn't producing anything useful.

peytondmurray commented 2 days ago

Wait, you get different errors depending on the python version? Just for clarification:

With 3.11.10, you get

...
E   TypeError: Descriptors cannot be created directly.
E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E   If you cannot immediately regenerate your protos, some other possible workarounds are:
E    1. Downgrade the protobuf package to 3.20.x or lower.
E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E   
E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
...

With 3.11.12, you get multiple PicklingError?

With respect to the first issue, it looks like protocol buffers changed at some point, and an env variable was added which has an impact on how message sharing happens. @smokestacklightnin did you try setting this env variable?

Applications that rely on sharing messages between Python and C++ break in the new version. Most developers won’t be affected by this, but users of [Nucleus](https://github.com/google/nucleus) and possibly other libraries may be. As a workaround, you can [set an environment variable](https://protobuf.dev/reference/python/python-generated#sharing-messages) that forces the library to preserve compatibility.

peytondmurray commented 1 day ago

I'm able to reproduce the pickling issues:

_________________________ FlipCountTest.testFlipCount __________________________

o = CallableWrapperDoFn(<function Map.<locals>.<lambda> at 0x761de8745630>)
enable_trace = True, use_zlib = False

    def dumps(o, enable_trace=True, use_zlib=False) -> bytes:
      """For internal use only; no backwards-compatibility guarantees."""
      with _pickle_lock:
        try:
>         s = dill.dumps(o, byref=settings['dill_byref'])

It looks like somewhere we're trying to pickle a generator.

tensorflow / model-analysis

Use `pytest` and make GitHub Workflow for tests #183