search

RTIInternational / gobbli

Deep learning with text doesn't have to be scary.
Apache License 2.0
275 stars 23 forks source link

Failed to unpickle when running the ClassificationExperiment #10

Closed theccalderon closed 5 years ago

theccalderon commented 5 years ago

System Information

Description

Trying to execute the example on https://medium.com/rti-cds/introducing-gobbli-b625c0a0adfe we face the following error:

2019-09-23 17:46:46,883 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

Code for Minimal Reproducible Example

from gobbli.experiment.classification import ClassificationExperiment
from gobbli.model.bert import BERT
X = [
    "This is positive.",
    "This is negative.",
    ...
]

y = [
    "Good",
    "Bad",
    ...
]

exp = ClassificationExperiment(
    model_cls=BERT,
    dataset=(X, y)
)

results = exp.run()

Output

2019-09-23 18:37:26,757 WARNING resource_spec.py:163 -- Warning: Capping object memory store to 20.0GB. To increase this further, specify object_store_memory when calling ray.init() or ray start. 2019-09-23 18:37:26,758 INFO resource_spec.py:205 -- Starting Ray with 60.16 GiB memory available for workers and up to 18.63 GiB for objects. You can adjust these settings with ray.remote(memory=, object_store_memory=). 2019-09-23 18:37:28,039 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,041 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,041 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,042 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,042 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,043 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,044 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,044 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,045 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,045 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,046 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,046 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,047 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,047 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,048 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,048 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,049 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,049 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,050 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,050 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,051 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,062 ERROR worker.py:1791 -- Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 942, in _process_task function_name, args) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 855, in _get_arguments_for_execution values = self.get_object(object_ids) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 553, in get_object final_results = self.retrieve_and_deserialize(plain_object_ids, 0) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 443, in retrieve_and_deserialize self.current_job_id) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 255, in get_serialization_context _initialize_serialization(job_id) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1234, in _initialize_serialization serialization_context = pyarrow.default_serialization_context() File "/opt/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context register_default_serialization_handlers(context) File "/opt/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers _register_custom_pandas_handlers(serialization_context) File "/opt/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers import pandas as pd File "/opt/anaconda3/lib/python3.7/site-packages/pandas/init.py", line 23, in from pandas.compat.numpy import * File "/opt/anaconda3/lib/python3.7/site-packages/pandas/compat/init.py", line 446, in import typing File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in ray.worker.global_worker.main_loop() File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1135, in main_loop self._wait_for_and_process_task(task) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1086, in _wait_for_and_process_task self._process_task(task, execution_info) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 946, in _process_task ray.utils.format_error_message(traceback.format_exc())) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1026, in _handle_process_task_failure self._store_outputs_in_object_store(return_object_ids, failure_objects) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 896, in _store_outputs_in_object_store self.put_object(object_ids[i], outputs[i]) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 397, in put_object self._try_store_and_register(object_id, value) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 417, in _try_store_and_register self.store_and_register(object_id, value) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 326, in store_and_register self.current_job_id)) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 255, in get_serialization_context _initialize_serialization(job_id) File "/opt/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1234, in _initialize_serialization serialization_context = pyarrow.default_serialization_context() File "/opt/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context register_default_serialization_handlers(context) File "/opt/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers _register_custom_pandas_handlers(serialization_context) File "/opt/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers import pandas as pd File "/opt/anaconda3/lib/python3.7/site-packages/pandas/init.py", line 23, in from pandas.compat.numpy import * File "/opt/anaconda3/lib/python3.7/site-packages/pandas/compat/init.py", line 446, in import typing File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

This error is unexpected and should not have happened. Somehow a worker crashed in an unanticipated way causing the main_loop to throw an exception, which is being caught in "python/ray/workers/default_worker.py".

2019-09-23 18:37:28,064 ERROR worker.py:1791 -- A worker died or was killed while executing task 7d58f415c89effffffff01000000. 2019-09-23 18:37:28,149 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,150 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-09-23 18:37:28,151 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 2, in from typing import Any, Dict, List File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1356, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/anaconda3/lib/python3.7/site-packages/typing.py", line 1004, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

RayWorkerError Traceback (most recent call last)

in 10 ) 11 ---> 12 results = exp.run() /opt/anaconda3/lib/python3.7/site-packages/gobbli/experiment/classification.py in run(self, dataset_split, seed, train_batch_size, valid_batch_size, test_batch_size, num_train_epochs) 648 self.distributed, 649 ) --> 650 for params in grid 651 ] 652 ) /opt/anaconda3/lib/python3.7/site-packages/ray/worker.py in get(object_ids) 2330 if isinstance(value, RayError): 2331 last_task_error_raise_time = time.time() -> 2332 raise value 2333 2334 # Run post processors. RayWorkerError: The worker died unexpectedly while executing this task.
jasonnance commented 5 years ago

I ran into this some when developing -- it happens when you have the 3rd-party typing library installed, usually as a dependency of some other package. The fix should be to pip uninstall typing -- if you're running Python 3.7+, you don't need it anyway, since it's in the standard library. Please let me know if that doesn't fix it for you.

Relevant issues: https://github.com/ray-project/ray/issues/5025 and https://github.com/python/typing/issues/573 for more details.

theccalderon commented 5 years ago

Hi @jasonnance thanks for getting back to me. I had already tried that and here is the stack trace after uninstalling typing:

2019-09-24 14:04:33,840 WARNING resource_spec.py:163 -- Warning: Capping object memory store to 20.0GB. To increase this further, specify object_store_memory when calling ray.init() or ray start. 2019-09-24 14:04:33,841 INFO resource_spec.py:205 -- Starting Ray with 60.3 GiB memory available for workers and up to 18.63 GiB for objects. You can adjust these settings with ray.remote(memory=, object_store_memory=). 2019-09-24 14:04:40,311 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,313 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,314 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,315 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,315 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,316 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,316 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,317 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,318 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,318 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,319 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,320 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,320 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,321 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,321 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,322 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,322 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,323 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,324 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,324 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,325 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,325 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,326 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

2019-09-24 14:04:40,326 ERROR worker.py:1791 -- Failed to unpickle the remote function 'gobbli.experiment.classification.train' with function ID b4b3e55ece2bc00875449aa91491c38f3316065e. Traceback: Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/ray/function_manager.py", line 408, in fetch_and_register_remote_function function = pickle.loads(serialized_function) File "/opt/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 965, in subimport import(name) File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/init.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/init.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/init.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/init.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/init.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

RayTaskError Traceback (most recent call last)

in 10 ) 11 ---> 12 results = exp.run() /opt/anaconda3/lib/python3.7/site-packages/gobbli/experiment/classification.py in run(self, dataset_split, seed, train_batch_size, valid_batch_size, test_batch_size, num_train_epochs) 648 self.distributed, 649 ) --> 650 for params in grid 651 ] 652 ) /opt/anaconda3/lib/python3.7/site-packages/ray/worker.py in get(object_ids) 2330 if isinstance(value, RayError): 2331 last_task_error_raise_time = time.time() -> 2332 raise value 2333 2334 # Run post processors. RayTaskError: ray_worker:gobbli.experiment.classification.train() (pid=1606, host=my-fastai-instance) File "pyarrow/serialization.pxi", line 461, in pyarrow.lib.deserialize File "pyarrow/serialization.pxi", line 424, in pyarrow.lib.deserialize_from File "pyarrow/serialization.pxi", line 275, in pyarrow.lib.SerializedPyObject.deserialize File "pyarrow/serialization.pxi", line 174, in pyarrow.lib.SerializationContext._deserialize_callback File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/__init__.py", line 1, in import gobbli.augment as augment File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/__init__.py", line 1, in from gobbli.augment.bert import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/__init__.py", line 1, in from .model import BERTMaskedLM File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/augment/bert/model.py", line 6, in from gobbli.model.base import BaseModel File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/__init__.py", line 1, in from gobbli.model.bert import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/__init__.py", line 1, in from .model import BERT File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/model/bert/model.py", line 10, in import gobbli.io File "/opt/anaconda3/lib/python3.7/site-packages/gobbli/io.py", line 78, in class TrainInput(TaskIO): File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 801, in for name, type in cls_annotations.items()] File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/opt/anaconda3/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar'

​

jasonnance commented 5 years ago

Ah, this looks like an analogous issue with the third-party dataclasses package. Looks like it was expecting the third-party typing module rather than the first-party one. Try pip uninstall dataclasses.

EDIT: If you somehow have the code from the third-party dataclasses package without having installed it via pip, I'd guess you might have some kind of pre-release of Python 3.7? Are you able to upgrade to a more recent version?

theccalderon commented 5 years ago

pip uninstall dataclasses fixed the problem! Thank you!

jasonnance commented 5 years ago

Great, thanks for the report! I've added a check to identify the conflict with these dependencies on import and warn the user. Will be added in a new release (0.0.6).

daniel-jebarson commented 2 years ago

Once I installed typing in replit. i Can't run the program and i can't use any commands like install, uninstall with pip in console. image This is my pyproject.toml file is [tool.poetry] name = "repl_python3_kingdom-fight-check" version = "0.1.0" description = "" authors = ["Your Name you@example.com"]

[tool.poetry.dependencies] python = "^3.8" replit = "^3.1.0" Flask = "^2.0.1" srv = "^1.1.0" youtube_dl = "^2021.6.6" DiscordUtils = "^1.3.4" "akinator.py" = "^5.0.0" pytz = "^2021.3" discord = "^1.7.3" pymongo = "^3.12.1" discord-components = "^2.1.2" pyNacl="^1.4.0" asyncpg = "^0.25.0" dnspython="^2.1.0" qrcode = "^7.3.1" bs4 = "^0.0.1" wikipedia = "^1.4.0" googletrans = "^3.1.0a0"

[tool.poetry.dev-dependencies]

[build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api"

This is the error pop up while running the program Traceback (most recent call last): File "main.py", line 1, in from discord.ext.commands import Bot File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/init.py", line 25, in from .client import Client File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/client.py", line 27, in import asyncio File "/usr/lib/python3.8/asyncio/init.py", line 8, in from .base_events import Traceback (most recent call last): File "main.py", line 1, in from discord.ext.commands import Bot File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/init.py", line 25, in from .client import Client File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/client.py", line 27, in import asyncio File "/usr/lib/python3.8/asyncio/init.py", line 8, in from .base_events import Traceback (most recent call last): File "main.py", line 1, in from discord.ext.commands import Bot File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/init.py", line 25, in from .client import Client File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/client.py", line 27, in import asyncio File "/usr/lib/python3.8/asyncio/init.py", line 8, in from .base_events import Traceback (most recent call last): File "main.py", line 1, in from discord.ext.commands import Bot File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/init.py", line 25, in from .client import Client File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/client.py", line 27, in import asyncio File "/usr/lib/python3.8/asyncio/init.py", line 8, in from .base_events import Traceback (most recent call last): File "main.py", line 1, in from discord.ext.commands import Bot File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/init.py", line 25, in from .client import Client File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/client.py", line 27, in import asyncio File "/usr/lib/python3.8/asyncio/init.py", line 8, in from .base_events import Traceback (most recent call last): File "main.py", line 1, in from discord.ext.commands import Bot File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/init.py", line 25, in from .client import Client File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/client.py", line 27, in import asyncio File "/usr/lib/python3.8/asyncio/init.py", line 8, in from .base_events import File "/usr/lib/python3.8/asyncio/base_events.py", line 45, in from . import staggered File "/usr/lib/python3.8/asyncio/staggered.py", line 6, in import typing File "/opt/virtualenvs/python3/lib/python3.8/site-packages/typing.py", line 1359, in class Callable(extra=collections_abc.Callable, metaclass=CallableMeta): File "/opt/virtualenvs/python3/lib/python3.8/site-packages/typing.py", line 1007, in new self._abc_registry = extra._abc_registry AttributeError: type object 'Callable' has no attribute '_abc_registry'

Can someone help me how to fix this problem?

pmbaumgartner commented 2 years ago

@Dani-Hacker I don't see the traceback indicating any issue with gobbli. In general, I would not expect this to work with something like replit, since it requires you to have docker installed and that's not the case with an environment like replit.