AttributeError: module 'dill._dill' has no attribute 'PY3'

[demongolem-biz2]

I am trying to learn how to fine-tune. As such, I am running to run preparation for end-to-end question generations as given:

python prepare_data.py \ --task e2e_qg \ --valid_for_qg_only \ --model_type t5 \ --dataset_path data/squad_multitask/ \ --qg_format highlight_qg_format \ --max_source_length 512 \ --max_target_length 32 \ --train_file_name train_data_e2e_qg_t5.pt \ --valid_file_name valid_data_e2e_qg_t5.pt \

I get the following stack trace:

File "/home/gregory.werner/question_generation/prepare_data.py", line 165, in main train_dataset = train_dataset.filter(TASK_TO_FILTER_FN[data_args.task]) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/arrow_dataset.py", line 1053, in filter return self.map( File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/arrow_dataset.py", line 902, in map cache_file_name = self._get_cache_file_path(function, cache_kwargs) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/arrow_dataset.py", line 756, in _get_cache_file_path function_bytes = dumps(function) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/utils/py_utils.py", line 278, in dumps dump(obj, file) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/utils/py_utils.py", line 271, in dump Pickler(file).dump(obj) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 418, in dump StockPickler.dump(self, obj) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 487, in dump self.save(obj) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 412, in save StockPickler.save(self, obj, save_persistent_id) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 1965, in save_function _save_with_postproc(pickler, (_create_function, ( File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 1093, in _save_with_postproc pickler.save_reduce(*reduction, obj=obj) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 692, in save_reduce save(args) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 412, in save StockPickler.save(self, obj, save_persistent_id) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 902, in save_tuple save(element) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 412, in save StockPickler.save(self, obj, save_persistent_id) File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/utils/py_utils.py", line 304, in save_code if dill._dill.PY3: AttributeError: module 'dill._dill' has no attribute 'PY3'

Is this a versioning thing maybe where I have to old or new of a version of something?

[demongolem-biz2]

I don't really get nlp vs datasets. I guess I thought datasets was the same, but newer.

But anyhow, I took code from datasets file utils/py_utils.py in save_code method to hardcode the correct args in the nlp version and it worked. So there seems to be something about the tutorial which needs updating? I had initially tried replacing nlp with datasets in the prepare_data.py code but that did not work for some other reason.

[THANH-SSV]

AttributeError: module 'dill._dill' has no attribute 'PY3' Have you solved this problem? Can you tell me? I really don't know what to deal with

[ZZZZkp]

I don't really get nlp vs datasets. I guess I thought datasets was the same, but newer.

But anyhow, I took code from datasets file utils/py_utils.py in save_code method to hardcode the correct args in the nlp version and it worked. So there seems to be something about the tutorial which needs updating? I had initially tried replacing nlp with datasets in the prepare_data.py code but that did not work for some other reason.

Don't forget the usage of nlp in squad_multitask.py.

[nitheeshk03]

AttributeError: module 'dill._dill' has no attribute 'PY3' Have you solved this problem? Can you tell me? I really don't know what to deal with

I faced the same issue and installing dill version 0.3.5 worked for me pip install dill==0.3.5

[imkhoibui]

AttributeError: module 'dill._dill' has no attribute 'PY3' Have you solved this problem? Can you tell me? I really don't know what to deal with

You should either use a lower version of nlp (specifically 0.2.0) or just use datasets package and adapt everything else altogether.