Error running uniform partition for text classification

[bangawayoo]

Hi. I am encountering a EOFError when trying to run uniform partition for text classification.

run_text_classification.sh FedOPT "uniform" 5e-5 0.1 51 4

27440 2022-01-09,13:28:37.575 - {base_data_manager.py (306)} - _load_data_loader_from_cache(): Loading features from cached file cache_dir/distilbert_distilbert-base-uncased_cached_256_ClassificationModel_20news_uniform_75 Traceback (most recent call last): File "/home/ky/miniconda3/envs/fednlp/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/ky/miniconda3/envs/fednlp/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ky/Research/NLP/FL/FedNLP/experiments/distributed/transformer_exps/run_tc_exps/fedavg_main_tc.py", line 140, in train_data_local_dict, test_data_local_dict, num_clients = dm.load_federated_data(process_id=process_id) File "/home/ky/Research/NLP/FL/FedNLP/data_manager/base_data_manager.py", line 142, in load_federated_data return self._load_federated_data_local() File "/home/ky/Research/NLP/FL/FedNLP/data_manager/base_data_manager.py", line 240, in _load_federated_data_local state, res = self._load_data_loader_from_cache(client_idx) File "/home/ky/Research/NLP/FL/FedNLP/data_manager/base_data_manager.py", line 309, in _load_data_loader_from_cache train_examples, train_features, train_dataset, test_examples, test_features, test_dataset = pickle.load(handle) EOFError: Ran out of input

Non-iid partition method works fine. Any suggestion on how to fix this?

Thanks.

[bangawayoo]

Seems relevant to #16. Following the below solution fixes this.

please add --reprocess_input_data at the end of the bash script