search

CharlieDinh / pFedMe

Personalized Federated Learning with Moreau Envelopes (pFedMe) using Pytorch (NeurIPS 2020)
289 stars 88 forks source link

Unable to generate non-iid MNIST Data #10

Closed Sharad24 closed 3 years ago

Sharad24 commented 3 years ago

Describe the bug While generation of the non-iid MNIST data, generate_niid_20users.py runs into an error

To Reproduce Steps to reproduce the behavior:

  1. Go to 'data/Mnist'
  2. Run python generate_niid_20users.py

Trace

100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 38.74it/s]

Numb samples of each label:
 [6903, 7877, 6990, 7141, 6824, 6313, 6876, 7293, 6825, 6958]
idx 0        False
1        False
2        False
3        False
4         True
         ...  
69995    False
69996    False
69997    False
69998    False
69999    False
Name: class, Length: 70000, dtype: bool
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 135300.13it/s]
--------------
[0 1 2 3 4 5 6 7 8 9] [4 4 4 4 4 4 4 4 4 4]
6903
[2441, 1127, 1575, 1760]
7877
[2671, 1946, 1367, 1893]
6990
[2358, 1070, 841, 2721]
7141
[2630, 2202, 715, 1594]
6824
[1721, 1934, 1101, 2068]
6313
[2169, 1080, 1102, 1962]
6876
[2043, 1364, 1255, 2214]
7293
[2211, 2518, 598, 1966]
6825
[1506, 2480, 574, 2265]
6958
[1710, 1878, 1208, 2162]
--------------
[[2441, 1127, 1575, 1760], [2671, 1946, 1367, 1893], [2358, 1070, 841, 2721], [2630, 2202, 715, 1594], [1721, 1934, 1101, 2068], [2169, 1080, 1102, 1962], [2043, 1364, 1255, 2214], [2211, 2518, 598, 1966], [1506, 2480, 574, 2265], [1710, 1878, 1208, 2162]]
[2441, 1127, 1575, 1760]
[2671, 1946, 1367, 1893]
[2358, 1070, 841, 2721]
[2630, 2202, 715, 1594]
[1721, 1934, 1101, 2068]
[2169, 1080, 1102, 1962]
[2043, 1364, 1255, 2214]
[2211, 2518, 598, 1966]
[1506, 2480, 574, 2265]
[1710, 1878, 1208, 2162]
[2441, 1127, 1575, 1760]
[2671, 1946, 1367, 1893]
[2358, 1070, 841, 2721]
[2630, 2202, 715, 1594]
[1721, 1934, 1101, 2068]
[2169, 1080, 1102, 1962]
[2043, 1364, 1255, 2214]
[2211, 2518, 598, 1966]
[1506, 2480, 574, 2265]
[1710, 1878, 1208, 2162]
[2441, 1127, 1575, 1760]
[2671, 1946, 1367, 1893]
[2358, 1070, 841, 2721]
[2630, 2202, 715, 1594]
[1721, 1934, 1101, 2068]
[2169, 1080, 1102, 1962]
[2043, 1364, 1255, 2214]
[2211, 2518, 598, 1966]
[1506, 2480, 574, 2265]
[1710, 1878, 1208, 2162]
[2441, 1127, 1575, 1760]
[2671, 1946, 1367, 1893]
[2358, 1070, 841, 2721]
[2630, 2202, 715, 1594]
[1721, 1934, 1101, 2068]
[2169, 1080, 1102, 1962]
[2043, 1364, 1255, 2214]
[2211, 2518, 598, 1966]
[1506, 2480, 574, 2265]
[1710, 1878, 1208, 2162]
--------------
[2441, 2671, 2358, 2630, 1721, 2169, 2043, 2211, 1506, 1710, 1127, 1946, 1070, 2202, 1934, 1080, 1364, 2518, 2480, 1878, 1575, 1367, 841, 715, 1101, 1102, 1255, 598, 574, 1208, 1760, 1893, 2721, 1594, 2068, 1962, 2214, 1966, 2265, 2162]
  0%|                                                                                                                                           | 0/20 [00:00<?, ?it/s]value of L 0
value of count 0
  0%|                                                                                                                                           | 0/20 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "generate_niid_20users.py", line 86, in <module>
    X[user] += mnist_data[l][idx[l]:num_samples].tolist()
  File "/Users/sharadchitlangia/miniconda3/envs/FL/lib/python3.6/site-packages/pandas/core/frame.py", line 2881, in __getitem__
    indexer = convert_to_index_sliceable(self, key)
  File "/Users/sharadchitlangia/miniconda3/envs/FL/lib/python3.6/site-packages/pandas/core/indexing.py", line 2132, in convert_to_index_sliceable
    return idx._convert_slice_indexer(key, kind="getitem")
  File "/Users/sharadchitlangia/miniconda3/envs/FL/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3159, in _convert_slice_indexer
    self._validate_indexer("slice", key.start, "getitem")
  File "/Users/sharadchitlangia/miniconda3/envs/FL/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 5000, in _validate_indexer
    self._invalid_indexer(form, key)
  File "/Users/sharadchitlangia/miniconda3/envs/FL/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3271, in _invalid_indexer
    f"cannot do {form} indexing on {type(self).__name__} with these "
TypeError: cannot do slice indexing on Int64Index with these indexers [False] of type bool_
CharlieDinh commented 3 years ago

This may come from different versions of python maybe you are using 3.8. In the old version of python, 'np.bool_' can be scaled to be interpreted as an index. I will update the code later. You can download the dataset on google driver at this moment.

Sharad24 commented 3 years ago

My Python version is 3.6.12 😅

Sharad24 commented 3 years ago

This maybe of further help:

Package              Version
-------------------- -------------------
absl-py              0.11.0
appnope              0.1.2
argon2-cffi          20.1.0
astor                0.8.1
async-generator      1.10
attrs                20.3.0
backcall             0.2.0
bleach               3.2.3
cached-property      1.5.2
certifi              2020.12.5
cffi                 1.14.4
cycler               0.10.0
dataclasses          0.8
decorator            4.4.2
defusedxml           0.6.0
entrypoints          0.3
gast                 0.4.0
grpcio               1.35.0
h5py                 3.1.0
importlib-metadata   3.4.0
ipykernel            5.4.3
ipython              7.16.1
ipython-genutils     0.2.0
ipywidgets           7.6.3
jedi                 0.18.0
Jinja2               2.11.2
joblib               1.0.1
jsonschema           3.2.0
jupyter              1.0.0
jupyter-client       6.1.11
jupyter-console      6.2.0
jupyter-core         4.7.0
jupyterlab-pygments  0.1.2
jupyterlab-widgets   1.0.0
Keras-Applications   1.0.8
Keras-Preprocessing  1.1.2
kiwisolver           1.3.1
Markdown             3.3.3
MarkupSafe           1.1.1
matplotlib           3.3.3
mistune              0.8.4
mock                 4.0.3
nbclient             0.5.1
nbconvert            6.0.7
nbformat             5.1.2
nest-asyncio         1.4.3
notebook             6.2.0
numpy                1.16.4
packaging            20.8
pandas               1.1.5
pandocfilters        1.4.3
parso                0.8.1
pexpect              4.8.0
pickleshare          0.7.5
Pillow               8.1.0
pip                  21.0
prometheus-client    0.9.0
prompt-toolkit       3.0.14
protobuf             3.14.0
ptyprocess           0.7.0
pycparser            2.20
Pygments             2.7.4
pyparsing            2.4.7
pyrsistent           0.17.3
python-dateutil      2.8.1
pytz                 2020.5
pyzmq                21.0.2
qtconsole            5.0.2
QtPy                 1.9.0
scikit-learn         0.24.1
scipy                1.5.4
Send2Trash           1.5.0
setuptools           49.6.0.post20210108
six                  1.15.0
tensorboard          1.13.1
tensorflow           1.13.1
tensorflow-estimator 1.13.0
termcolor            1.1.0
terminado            0.9.2
testpath             0.4.4
threadpoolctl        2.1.0
torch                1.8.1
torchvision          0.9.1
tornado              6.1
tqdm                 4.59.0
traitlets            4.3.3
typing-extensions    3.7.4.3
wcwidth              0.2.5
webencodings         0.5.1
Werkzeug             1.0.1
wheel                0.36.2
widgetsnbextension   3.5.1
zipp                 3.4.0
CharlieDinh commented 3 years ago

I think one solution is you can convert False to 0 and True to 1. I am not sure which environment causes the error.