The test accuracy is not matched with your image.

[ghost]

Hi again.

I run your code and it goes well. But my test acc is like the following :

Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified Train a Model. /usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:82: UserWarning: Detected call of 'lr_scheduler.step()' before 'optimizer.step()'. In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before 'lr_scheduler.step()'. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning) Finished.
Trial 1/3 || Cycle 1/10 || Label set size 1000: Test acc 47.1 Train a Model. Finished.
Trial 1/3 || Cycle 2/10 || Label set size 2000: Test acc 60.14 Train a Model. Finished.
Trial 1/3 || Cycle 3/10 || Label set size 3000: Test acc 69.98 Train a Model. Finished.
Trial 1/3 || Cycle 4/10 || Label set size 4000: Test acc 79.45 Train a Model. Finished.
Trial 1/3 || Cycle 5/10 || Label set size 5000: Test acc 81.04 Train a Model. 74%|████████████████████████████████ | 35/47 [00:04<00:01, 8.66it/s]

I know this is in the middle of running, but the accuracy doesn't seem it is matched with your image. I run your code third times but all the results seem unmatched. How should I interpret this result?

Thanks advance.

[Mephisto405]

Hello Heonjin,

Reproducing the results of this paper, I found that the initial training subset greatly influences test accuracies in the first few cycles. That is, if your initial training set (which contains 1000 data points) is unfortunately biased, the test accuracy before the fifth cycle may be lower than the average performance. This situation can also be found in the authors' results (i.e., high variance in the first few cycles). This might be one pitfall of this paper or related literature.

Thus, in my experiments, this code produced similar accuracy patterns like yours sometimes. See the third trial in the following table:

learn loss | Trial 1 | Trial 2 | Trial 3 | Trial 4 | Trial 5 1k | 0.555 | 0.5078 | 0.4655 | 0.5207 | 0.5084 2k | 0.6839 | 0.6673 | 0.6103 | 0.6585 | 0.6538 3k | 0.765 | 0.763 | 0.6986 | 0.7541 | 0.7656 4k | 0.8261 | 0.8159 | 0.7961 | 0.8154 | 0.8212 5k | 0.8384 | 0.8366 | 0.8233 | 0.8344 | 0.836 6k | 0.8744 | 0.868 | 0.8584 | 0.8666 | 0.8686 7k | 0.8863 | 0.8816 | 0.8735 | 0.8825 | 0.8809 8k | 0.8938 | 0.8901 | 0.8871 | 0.8939 | 0.8896 9k | 0.9036 | 0.8979 | 0.8928 | 0.8999 | 0.895 10k | 0.9068 | 0.9048 | 0.9021 | 0.9067 | 0.9033

Unfortunately, I haven't found any other initial sampling scheme in the paper other than random sampling.

Anyway, good to hear that my code works fine.

[ghost]

If it's possible, please run the code with a fixed Pytorch/Numpy seed, and let me know what seed you take and the result accuracies. I couldn't reach the accuracy above(like Trial1,2,4, and 5) even though I run the code more than 10 times.(I think once/twice out of 20 trials)

Thanks in advance.

[Mephisto405]

Okay. Now I'm running my code with seed "Inyoung Cho", which is just a random string, as below.

And till now, I think it works fine and follows the results of authors'.

If the training finishes, I will update my comment.

[ghost]

Thanks. But you need to fix the Pytorch seed as well.

[Mephisto405]

Thanks for notifying me. I will follow this link: https://pytorch.org/docs/stable/notes/randomness.html after this training, and then I will update my comment with few results.

Updates (2020/5/19/9:05 PM) Only using random seed "Inyoung Cho":

Fix the Pytorch seed as well: Updates (2020/5/20/1:25 AM)

As you can see here, even if the reproduced results cannot beat the core-set unlike the claim in the paper, I can achieve comparable results.

[Mephisto405]

I update the test results, and FYI, I just show my 'conda list' results:

Name Version Build Channel

_ipyw_jlab_nb_ext_conf 0.1.0 py37_0
alabaster 0.7.12 py37_0
anaconda 2019.10 py37_0
anaconda-client 1.7.2 py37_0
anaconda-navigator 1.9.7 py37_0
anaconda-project 0.8.3 py_0
asn1crypto 1.0.1 py37_0
astroid 2.3.1 py37_0
astropy 3.2.1 py37he774522_0
atomicwrites 1.3.0 py37_1
attrs 19.2.0 py_0
babel 2.7.0 py_0
backcall 0.1.0 py37_0
backports 1.0 py_2
backports.functools_lru_cache 1.6.1 py_0
backports.os 0.1.1 py37_0
backports.shutil_get_terminal_size 1.0.0 py37_2
backports.tempfile 1.0 py_1
backports.weakref 1.0.post1 py_1
beautifulsoup4 4.8.0 py37_0
bitarray 1.0.1 py37he774522_0
bkcharts 0.2 py37_0
blas 1.0 mkl
bleach 3.1.0 py37_0
blosc 1.16.3 h7bd577a_0 bokeh 1.3.4 py37_0 boto 2.49.0 py37_0 bottleneck 1.2.1 py37h452e1ab_1 bzip2 1.0.8 he774522_0 ca-certificates 2019.8.28 0 certifi 2019.9.11 py37_0 cffi 1.12.3 py37h7a1dbc1_0 chardet 3.0.4 py37_1003 click 7.0 py37_0 cloudpickle 1.2.2 py_0 clyent 1.2.2 py37_1 cmake-converter 2.0.1 pypi_0 pypi colorama 0.4.1 py37_0 comtypes 1.1.7 py37_0 conda 4.8.1 py37_0 conda-build 3.18.9 py37_3 conda-env 2.6.0 1 conda-package-handling 1.6.0 py37h62dcd97_0 conda-verify 3.4.2 py_1 console_shortcut 0.1.1 3 contextlib2 0.6.0 py_0 cryptography 2.7 py37h7a1dbc1_0 cudatoolkit 10.1.243 h74a9793_0 cudnn 7.6.5 cuda10.1_0 curl 7.65.3 h2a8f88b_0 cycler 0.10.0 py37_0 cython 0.29.13 py37ha925a31_0 cytoolz 0.10.0 py37he774522_0 dask 2.5.2 py_0 dask-core 2.5.2 py_0 decorator 4.4.0 py37_1 defusedxml 0.6.0 py_0 distributed 2.5.2 py_0 docutils 0.15.2 py37_0 entrypoints 0.3 py37_0 et_xmlfile 1.0.1 py37_0 fastcache 1.1.0 py37he774522_0 filelock 3.0.12 py_0 flask 1.1.1 py_0 freetype 2.9.1 ha9979f8_1 fsspec 0.5.2 py_0 future 0.18.2 py37_0 get_terminal_size 1.0.0 h38e98db_0 gevent 1.4.0 py37he774522_0 glob2 0.7 py_0 greenlet 0.4.15 py37hfa6e2cd_0 h5py 2.9.0 py37h5e291fa_0 halide-pytorch 0.0.0 pypi_0 pypi hdf5 1.10.4 h7ebc959_0 heapdict 1.0.1 py_0 html5lib 1.0.1 py37_0 icc_rt 2019.0.0 h0cc432a_1 icu 58.2 ha66f8fd_1 idna 2.8 py37_0 imageio 2.6.0 py37_0 imagesize 1.1.0 py37_0 importlib_metadata 0.23 py37_0 intel-openmp 2019.4 245 ipykernel 5.1.2 py37h39e3cac_0 ipython 7.8.0 py37h39e3cac_0 ipython_genutils 0.2.0 py37_0 ipywidgets 7.5.1 py_0 isort 4.3.21 py37_0 itsdangerous 1.1.0 py37_0 jdcal 1.4.1 py_0 jedi 0.15.1 py37_0 jinja2 2.10.3 py_0 joblib 0.13.2 py37_0 jpeg 9b hb83a4c4_2 json5 0.8.5 py_0 jsonpatch 1.24 py_0 conda-forge jsonpointer 2.0 py_0 conda-forge jsonschema 3.0.2 py37_0 jupyter 1.0.0 py37_7 jupyter_client 5.3.3 py37_1 jupyter_console 6.0.0 py37_0 jupyter_core 4.5.0 py_0 jupyterlab 1.1.4 pyhf63ae98_0 jupyterlab_server 1.0.6 py_0 keyring 18.0.0 py37_0 kiwisolver 1.1.0 py37ha925a31_0 krb5 1.16.1 hc04afaa_7 lazy-object-proxy 1.4.2 py37he774522_0 libarchive 3.3.3 h0643e63_5 libcurl 7.65.3 h2a8f88b_0 libiconv 1.15 h1df5818_7 liblief 0.9.0 ha925a31_2 libpng 1.6.37 h2a8f88b_0 libsodium 1.0.16 h9d3ae62_0 libssh2 1.8.2 h7a1dbc1_0 libtiff 4.0.10 hb898794_2 libxml2 2.9.9 h464c3ec_0 libxslt 1.1.33 h579f668_0 llvmlite 0.29.0 py37ha925a31_0 locket 0.2.0 py37_1 lxml 4.4.1 py37h1350720_0 lz4-c 1.8.1.2 h2fa13f4_0 lzo 2.10 h6df0209_2 m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs 5.3.0 7 m2w64-gcc-libs-core 5.3.0 7 m2w64-gmp 6.1.0 2 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 markupsafe 1.1.1 py37he774522_0 matplotlib 3.1.1 py37hc8f65d3_0 mccabe 0.6.1 py37_1 menuinst 1.4.16 py37he774522_0 mistune 0.8.4 py37he774522_0 mkl 2019.4 245 mkl-service 2.3.0 py37hb782905_0 mkl_fft 1.0.14 py37h14836fe_0 mkl_random 1.1.0 py37h675688f_0 mock 3.0.5 py37_0 more-itertools 7.2.0 py37_0 mpmath 1.1.0 py37_0 msgpack-python 0.6.1 py37h74a9793_1 msys2-conda-epoch 20160418 1 multipledispatch 0.6.0 py37_0 navigator-updater 0.2.1 py37_0 nbconvert 5.6.0 py37_1
nbformat 4.4.0 py37_0 networkx 2.3 py_0 ninja 1.9.0 py37h74a9793_0 nltk 3.4.5 py37_0 nose 1.3.7 py37_2 notebook 6.0.1 py37_0 numba 0.45.1 py37hf9181ef_0 numexpr 2.7.0 py37hdce8814_0 numpy 1.16.5 py37h19fb1c0_0 numpy-base 1.16.5 py37hc3f5095_0 numpydoc 0.9.1 py_0 olefile 0.46 py37_0 opencv-contrib-python 3.4.2.16 pypi_0 pypi opencv-python 3.4.2.16 pypi_0 pypi openpyxl 3.0.0 py_0 openssl 1.1.1d he774522_2 packaging 19.2 py_0 pandas 0.25.1 py37ha925a31_0 pandoc 2.2.3.2 0 pandocfilters 1.4.2 py37_1 parso 0.5.1 py_0 partd 1.0.0 py_0 path.py 12.0.1 py_0 pathlib2 2.3.5 py37_0 patsy 0.5.1 py37_0 pep8 1.7.1 py37_0 pickleshare 0.7.5 py37_0 pillow 6.2.0 py37hdc69c19_0 pip 19.2.3 py37_0 pkginfo 1.5.0.1 py37_0 pluggy 0.13.0 py37_0 ply 3.11 py37_0 powershell_shortcut 0.0.1 2 prometheus_client 0.7.1 py_0 prompt_toolkit 2.0.10 py_0 psutil 5.6.3 py37he774522_0 py 1.8.0 py37_0 py-lief 0.9.0 py37ha925a31_2 pycodestyle 2.5.0 py37_0 pycosat 0.6.3 py37hfa6e2cd_0 pycparser 2.19 py37_0 pycrypto 2.6.1 py37hfa6e2cd_9 pycurl 7.43.0.3 py37h7a1dbc1_0 pyflakes 2.1.1 py37_0 pygments 2.4.2 py_0 pylint 2.4.2 py37_0 pyodbc 4.0.27 py37ha925a31_0 pyopenssl 19.0.0 py37_0 pyparsing 2.4.2 py_0 pyqt 5.9.2 py37h6538335_2 pyreadline 2.1 py37_1 pyrsistent 0.15.4 py37he774522_0 pysocks 1.7.1 py37_0 pytables 3.5.2 py37h1da0976_1 pytest 5.2.1 py37_0 pytest-arraydiff 0.3 py37h39e3cac_0 pytest-astropy 0.5.0 py37_0 pytest-doctestplus 0.4.0 py_0 pytest-openfiles 0.4.0 py_0 pytest-remotedata 0.3.2 py37_0 python 3.7.4 h5263a28_0 python-dateutil 2.8.0 py37_0 python-libarchive-c 2.8 py37_13 pytorch 1.3.1 py3.7_cuda101_cudnn7_0 pytorch pytz 2019.3 py_0 pywavelets 1.0.3 py37h8c2d366_1 pywin32 223 py37hfa6e2cd_1 pywinpty 0.5.5 py37_1000 pyyaml 5.1.2 py37he774522_0 pyzmq 18.1.0 py37ha925a31_0 qt 5.9.7 vc14h73c81de_0 [vc14] anaconda qtawesome 0.6.0 py_0 qtconsole 4.5.5 py_0 qtpy 1.9.0 py_0 requests 2.22.0 py37_0 rope 0.14.0 py_0 ruamel_yaml 0.15.46 py37hfa6e2cd_0 scikit-image 0.15.0 py37ha925a31_0 scikit-learn 0.21.3 py37h6288b17_0 scipy 1.3.1 py37h29ff71c_0 seaborn 0.9.0 py37_0 send2trash 1.5.0 py37_0 setuptools 41.4.0 py37_0 simplegeneric 0.8.1 py37_2 singledispatch 3.4.0.3 py37_0 sip 4.19.8 py37h6538335_0 six 1.12.0 py37_0 snappy 1.1.7 h777316e_3 snowballstemmer 2.0.0 py_0 sortedcollections 1.1.2 py37_0 sortedcontainers 2.1.0 py37_0 soupsieve 1.9.3 py37_0 sphinx 2.2.0 py_0 sphinxcontrib 1.0 py37_1 sphinxcontrib-applehelp 1.0.1 py_0 sphinxcontrib-devhelp 1.0.1 py_0 sphinxcontrib-htmlhelp 1.0.2 py_0 sphinxcontrib-jsmath 1.0.1 py_0 sphinxcontrib-qthelp 1.0.2 py_0 sphinxcontrib-serializinghtml 1.1.3 py_0 sphinxcontrib-websupport 1.1.2 py_0 spyder 3.3.6 py37_0 spyder-kernels 0.5.2 py37_0 sqlalchemy 1.3.9 py37he774522_0 sqlite 3.30.0 he774522_0 statsmodels 0.10.1 py37h8c2d366_0 sympy 1.4 py37_0 tbb 2019.4 h74a9793_0 tblib 1.4.0 py_0 terminado 0.8.2 py37_0 testpath 0.4.2 py37_0 tk 8.6.8 hfa6e2cd_0 toolz 0.10.0 py_0 torchfile 0.1.0 py_0 conda-forge torchvision 0.4.2 py37_cu101 pytorch tornado 6.0.3 py37he774522_0 tqdm 4.36.1 py_0 traitlets 4.3.3 py37_0 unicodecsv 0.14.1 py37_0 urllib3 1.24.2 py37_0 vc 14.1 h0510ff6_4 visdom 0.1.8.9 0 conda-forge vs2015_runtime 14.16.27012 hf0eaf9b_0 wcwidth 0.1.7 py37_0 webencodings 0.5.1 py37_1 websocket-client 0.56.0 py37_0 werkzeug 0.16.0 py_0 wheel 0.33.6 py37_0 widgetsnbextension 3.5.1 py37_0 win_inet_pton 1.1.0 py37_0 win_unicode_console 0.5 py37_0 wincertstore 0.2 py37_0 winpty 0.4.3 4 wrapt 1.11.2 py37he774522_0 xlrd 1.2.0 py37_0 xlsxwriter 1.2.1 py_0 xlwings 0.15.10 py37_0 xlwt 1.3.0 py37_0 xz 5.2.4 h2fa13f4_4 yaml 0.1.7 hc54c509_2 zeromq 4.3.1 h33f27b4_3 zict 1.0.0 py_0 zipp 0.6.0 py_0 zlib 1.2.11 h62dcd97_3 zstd 1.3.7 h508b16e_0

[ghost]

Thanks for notifying me. I will follow this link: https://pytorch.org/docs/stable/notes/randomness.html after this training, and then I will update my comment with few results.

Updates (2020/5/19/9:05 PM) Only using random seed "Inyoung Cho":

Fix the Pytorch seed as well: Updates (2020/5/20/1:25 AM)

As you can see here, even if the reproduced results cannot beat the core-set unlike the claim in the paper, I can achieve comparable results.

Thanks for sharing your results. By the way, what Pytorch seed did u you use? I'll run your code tmrw in order to check the results.

[Mephisto405]

I just used 0 as shown in the link.

[ghost]

Thanks.

I will also see my outcomes. I'd like to get the 81+% acc at Label set size 4000. At Label set size 10000 is not my focus. I'll upload mine after it finishes.

[ghost]

(will be updated)