S16 - Creating Classifier Ensembles through Meta-heuristic Algorithms for Aerial Scene Classification

[fariafabio]

General info

• [x] Conference: International Conference on Pattern Recognition 2020
• [x] Papers title: Creating Classifier Ensembles through Meta-heuristic Algorithms for Aerial Scene Classification
• [x] Source paper: paper 108
• [x] GitHub repo: https://github.com/gugarosa/evolutionary_ensembles
• [x] Submission date: 09/23/2020
• [x] Platform: github
• [x] Language: Python
• [x] Results to be reproduced: Table III and IV
• [x] Reproduction main step instructions: See README of Github repo

Reviewer feedback

• [x] Tested platform: MacBook Pro 2019, Intel Core i9 2.3Ghz, MacOS 10.15.3
• [x] Result reproduced: Only Table III
• [x] Reviewer comment:

The execution of the commands did not pose any particular problem and the output was produced as expected. Results relative to accuracies (Table III) are not exactly the same as the ones presented in the paper. However, as stated by the authors, "as the algorithms are non-deterministic approaches, it's possible to get different classification results within standard deviation ranges". The ranking of the different methods is however consistent with the results presented in the paper. Results relative to execution time could however not really be related to the ones presented in Table IV.

Details Results

• Execution without errors:
  • [x] without changes
  • [ ] with small changes
  • [ ] with a lot of changes
• Results are obtained:
  • [x] always
  • [ ] sometimes
  • [ ] never
• Results correspond to the paper's figures/tables:
  • [ ] exactly
  • [x] with negligible differences
  • [ ] with differences
  • [ ] with a lot of differences

[akrah]

Hi @fariafabio, Thanks for your submission! All seems good except that I don't have access to your paper. Probably it's a private repo? If you don't want add a public link here, please sent us your paper by private email.

[akrah]

Moreover, could you please indicate in "Platform" which OS (and computer if needed) you used to compile/execute your code? (ex: Ubuntu 20.04, MacOS 10.15.6, etc.)

[akrah]

@fariafabio I started to try to reproduce your results. My remarks:

• I use python3 with pip3
• majority_voting.py: all is ok
• ensemble_learning.py: The opytimizer (https://opytimizer.readthedocs.io/en/latest/api/opytimizer.html) has probably evolved because the imports in utils/mh.py and utils/wrapper.py are not good. I need to change for this one:

  from opytimizer.optimizers.swarm import abc, ba, cs, fa, fpa, pso
  from opytimizer.optimizers.science import bh
  from opytimizer.optimizers.evolutionary.gp import GP

  and change line about bho that is now called bh

• To be able to reproduce your results, could you provide the set of instructions you have used? For that I suggest to precise that in the README.md of your git repo, or, better, to create a bash script that call this set of instructions. With this script, we just need to run it to reproduce your results. Maybe one script for Table III and another one for table IV?

[akrah]

@fariafabio Thanks to have take into account the remarks.

I tested the script pipeline.sh and I have four remarks:

1. precise the shell used as first line (like #!/bin/sh or #!/bin/bash) to be compliant with usual good practices of script shell

2. like your README now specify that you need ti use python3, you can change the calls to python by calls to python3

3. the line FOLDS=(1 2 3 4 5 6) is not consistent with the cotent of folders data/XXX/val/ that only have xxx_1.txt to xxx_5.txt but not xxx_6.txt. You need to change this line or add the xxx_6.txt files.

4. The call to ensemble_learning_with_umda.py generate the following error:

   --- Logging error ---
   Traceback (most recent call last):
   File "/usr/lib/python3.7/logging/__init__.py", line 1034, in emit
   msg = self.format(record)
   File "/usr/lib/python3.7/logging/__init__.py", line 880, in format
   return fmt.format(record)
   File "/usr/lib/python3.7/logging/__init__.py", line 619, in format
   record.message = record.getMessage()
   File "/usr/lib/python3.7/logging/__init__.py", line 380, in getMessage
   msg = msg % self.args
   TypeError: not all arguments converted during string formatting
   Call stack:
   File "ensemble_learning_with_umda.py", line 65, in <module>
   history = w.optimize_umda(opt_fn, n_agents, n_variables, n_iterations, hyperparams)
   File "/.../evolutionary_ensembles/utils/wrapper.py", line 110, in optimize_umda
   optimizer = UMDA(hyperparams=hyperparams)
   File "/home/xxx/.local/lib/python3.7/site-packages/opytimizer/optimizers/boolean/umda.py", line 49, in __init__
   self._build(hyperparams)
   File "/home/xxx/.local/lib/python3.7/site-packages/opytimizer/optimizers/boolean/umda.py", line 140, in _build
   self.built)
   Message: 'Algorithm: {self.algorithm} | Hyperparameters: p_selection = %s, bounds = ({%s, %s) | Built: %s.'
   Arguments: ('UMDA', 0.75, 0.05, 0.95, True)

[akrah]

@fariafabio I temporary fixed the problems of the script that generates a lot of files (375) in the output folder.

Now my question: How to generate the results of figures III and IV from all these files? To be considered as reproducible, you should give the procedure (or furnish a script) that directly generate the values of tables. In case of a script, the output could be an image, a latex code or a text file (i.e. a Markdown table).

[fariafabio]

Hi Mr. Pierre,

The project has been updated.

Please, check it and let me know if you have any problem.

Thank you very much and take care!