Closed firefly-cpp closed 3 years ago
"elementwise comparison failed; this will raise an error in the future." is usually because of a type mismatch. For instance:
import numpy as np
arr = np.arange(5) # 0, 1, 2, 3, 4 dtype = np.int64
"1" in arr # returns False because types don't match and puts out a deprecation warning
arr2 = np.array(['0', '1', '2', '3', '4'])
np.array_equal(arr, arr2) # returns False and puts out deprecation warning
I don't see what would cause this in the ES algorithm though. It only seems to get reported in the tests as well. If you run the ES example it won't report any warnings.
" VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray"
This one is because of a bug in CrossRandCurr2Pbest
: https://github.com/NiaOrg/NiaPy/blob/dc1cd459b92d2da8077b131d26c3975240c176a8/NiaPy/algorithms/modified/jade.py#L53
Here the x list elements are a mix of scalars and ndarrays, so a matrix cannot be constructed from it, so asarray casts all elements to object type. This is also what causes the other warning about failed comparissons in that test.
This test can be skipped since the function isn't used anywhere.
Have a good day everyone, I am happy that the current repository is getting updated. NiaPy is a very useful library for many people. Try to run the ES tests like this to get a Backtrace of the warnings:
python3 -m pytest -k "_es" -W error
You could do the same for JADE with the command shown below:
python3 -m pytest -k "_jade" -W error
In this way, each warning would result in an error.
As it was wisely pointed out by @zStupan , there is an inconsistency at some comparisons where a numpy.ndarray
is compared against a Python List
for some tests. With the testing commands below and debugging breakpoints, these cases could be studied more in-depth.
I am glad to see the valuable comments of @zStupan and @sisco0.
From my point of view, there are still many "hidden" bugs in current release candidate. It means that we are still far away from NiaPy v 2.0.
Has anyone time to fix these bugs that were reported by @zStupan?
Some additional warnings appeared when building our docx.
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/algorithms/basic/es.py:docstring of NiaPy.algorithms.basic.EvolutionStrategy1p1.initPopulation:9: WARNING: Unexpected indentation.
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/algorithms/basic/es.py:docstring of NiaPy.algorithms.basic.EvolutionStrategy1p1.runIteration:23: WARNING: Unexpected indentation.
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/algorithms/basic/es.py:docstring of NiaPy.algorithms.basic.EvolutionStrategyML.initPopulation:7: WARNING: Enumerated list ends without a blank line; unexpected unindent.
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/algorithms/other/rs.py:docstring of NiaPy.algorithms.other.RandomSearch.setParameters:5: WARNING: Unexpected indentation.
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/benchmarks/sphere.py:docstring of NiaPy.benchmarks.Sphere:1: WARNING: duplicate object description of NiaPy.benchmarks.Sphere, other instance in api/benchmarks, use :noindex: for one of them
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/benchmarks/sphere.py:docstring of NiaPy.benchmarks.Sphere.Name:1: WARNING: duplicate object description of NiaPy.benchmarks.Sphere.Name, other instance in api/benchmarks, use :noindex: for one of them
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/benchmarks/sphere.py:docstring of NiaPy.benchmarks.Sphere.init:1: WARNING: duplicate object description of NiaPy.benchmarks.Sphere.init, other instance in api/benchmarks, use :noindex: for one of them
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/benchmarks/sphere.py:docstring of NiaPy.benchmarks.Sphere.function:1: WARNING: duplicate object description of NiaPy.benchmarks.Sphere.function, other instance in api/benchmarks, use :noindex: for one of them
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/benchmarks/sphere.py:docstring of NiaPy.benchmarks.Sphere.latex_code:1: WARNING: duplicate object description of NiaPy.benchmarks.Sphere.latex_code, other instance in api/benchmarks, use :noindex: for one of them
/builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/runner.py:docstring of NiaPy.Runner:1: WARNING: duplicate object description of NiaPy.Runner, other instance in api/niapy, use :noindex: for one of them looking for now-outdated files... none found
I got to fix some warnings, which are present in https://github.com/sisco0/NiaPy/commit/00aa68b7e251448c7c8bbe174fcf9ea3c047d020 and https://github.com/sisco0/NiaPy/commit/e5c753fc496e5cd9da0f8023cf393e69a0349881.
But, when trying to fix the JADE issue it seems that x
is not consistently calculated at CrossRandCurr2Pbest
. I attach the interesting source lines under study here:
https://github.com/NiaOrg/NiaPy/blob/dc1cd459b92d2da8077b131d26c3975240c176a8/NiaPy/algorithms/modified/jade.py#L17-L54
It seems that some elements are numbers, while others are arrays in the x
object. If all of them should be positions then all of them should be arrays. I am not sure of my conclusions as now I am researching the paper linked.
I attach the output of pdb
for x
here as well:
> /home/thebittendonut/dev/NiaPy/NiaPy/algorithms/modified/jade.py(55)CrossRandCurr2Pbest()
-> import pdb; pdb.set_trace()
(Pdb) p x
[-91.17912633708372, -22.72754052518438, 86.4972937536084, array([37.6929134 , 18.0288306 , 66.23389883, 69.48774771, 12.70002755,
78.95772088, 80.91229664, 57.74606603, 46.52453118, 40.05271575]), 98.31891564794938, array([-18.000223 , -34.0996604 , -77.98610639, -76.0043742 ,
-97.53318287, -54.40052513, -20.29343077, -78.85750992,
-13.79769366, -11.57254025]), array([-27.72220261, -50.76700597, 17.60615978, -32.09545235,
-17.55285624, 29.06832172, 6.29216357, 10.45664726,
-44.79441646, -41.08245657]), -3.609677182164205, array([ -45.76234021, -105.20215609, -37.42246389, -88.4825519 ,
-99.20263108, -62.21999831, -74.78654291, -19.64553972,
-55.51563055, -5.54763639]), -40.640832103704106]
These should be modified to be a vstack
instead of asarray
as well in order to work correctly with Numpy under my opinion.
Is not the error related to these lines attached below? https://github.com/NiaOrg/NiaPy/blob/dc1cd459b92d2da8077b131d26c3975240c176a8/NiaPy/algorithms/modified/jade.py#L42-L44
In my opinion, it seems that this function should get the population scores and do the argsort
over these and not the positions (agents).
After the research, JADE was fixed as stated in https://github.com/sisco0/NiaPy/commit/c31a243a5aa1e6b570ca49827e0d3b9733caa87e
I would continue to fix these warnings, after that the Pull Request would be done.
Currently I am stuck at the ASO tests, as it hangs at the part shown:
❯ python3 -m pytest -k "_aso" -W error --verbose
===================================================================================== test session starts =====================================================================================
platform linux -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/thebittendonut/.local/share/virtualenvs/NiaPy-ZNxshtqu/bin/python3
cachedir: .pytest_cache
rootdir: /home/thebittendonut/dev/NiaPy
collected 966 items / 938 deselected / 28 selected
NiaPy/tests/test_aso.py::AlgorithmTestCase::test_algorithm_get_parameters_fine PASSED [ 3%]
NiaPy/tests/test_aso.py::AlgorithmTestCase::test_algorithm_info_fine PASSED [ 7%]
NiaPy/tests/test_aso.py::AlgorithmTestCase::test_algorithm_run PASSED [ 10%]
NiaPy/tests/test_aso.py::AlgorithmTestCase::test_algorithm_type_parameters PASSED [ 14%]
NiaPy/tests/test_aso.py::ASOTestCase::test_algorithm_get_parameters_fine PASSED [ 17%]
NiaPy/tests/test_aso.py::ASOTestCase::test_algorithm_info_fine PASSED [ 21%]
NiaPy/tests/test_aso.py::ASOTestCase::test_algorithm_run PASSED [ 25%]
NiaPy/tests/test_aso.py::ASOTestCase::test_algorithm_type_parameters PASSED [ 28%]
NiaPy/tests/test_aso.py::ASOTestCase::test_parameter_types PASSED [ 32%]
NiaPy/tests/test_aso.py::ASOElitismTestCase::test_algorithm_get_parameters_fine PASSED [ 35%]
NiaPy/tests/test_aso.py::ASOElitismTestCase::test_algorithm_info_fine PASSED [ 39%]
NiaPy/tests/test_aso.py::ASOElitismTestCase::test_algorithm_run PASSED [ 42%]
NiaPy/tests/test_aso.py::ASOElitismTestCase::test_algorithm_type_parameters PASSED [ 46%]
NiaPy/tests/test_aso.py::ASOElitismTestCase::test_custom_works_fine
@sisco0 Running with -Werror will hang because the algorithm test case runs 2 identical algorithm instances in separate threads, and the exception isn't handled in the thread. It should be handled in the Algorithm class itself, but that feature isn't implemented correctly at the moment. The run method should catch any exception and save it in self.exception I'm assuming, and then you could call Algorithm.bad_run() to check if an error was thrown.
EDIT: This should do the trick:
def run(self, task):
r"""Start the optimization.
Args:
task (Task): Optimization task.
Returns:
Tuple[numpy.ndarray, float]:
1. Best individuals components found in optimization process.
2. Best fitness value found in optimization process.
See Also:
* :func:`NiaPy.algorithms.Algorithm.runTask`
"""
try:
r = self.runTask(task)
return r[0], r[1] * task.optType.value
except (FesException, GenException, TimeException, RefException):
return task.x, task.x_f * task.optType.value
except Exception as e:
self.exception = e
return None, None
Thank you for the brilliant answer @zStupan , as you wisely pointed out the ASO test is not working when using -W error
, but works correctly (with warnings) when not using that option. I am trying to research what is causing the warning for that.
Update: Thanks for the help given, I could finally fix that warning.
In order to run all the tests in parallel I used the recommendation at here. Installed the package and used the -n NPROCESSORS
@sisco0 Awesome!
@GregaVrbancic Could you maybe try adding that pytest extension to the CI, so the test would execute quicker on pull requests?
The complete tests suite seem to be passing successfully without any warning now.
I created the Pull Request attached above and updated it in order to pass flake8 linting test. I am not sure why does in my log file some tests appear as skipped (maybe they are deactivated for some reason?). Even more, I would like to know if an Issue could be created in order to expand how JADE is tested, as, hopefully, a better test suite for JADE could have shown the error we were having.
Added to the aforementioned, I would like to thank @zStupan for guiding me through the debugging process.
@sisco0 No problem, always happy to help.
The JADE and SADE tests,are skipped because the algorithms aren't implemented yet.
I have been thinking about the parallel CI workflow and I came to a solution I would like to show you, which is present at my forked repository https://github.com/sisco0/NiaPy/tree/faster-ci . Please take a look at the Actions section where techniques as cache are used and timing results are showing.
If reducing the CI time would be useful in this way, while uploading a coverage report artifact, please inform me to create the corresponding Pull Request @firefly-cpp @GregaVrbancic .
These are the timing results for (Linux, Windows) and Python 3.8.
I have been thinking about the parallel CI workflow and I came to a solution I would like to show you, which is present at my forked repository https://github.com/sisco0/NiaPy/tree/faster-ci . Please take a look at the Actions section where techniques as cache are used and timing results are showing.
If reducing the CI time would be useful in this way, while uploading a coverage report artifact, please inform me to create the corresponding Pull Request @firefly-cpp @GregaVrbancic .
These are the timing results for (Linux, Windows) and Python 3.8.
@sisco0 thanks for contributing. Can you please elaborate a bit more on how the testing is quicker (looking at the first screenshot for Python 2.7 on Ubuntu) since the test it is just split into multiple smaller parts? But when you sum up the consumed time of those smaller parts it exceeds the time consumed in our current workflow (for example: https://github.com/NiaOrg/NiaPy/runs/2226732631).
But I do agree, that quicker CI would be awesome. First, we could increase the number of max-parallel to 20. If I am not mistaken this is the maximum value available for our project. I would suggest, we improve the workflow step by step and in the process monitor if everything works as expected.
Additionally, if making any changes to the workflow, please make sure you only update the main.yml. The other definitions are for automatic publishing new releases on pypi and conda and are currently working perfectly.
I understand @GregaVrbancic. Tomorrow I would try to run some tests and hopefully I could return with a solution where the sum of minutes be the same or lower using a parallel approach. It must be taken into account that GitHub actions cache are now used at the fork so a good management of these caches could reduce our times (pipenv install times for example).
Deprecation warnings were fixed in https://github.com/NiaOrg/NiaPy/pull/299, closing this issue. As stated in the conversation, maybe some improvement could be made on CI times in the future.
Hi everyone!
During the build of NiaPy rpm package, pytest reported several deprecation warnings.
I am attaching a partial report created by pytest.
=============================== warnings summary =============================== NiaPy/tests/test_aso.py::ASOElitismTestCase::test_custom_works_fine NiaPy/tests/test_aso.py::ASOElitismTestCase::test_griewank_works_fine NiaPy/tests/test_fa.py::FATestCase::test_griewank_works_fine NiaPy/tests/test_fa.py::FATestCase::test_works_fine NiaPy/tests/test_jade.py::CrossRandCurr2pbestTestCase::test_function_fine /usr/lib/python3.9/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray return array(a, dtype, copy=False, order=order) NiaPy/tests/test_es.py: 16660 warnings /builddir/build/BUILD/NiaPy-2.0.0rc13/NiaPy/algorithms/basic/es.py:329: DeprecationWarning: elementwise comparison failed; this will raise an error in the future. if e not in c: k += 1 NiaPy/tests/test_jade.py: 200 warnings /usr/lib/python3.9/site-packages/numpy/core/numeric.py:2378: DeprecationWarning: elementwise comparison failed; this will raise an error in the future. return bool(asarray(a1 == a2).all()) -- Docs: https://docs.pytest.org/en/stable/warnings.html