Closed ChaoticRoman closed 1 year ago
Pytest can discover and run unittests but pytest-memray does not work on these.
Can you clarify what's the error here?
Pytest can discover and run unittests but pytest-memray does not work on these.
Can you clarify what's the error here?
There are two most common ways how to write tests in Python:
unittest
module: subclassing unittest.TestCase
, tests are then those its methods with name starting with test_
prefix.pytest
extension module. Tests are all functions in top-level namespace with name starting with test_
prefix.This example illustrates both these ways: https://github.com/ChaoticRoman/pytest-monitor-example/blob/memray-example/tests/test_float.py
The officially supported way is the one using unittest
module and most open source and enterprise projects use this one. But pytest
is very often used as a test discovery and test runner for these, because pytest
does support both of these, it has more customizable output and there are many extensions e.g. for parallel execution, CI/CD integration and others. The unfortunate fact is that both available memory profiling extensions (pytest-monitor
and pytest-memray
) do not support other tests than those in native pytest
format.
This 40 lines long script does implement memory profiler with unittest
s discovery:
https://github.com/ChaoticRoman/pytest-monitor-example/blob/custom_memory_profiler/unittestit.py
but it would be nice to have full support through a pytest
extension, either via pytest-monitor
or pytest-memray
instead of three tools with incomplete features.
My question was why does it not support it? Doesn't just works? What happens if you try to use it?
@gaborbernat It does not :( Memory allocation is reported for native pytests but not for unittests (although those are discovered and executed by pytest). Using the example referenced above:
# pytest -v --memray tests/
========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.9.2, pytest-7.1.2, pluggy-0.13.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /opt/pytest-monitor-example
plugins: memray-1.1.0, monitor-1.6.3
collected 6 items
tests/test_float.py::TestStringMethods::test_correct_string PASSED [ 16%]
tests/test_float.py::TestStringMethods::test_empty_string PASSED [ 33%]
tests/test_float.py::TestStringMethods::test_invalid_string PASSED [ 50%]
tests/test_float.py::test_empty_string PASSED [ 66%]
tests/test_float.py::test_allocate_1MB PASSED [ 83%]
tests/test_float.py::test_allocate_100MB PASSED [100%]
============================================================================================= MEMRAY REPORT ==============================================================================================
Allocations results for tests/test_float.py::test_allocate_100MB
📦 Total memory allocated: 107.5MiB
📏 Total allocations: 200
📊 Histogram of allocation sizes: |█|
🥇 Biggest allocating functions:
- <listcomp>:/opt/pytest-monitor-example/tests/test_float.py:34 -> 107.5MiB
Allocations results for tests/test_float.py::test_allocate_1MB
📦 Total memory allocated: 1.1MiB
📏 Total allocations: 122
📊 Histogram of allocation sizes: |█|
🥇 Biggest allocating functions:
- <listcomp>:/opt/pytest-monitor-example/tests/test_float.py:34 -> 1.1MiB
=========================================================================================== 6 passed in 1.02s ============================================================================================
Just +1 on this, i've tried this today with a package using unittest.TestCase
for all the tests and I don't get any memray output:
$ pytest -v --memray
====================================== test session starts =======================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /bin/python
cachedir: .pytest_cache
rootdir: XXXXXX
plugins: memray-1.2.0
collected 21 items
tests/test_agent.py::testAgentClass::test_get_journey PASSED [ 4%]
tests/test_agent.py::testAgentClass::test_get_name PASSED [ 9%]
tests/test_agent.py::testAgentClass::test_get_name_unique PASSED [ 14%]
tests/test_agent.py::testAgentClass::test_get_position PASSED [ 19%]
tests/test_agent.py::testAgentClass::test_update PASSED [ 23%]
tests/test_agent.py::testRandomWalkClass::test_instantiation PASSED [ 28%]
tests/test_agent.py::testRandomWalkClass::test_step PASSED [ 33%]
tests/test_agent.py::testRandomWalkClass::test_update PASSED [ 38%]
tests/test_world.py::testWorldClass::test_get_name PASSED [ 42%]
tests/test_world.py::testWorldClass::test_get_name_unique PASSED [ 47%]
tests/test_world.py::testWorldClass::test_get_population PASSED [ 52%]
tests/test_world.py::testWorldClass::test_limits PASSED [ 57%]
tests/test_world.py::testWorldClass::test_populate PASSED [ 61%]
tests/test_world.py::TestPopulationClass::test_instanstiation PASSED [ 66%]
tests/test_world.py::TestPopulationClass::test_spawn PASSED [ 71%]
tests/test_world.py::TestPopulationClass::test_spawn_fail PASSED [ 76%]
tests/test_world.py::TestToeGuardClass::test_instanstiation PASSED [ 80%]
tests/test_world.py::TestToeGuardClass::test_step PASSED [ 85%]
tests/test_world.py::TestToeGuardClass::test_step_limits PASSED [ 90%]
tests/test_world.py::TestToeGuardClass::test_step_noShared PASSED [ 95%]
tests/test_world.py::TestToeGuardClass::test_step_value PASSED [100%]
========================================= MEMRAY REPORT ==========================================
======================================= 21 passed in 4.25s =======================================
Although in all fairness the pytest
docs on using unittest says:
The following pytest features do not work, and probably never will due to different design philosophies:
- Fixtures (except for autouse fixtures, see below);
- Parametrization;
- Custom hooks;
Third party plugins may or may not work well, depending on the plugin and the test suite.
I'm unclear what changes might be required to make this work with unittests but will do some reading.
We use custom hooks, so unfortunately is very unlikely that we will be able to support it if pytest doesn't :(
Closing this as purest doesn't allow us to do this easily
Feature Request
Pytest can discover and run unittests but pytest-memray does not work on these. An example:
https://github.com/ChaoticRoman/pytest-monitor-example/tree/memray-example
There is an exactly same issue with alternative pytest memory profiler called pytest-monitor:
https://github.com/CFMTech/pytest-monitor/issues/39
I made very simple memory profiler for unittests that can be helpful for someone encountering this issue:
https://github.com/ChaoticRoman/pytest-monitor-example/blob/custom_memory_profiler/unittestit.py