search

osrf / osrf_pycommon

Commonly needed Python modules, used by Python software developed at OSRF
http://osrf-pycommon.readthedocs.org/
Apache License 2.0
7 stars 18 forks source link

Lots of leaked resources and unclosed files when running tests #62

Open rotu opened 4 years ago

rotu commented 4 years ago

Ros2 Foxy on Ubuntu Focal with Python3.8

 colcon test --packages-select osrf_pycommon                  
Starting >>> osrf_pycommon
[1.106s] ERROR:colcon.colcon_core.task.python.test:Exception in Python testing step extension 'pytest': 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/home/dan/.local/lib/python3.8/site-packages/colcon_core/task/python/test/__init__.py", line 54, in test
    matched = extension.match(self.context, env, setup_py_data)
  File "/home/dan/.local/lib/python3.8/site-packages/colcon_core/task/python/test/pytest.py", line 49, in match
    return has_test_dependency(setup_py_data, 'pytest')
  File "/home/dan/.local/lib/python3.8/site-packages/colcon_core/task/python/test/__init__.py", line 214, in has_test_dependency
    for d in tests_require:
TypeError: 'NoneType' object is not iterable

--- stderr: osrf_pycommon                   
test_call_prepare_arguments (tests.unit.test_cli_utils.test_verb_pattern.TestCliUtilsVerbPattern) ... /opt/ros/master/src/osrf/osrf_pycommon/osrf_pycommon/cli_utils/verb_pattern.py:44: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  arguments, _, _, defaults = inspect.getargspec(func)
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_cli_utils/test_verb_pattern.py:69: DeprecationWarning: Please use assertRaisesRegex instead.
  with self.assertRaisesRegexp(ValueError, 'one or two parameters'):
/opt/ros/master/src/osrf/osrf_pycommon/osrf_pycommon/cli_utils/verb_pattern.py:55: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  ', '.join(inspect.getargspec(func)[0])))
ok
test_split_arguments_by_verb (tests.unit.test_cli_utils.test_verb_pattern.TestCliUtilsVerbPattern) ... ok
test_async_execute_process_no_emulation_shell_false (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908384 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed> stderr=<_UnixReadPipeTransport closed fd=10 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_false_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908386 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_true (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908388 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed> stderr=<_UnixReadPipeTransport closed fd=10 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_true_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908391 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_with_emulation_shell_false (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_false_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_true (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_true_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test__execute_process_nopty (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedReader name=10>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_combined (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:104: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:104: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_combined_unbuffered (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:58: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:58: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_unbuffered (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedReader name=10>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__process_incomming_lines (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... ok
test__execute_process_pty (tests.unit.test_process_utils.test_execute_process_pty.TestProcessUtilsExecuteNoPty) ... ok
test__execute_process_pty_combined (tests.unit.test_process_utils.test_execute_process_pty.TestProcessUtilsExecuteNoPty) ... ok
test__execute_process_pty_combined_unbuffered (tests.unit.test_process_utils.test_execute_process_pty.TestProcessUtilsExecuteNoPty) ... ok
test__execute_process_pty_unbuffered (tests.unit.test_process_utils.test_execute_process_pty.TestProcessUtilsExecuteNoPty) ... ok
test_which (tests.unit.test_process_utils.test_impl.TestProcessUtilsImpl) ... ok
test_remove_ansi_escape_sequences (tests.unit.test_terminal_color.test_ansi_re.TestTerminalColorAnsiRe) ... ok
test_remove_ansi_escape_sequences_false_positives (tests.unit.test_terminal_color.test_ansi_re.TestTerminalColorAnsiRe) ... ok
test_split_by_ansi_escape_sequence (tests.unit.test_terminal_color.test_ansi_re.TestTerminalColorAnsiRe) ... ok
test_ansi (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_enable_and_disable_ansi_color_substitution_globally (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_format_color (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_get_ansi_dict (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_sanitize (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_is_tty (tests.unit.test_terminal_utils.TestTerminalUtils) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908394 returncode=0 stdin=<_UnixWritePipeTransport closed fd=11 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908396 returncode=0 stdin=<_UnixWritePipeTransport closed fd=9 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908398 returncode=0 stdin=<_UnixWritePipeTransport closed fd=11 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908401 returncode=0 stdin=<_UnixWritePipeTransport closed fd=9 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok

----------------------------------------------------------------------
Ran 29 tests in 0.261s

OK
---
Finished <<< osrf_pycommon [0.60s]
rotu commented 4 years ago

still an issue https://ci.ros2.org/job/ci_osx/8271/consoleText:

osrf_pycommon
[94.681s] WARNING:colcon.colcon_core.verb.test:Ignored '--retest-until-pass' for package 'osrf_pycommon' since 'unittest' does not support the usage
test_call_prepare_arguments (tests.unit.test_cli_utils.test_verb_pattern.TestCliUtilsVerbPattern) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/osrf_pycommon/cli_utils/verb_pattern.py:44: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  arguments, _, _, defaults = inspect.getargspec(func)
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_cli_utils/test_verb_pattern.py:69: DeprecationWarning: Please use assertRaisesRegex instead.
  with self.assertRaisesRegexp(ValueError, 'one or two parameters'):
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/osrf_pycommon/cli_utils/verb_pattern.py:55: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  ', '.join(inspect.getargspec(func)[0])))
ok
test_split_arguments_by_verb (tests.unit.test_cli_utils.test_verb_pattern.TestCliUtilsVerbPattern) ... ok
test_async_execute_process_no_emulation_shell_false (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_subprocess.py:126: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=59964 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed> stderr=<_UnixReadPipeTransport closed fd=10 closed>>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_false_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_subprocess.py:126: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=59965 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed>>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_true (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_subprocess.py:126: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=59966 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed> stderr=<_UnixReadPipeTransport closed fd=10 closed>>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_true_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_subprocess.py:126: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=59967 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed>>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_with_emulation_shell_false (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_false_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_true (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_true_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test__execute_process_nopty (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedReader name=10>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_combined (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:104: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:104: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_combined_unbuffered (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:58: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:58: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_unbuffered (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedReader name=10>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok