Status of "in code" roscore execution

What is the status of the roscore execution which does not depend on running it within a subprocess of the draft code? (I am currently working at the same topic for the project rosdbc (Issue: Get rid of need for spawn subprocess for roscore execution).)

The script you linked to here is just a dumb script I use to profile pyros execution when needed, where I launch everything at once.

Pyros code assumes roscore is already launched somewhere, which is usually done by the user or roslaunch itself if needed, when a ROS user launches a ROS system.

I am using however this code sometimes, when I run purely in a python workflow, and where one might never use roslaunch at all : https://github.com/pyros-dev/pyros-utils/blob/devel/pyros_utils/ros_utils.py

Ok. It is still not clear to me how the tests in tests/test_ros_utils.py can be run without manually starting roscore before. Did I miss some additional setup code?

It looks like you can just run the test module with nose, as a usual nose test module. roscore will get started by the test if needed (see the get_master() call ?)

You need to be careful with your processes though, as nose will launch one process for all tests, meaning all test might (if roscore is already launched) or might not, share the same roscore process which might lead to problems given ROS design.

I had the intention to replace nose with pytest eventually because pytest has the '--boxed' option to make sure each test runs in its own process, but then running the tests becomes even less obvious, so not sure yet when I ll come around to do it.

On Nov 13, 2017 1:01 AM, "thinwybk" notifications@github.com wrote:

Ok. It is still not clear to me how the tests in tests/test_ros_utils.py https://github.com/pyros-dev/pyros-utils/blob/devel/pyros_utils/tests/test_ros_utils.py#L112 can be run without manually starting roscore before. Did I miss some additional setup code?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/asmodehn/pyros/issues/139#issuecomment-343747138, or mute the thread https://github.com/notifications/unsubscribe-auth/AANgSJDN7b8IrKE4ZrEpDXuMHOg_ZyPvks5s1xZ1gaJpZM4Qa1Nv .

It looks like you can just run the test module with nose, as a usual nose test module. roscore will get started by the test if needed (see the get_master() call ?)

Ok.

You need to be careful with your processes though, as nose will launch one process for all tests, meaning all test might (if roscore is already launched) or might not, share the same roscore process which might lead to problems given ROS design.

I am using a simple unittest class level setup and teardown for the roscore and execute the tests with rosunit.unitrun(...). This works but I am searching for and I am open for better solution.

I had the intention to replace nose with pytest eventually because pytest has the '--boxed' option to make sure each test runs in its own process, but then running the tests becomes even less obvious, so not sure yet when I ll come around to do it.

I like pytest but did not want to depend on it so far.

I used to do my tests like you're doing but I enventually decided to depend on pytest, because it s better than nose for my use in the amount of flexibility you get, the API, and the whole test runner behavior (plus plugins).

Thats why I also support pytest in catkin_pip, so it should be easy to integrate with your ROS pipeline if you decide to go that way.

The other option would be to stick to unittest API strictly, but in that case patching rostest might be the way to go, for every one to get the benefit of the features, provided maintainers merge them.

On Nov 13, 2017 1:46 AM, "thinwybk" notifications@github.com wrote:

It looks like you can just run the test module with nose, as a usual nose test module. roscore will get started by the test if needed (see the get_master() call ?)

Ok.

You need to be careful with your processes though, as nose will launch one process for all tests, meaning all test might (if roscore is already launched) or might not, share the same roscore process which might lead to problems given ROS design.

I am using a simple unittest class level setup and teardown for the roscore https://github.com/fkromer/rosdbc/blob/master/clients/rosdbcpy/test/test_parameter_checks.py and execute the tests with rosunit.unitrun(...). This works but I am searching for and I am open for better solution.

I had the intention to replace nose with pytest eventually because pytest has the '--boxed' option to make sure each test runs in its own process, but then running the tests becomes even less obvious, so not sure yet when I ll come around to do it.

I like pytest but did not want to depend on it so far.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/asmodehn/pyros/issues/139#issuecomment-343750209, or mute the thread https://github.com/notifications/unsubscribe-auth/AANgSFchp288hcBvqy2kXr4Z5r_4MG0Zks5s1yDrgaJpZM4Qa1Nv .

I used to do my tests like you're doing but I enventually decided to depend on pytest, because it s better than nose for my use in the amount of flexibility you get, the API, and the whole test runner behavior (plus plugins).

Have you thought about creating a pytest plugin (with generic ROS functionality) instead of using pytest (with your custom code) as well? I though about that because the concepts provided in pytest are very powerful (e.g. fixtures) and could not added to rostest with reasonable effort I guess.

Thats why I also support pytest in catkin_pip, so it should be easy to integrate with your ROS pipeline if you decide to go that way.

That seems very helpful. (That should work with a pytest plugin for ROS as well.)

The other option would be to stick to unittest API strictly, but in that case patching rostest might be the way to go, for every one to get the benefit of the features, provided maintainers merge them.

It seems like the maintainers interest/focus is not on working on ROS1 but ROS2 instead. That's good but would limit the availability of merged functionality. It would be hard to impossible to get all "pytest nice to have" concepts into rostest at all. (Probably I will be working with ROS2 before that happens :wink: From what I know so far the test capabilities of ROS2 are a lot more powerful than the ones of ROS1.)

Have you thought about creating a pytest plugin (with generic ROS functionality) instead of using pytest (with your custom code) as well? I though about that because the concepts provided in pytest are very powerful (e.g. fixtures) and could not added to rostest with reasonable effort I guess.

Yes I thought about that, but I am not sure it is worth it. The main differences between python and ROS as far as I can see are :

serialization format for network messages.
static description of launch process.

So roughly speaking, pytest can already test any python program. Now to allow it to test any ROS program, we could add plugins to understand the .msg format and the .launch format. But making a pytest plugin for that would make it specific to pytest, where one might want to have that feature in another context ( his custom test framework, some runtime checking library, etc.) In the end I think these would be more useful as implemented separately, and actually, they already are, somewhere in the core ros libraries.

So I just think the current problems are "implementation problems" that should be fixed in these libraries, to make them behave as a python user would expect, and usable by "de facto standard frameworks" like pytest, or nose or any other. Which is why I am taking a "bottom up" approach to use ROS as a python dev, and sends PR for the "problems" I encounter along the way.

It seems like the maintainers interest/focus is not on working on ROS1 but ROS2 instead.

Yes, however, I suspect many overlooked issues of ROS1 are still creeping into ROS2 (like non-unicode string type for messages for example), mainly because people working on ROS2 didn't encounter significant problems with them in ROS1 (usually from a C++ perspective, python devs being the minority). Working on these issues in ROS1 in python allows to kickstart some relevant prototyping and informed debate from the python perspective, hopefully before setting anything for ROS2 in stone.

I have been focusing mostly on the ability to do dynamic setup and message generation, but I am pretty sure there are some low hanging fruits somewhere in roslaunch and rostest, that could improve user experience from python a lot (rostest is pretty terrible, I had it sometimes returning "PASSED" after a test node crashed for example - but didn't investigate).

Finding these problems and fixing them as a usual python software dev would expect them to work, would inform how to develop their ROS2 counterparts to not have them in the first place ( ideally by reusing some python feature, lib, tool, or even specification, instead of - badly - reimplementing it ).

But making a pytest plugin for that would make it specific to pytest, where one might want to have that feature in another context ( his custom test framework, some runtime checking library, etc.) In the end I think these would be more useful as implemented separately, and actually, they already are, somewhere in the core ros libraries.

That's true. Implementing the functionality separate would be better. What functionality in the ros library do you mean and where can I find it?

So I just think the current problems are "implementation problems" that should be fixed in these libraries, to make them behave as a python user would expect, and usable by "de facto standard frameworks" like pytest, or nose or any other.

I am not sure if these "implementation problems" can be patched at all because from my perspective at least some would require major extension/refactoring of functionality. An example: In some tests you would like to start several nodes in a deterministic way (w.r.t. to order not timing). As roslaunch is not deterministic in how it starts nodes it is not possible to start nodes in rostest in a deterministic manner as well. Right now the only way to workaround this limitation seems to use dirty hacks like starting roslaunch several times from a script with delays between specific node starts.

Yes, however, I suspect many overlooked issues of ROS1 are still creeping into ROS2 (like non-unicode string type for messages for example), mainly because people working on ROS2 didn't encounter significant problems with them in ROS1 (usually from a C++ perspective, python devs being the minority). Working on these issues in ROS1 in python allows to kickstart some relevant prototyping and informed debate from the python perspective, hopefully before setting anything for ROS2 in stone.

Wow, for obvious issue like your example that shouldn't happen. I have an embedded background, did not develop C++ in ROS yet and am mainly involved in test automation right now. That means I am not biased from the ROS C++ developer perspective. It seems to me that (a) people which were involved in (structured) testing before know about problems and limitations w.r.t. testing but skip node integration testing more or less (they must because I wouldn't believe that they can test in a structured manner with dirty hacks like the one described above) or (b) they know about how to test in a structured manner at all (or make people believe they are not able to do it that they don't have to do it what is even worse).

I am taking the same exactly approach with e.g. rosdbc to make people aware of limitations and possible workarounds (in the case of rosdbc it is the limitation in testing which could be compensated with debugging/monitoring to some degree).

(BTW: Many ROS C++ only developers seem to not know about and do not understand the value of Python in ROS development because they do not understand the programming language value/suitability for production code vs. infrastructure/test code.)

I have been focusing mostly on the ability to do dynamic setup and message generation, but I am pretty sure there are some low hanging fruits somewhere in roslaunch and rostest, that could improve user experience from python a lot (rostest is pretty terrible, I had it sometimes returning "PASSED" after a test node crashed for example - but didn't investigate).

Dynamic setup and message generation is very valuable. What low "hanging fruits" do you mean? From my perspective adding generic fake nodes and test nodes to rostest would be plain adding (instead of refactoring what could take longer to be PR accepted).

Finding these problems and fixing them as a usual python software dev would expect them to work, would inform how to develop their ROS2 counterparts to not have them in the first place ( ideally by reusing some python feature, lib, tool, or even specification, instead of - badly - reimplementing it ).

I try to make people involved in the development of ROS2 aware of potential problems like the support of the test framework for grey-box testing as well.

With "low hanging fruits" I was mostly referring to little bugs here and there making the software unusable for user a bit more picky than the average. Fixing these would be the first step IMHO.

An example: In some tests you would like to start several nodes in a deterministic way (w.r.t. to order not timing).

As far as I can recall this is not even specified in ROS1. One might argue that it is better because it forces the developer to make sure it works in all possible launch sequences. Yet it is not easily testable...

So for this, it would be useful to check the ROS2 spec and implementation of launch and find out what needs to be fixed in ROS1 implementation, and if something needs to be fixed in ROS2 spec. I would expect by the way ROS2 tools to eventually work with ROS1 file format, and that can be a missing spec as well...

On Nov 19, 2017 8:05 PM, "thinwybk" notifications@github.com wrote:

But making a pytest plugin for that would make it specific to pytest, where one might want to have that feature in another context ( his custom test framework, some runtime checking library, etc.) In the end I think these would be more useful as implemented separately, and actually, they already are, somewhere in the core ros libraries.

That's true. Implementing the functionality separate would be better. What functionality in the ros library do you mean and where can I find it?

So I just think the current problems are "implementation problems" that should be fixed in these libraries, to make them behave as a python user would expect, and usable by "de facto standard frameworks" like pytest, or nose or any other.

I am not sure if these "implementation problems" can be patched at all because from my perspective at least some would require major extension/refactoring of functionality. An example: In some tests you would like to start several nodes in a deterministic way (w.r.t. to order not timing). As roslaunch is not deterministic in how it starts nodes it is not possible to start nodes in rostest in a deterministic manner as well. Right now the only way to workaround this limitation seems to use dirty hacks like starting roslaunch several times from a script with delays between specific node starts https://answers.ros.org/question/233353/set-delay-between-starting-nodes-within-launch-file/?answer=233740#post-id-233740 .

Yes, however, I suspect many overlooked issues of ROS1 are still creeping into ROS2 (like non-unicode string type for messages for example), mainly because people working on ROS2 didn't encounter significant problems with them in ROS1 (usually from a C++ perspective, python devs being the minority). Working on these issues in ROS1 in python allows to kickstart some relevant prototyping and informed debate from the python perspective, hopefully before setting anything for ROS2 in stone.

Wow, for obvious issue like your example that shouldn't happen. I have an embedded background, did not develop C++ in ROS yet and am mainly involved in test automation right now. That means I am not biased from the ROS C++ developer perspective. It seems to me that (a) people which were involved in (structured) testing before know about problems and limitations w.r.t. testing but skip node integration testing more or less (they must because I wouldn't believe that they can test in a structured manner with dirty hacks like the one described above) or (b) they know about how to test in a structured manner at all (or make people believe they are not able to do it that they don't have to do it what is even worse).

I am taking the same exactly approach with e.g. rosdbc https://github.com/fkromer/rosdbc to make people aware of limitations and possible workarounds (in the case of rosdbc it is the limitation in testing which could be compensated with debugging/monitoring to some degree).

(BTW: Many ROS C++ only developers seem to not know about and do not understand the value of Python in ROS development because they do not understand the programming language value/suitability for production code vs. infrastructure/test code.)

I have been focusing mostly on the ability to do dynamic setup and message generation, but I am pretty sure there are some low hanging fruits somewhere in roslaunch and rostest, that could improve user experience from python a lot (rostest is pretty terrible, I had it sometimes returning "PASSED" after a test node crashed for example - but didn't investigate).

Dynamic setup and message generation is very valuable. What low "hanging fruits" do you mean? From my perspective adding generic fake nodes and test nodes to rostest would be plain adding (instead of refactoring what could take longer to be PR accepted).

Finding these problems and fixing them as a usual python software dev would expect them to work, would inform how to develop their ROS2 counterparts to not have them in the first place ( ideally by reusing some python feature, lib, tool, or even specification, instead of - badly - reimplementing it ).

I try to make people involved in the development of ROS2 aware of potential problems like the support of the test framework for grey-box testing https://discourse.ros.org/t/grey-box-node-nodelet-unit-testing-with-ros2/3037 as well.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/asmodehn/pyros/issues/139#issuecomment-345508382, or mute the thread https://github.com/notifications/unsubscribe-auth/AANgSFS9Y8Ws8nwJCTcRdl42VfQRNv82ks5s4AuUgaJpZM4Qa1Nv .

pyros-dev / pyros

Status of "in code" roscore execution #139