search

pyros-dev / pyros

Python interfacing for multiprocess software - A Python blanket for ROS to hide inside
BSD 3-Clause "New" or "Revised" License
25 stars 4 forks source link

Pyros for exposing ros services and topics as pure python API #70

Open dhirajdhule opened 8 years ago

dhirajdhule commented 8 years ago

I am following your work from rostful. Thank you very much for all the amazing work. I am figuring out the prospects of pyros for dynamically creating pure python APIs for the services and topics available inside ros environment. I read the documentation and feel like pyros can be used for such applications. Currently I am using boost python libraries on top of cpp bindings which is difficult to scale. What I am unclear about is how to get started. If I could get examples / tutorials for interfacing simple topics and services, I will be able to test it fast. Also I am curious if I can contribute to the project in any form.

dhirajdhule commented 8 years ago

Btw, I cloned pyros and pyros-setup. And tried running few tests in pyros using rosrun, many of which failed. I guess I am not going in the right direction. It will be great if you could direct me to get started.

asmodehn commented 8 years ago

Indeed that is my goal for pyros : dynamically build python interfaces for existing ros environment. This will enable using normal pure python packages on top of running ROS systems.

Both pyros and pyros-setup are still under development, but I am quite limited in the time I can assign to work on them, so anything you can think about in order to achieve the goal would be useful to add. I usually store my ideas of "what would be good to add" as issues, so feel free to do the same. It all start with the right idea/concept.

Maybe I should start by describing my development environment, since I haven't done much/any work on supporting other environments yet, and that may be the first reason you encounter issues :

I try to work on these project like on any other pure python project. The main benefit being that I can use all the usual features, tools and framework that work with python ( multi environment tests, isolated environment tests, versions of dependencies, distributed multiprocess framework, web frameworks, etc.) But I am encountering some hurdle due to ROS environment that I m trying to solve as they occur.

I think you should start getting familiar with pyros-setup as a first step. It's a tiny package that aims at doing the whole ROS setup dynamically if needed, but also shouldn't get in the way if it's not. After pyros-setup is imported correctly, the correct ROS libraries will be available to use. Although tiny it can be quite tricky to get right since it needs to do its job at import time (which is why I separated it from pyros), and there is still a lot of improvements needed there... So please register an issue for what you tried and what was behaving differently that expected in pyros-setup. I ll address them one by one there.

Pyros deals with the multiprocess and communication aspect of that interface. But to be able to bridge one multiprocess library with others, I also need a multiprocess library (ZMP folder in pyros - which I aim to externalize as pip package at some point). The interfacing is at the moment quite limited ( everything is exposed as a service ) - but it s enough already for a service based interface (request - reponse model)

And looking at this issue, it seems the first thing that would be very useful for these package, would be documentation (readme or sphinx docs). Especially as you are discovering them, documenting the different steps while you're getting to know the packages will be extremely useful (how to setup, how to test, how to use). I've been inside them for quite a long time, and I often don't see the obvious things that users need to know about.

I advise you check also the .travis.yml file. Travis run the tests, and if the tests are passing for travis they should pass for you as well. Also travis is doing it the ROS way which seems is what you are most familiar with, so I think it would be a good start for you. Ultimately I wish these packages would be usable the ROS way, or the python way, which ever fit the user, but I'm still searching for a way to achieve that...

asmodehn commented 8 years ago

About tutorials and examples, for rostful I did this : https://github.com/asmodehn/rostful-examples thinking that it would be useful for new comers.

But I have absolutely no time to maintain it, which means it's already obsolete, and probably more confusing than useful. So trying to learn this lesson, I think there should be another way to have tutorials and examples. Probably documentation with doctests, that will eventually end up in sphinx... The main blocker to this right now being "how to simulate ROS systems" in such simple use case, which is the point of the "mockinterface" more or less, but it still needs a lot of improvements.

So I guess the closest thing to a tutorial that is available right now, would be this : https://github.com/asmodehn/pyros/blob/indigo-devel/src/pyros/tests/testPyrosClient.py Quick summary :

  1. setup pyros node (PyrosMock or PyrosROS)
  2. setup pyros client (PyrosClient)
  3. call client methods from your code

steps 1. and 2. are already done for you if you use the context manager : https://github.com/asmodehn/pyros/blob/indigo-devel/src/pyros/tests/testPyrosCtxServer.py#L26 This way you don't have to care about the cleanup, and this is how resources should be handled in python given all the possible exceptions that will occur in many places.

dhirajdhule commented 8 years ago

FIne. I will start with steps in quick summary. Will post the issues I come across here. And yes I can document all the obvious things that user might want to know so anybody else can get started easily. Initially I will try to do things pythonic way as you recommended and then try out the ros way as well. Because my rest of the robot arch is in ros and I need just pure python APIs. Lets see, If I find pythonic way more simple then It will be good to use. I will post as I come across new issues. And yes I will first try to learn from all the docs that you have written, until now I was reading only the readme. Once I am successful in running it I will try to post some kind of tutorial to integrate pyros with your environment for other users. For now that could be the useful contribution from me. Thanks for the amazing work.

dhirajdhule commented 8 years ago

Here is what I tried. I am just trying to wrap my head around pyros and pyros-setup. Pyros trials.

1 cloned pyros and pyros-setup from git and created a catkin repo. compiled with catkin_make 2 sourced devel/setup.bash from catkin repo. 3 Tried running nosetest on pyros-setup/pyros_setup/tests/ successful files: test_ros_setup.py failed files: test_ros_utils.py, test_rostest_nose.py

ERROR: pyros_setup.tests.test_rostest_nose.test_import_before_delay_setup_teardown_module

Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/flight/alphaws/src/pyros-setup/pyros_setup/tests/test_rostest_nose.py", line 122, in test_import_before_delay_setup_teardown_module main_test_part(rostest_nose, roslaunch, rospy, rosnode) File "/home/flight/alphaws/src/pyros-setup/pyros_setup/tests/test_rostest_nose.py", line 57, in main_test_part talker_process = launch.launch(node) File "/opt/ros/indigo/lib/python2.7/dist-packages/roslaunch/scriptapi.py", line 100, in launch raise RLException("failed to launch %s/%s"%(node.package, node.type)) RLException: failed to launch pyros_test/string_pub_node.py

-------------------- >> begin captured stdout << ---------------------

ERROR: pyros_setup.tests.test_ros_utils.test_rosnode_started

Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/flight/alphaws/src/pyros-setup/pyros_setup/tests/test_ros_utils.py", line 113, in test_rosnode_started echo_process = launch.launch(echo_node) File "/opt/ros/indigo/lib/python2.7/dist-packages/roslaunch/scriptapi.py", line 100, in launch raise RLException("failed to launch %s/%s"%(node.package, node.type)) RLException: failed to launch pyros_test/echo.py -------------------- >> begin captured stdout << ---------------------

4 Pulled pyros-setup master 5 now catkin make fails with error

CMake Error at /opt/ros/indigo/share/catkin/cmake/catkinConfig.cmake:75 (find_package): Could not find a package configuration file provided by "catkin_pure_python" with any of the following names:

catkin_pure_pythonConfig.cmake
catkin_pure_python-config.cmake

Add the installation prefix of "catkin_pure_python" to CMAKE_PREFIX_PATH or set "catkin_pure_python_DIR" to a directory containing one of the above files. If "catkin_pure_python" provides a separate development package or SDK, be sure it has been installed. Call Stack (most recent call first): pyros-setup/CMakeLists.txt:11 (find_package)

6 tried running nosetests 7 now all files failed, maybe because catkin repo wasnt built.

I can see some PRs with catkin_pure_python, is it solved yet?

And do I really need to get into pyros-setup? Or else I can just hop onto configuring pyros? Btw nosetests on pyros ran successfully including rosinterface.

dhirajdhule commented 8 years ago

Checked this page https://pypi.python.org/pypi/pyros_setup/0.1.0 now trying to build pyros_setup catkin with catkin_pure_python

Catkin_make successful. But pyros-setup tests are still failing. I guess echo.py doesn't exist yet. And so a node is not being setup. Is there anything I am missing something here?

I also tried running nosetests with pyros-setup from sources. Now it is not able to find ros_setup.py here is the nosetest output.

nosetests ~/alphaws/src/pyros/src/pyros/tests/ .......................Process pyros: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/flight/alphaws/src/pyros/src/pyros/rosinterface/pyros_ros.py", line 150, in run sys.modules["pyros_setup"] = pyros_setup.delayed_import_auto(distro='indigo', base_path=self.base_path) File "/home/flight/alphaws/src/pyros-setup/pyros_setup/utils.py", line 15, in newFunc return func(_args, *_kwargs) File "/home/flight/alphaws/src/pyros-setup/pyros_setup/init.py", line 90, in delayed_import_auto from .ros_setup import ROS_find_workspaces ImportError: No module named ros_setup

E............

ERROR: pyros.tests.testPyrosCtxServer.testPyrosROSCtx

Traceback (most recent call last): File "/home/flight/alphaws/devel/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/flight/alphaws/src/pyros/src/pyros/tests/testPyrosCtxServer.py", line 26, in testPyrosROSCtx with pyros_ctx(mock_node=False, base_path=os.path.join(os.path.dirname(file), '..', '..', '..', '..', '..')) as ctx: File "/usr/lib/python2.7/contextlib.py", line 17, in enter return self.gen.next() File "/home/flight/alphaws/src/pyros/src/pyros/pyros_ctx_server.py", line 55, in pyros_ctx yield ctx(client=pyros_client.PyrosClient(client_conn)) File "/home/flight/alphaws/src/pyros/src/pyros/pyros_client.py", line 60, in init raise PyrosServiceNotFound('msg_build') PyrosServiceNotFound: msg_build -------------------- >> begin captured logging << -------------------- rospy.topics: INFO: topicmanager initialized root: WARNING: Setting up pyros ROS node... root: WARNING: Setting up pyros actual client... --------------------- >> end captured logging << ---------------------

Ran 36 tests in 18.013s

FAILED (errors=1)

asmodehn commented 8 years ago

About pyros-setup, it is needed mostly to autodetect your ros environment ( it will do something similar to 'source setup.bash' for you. If you are used to source the proper ros environment every time before running anything, it is mostly useless for you.

The dependency chain is : Pyros-test <- pyros-setup <- pyros

With the aim for the requirement to be : ROS <- catkin-pip <- python But this is still a work in progress ( going on in the most recent branch - might be already usable though...)

So if you miss ROS components for tests, it s likely to be an issue with pyros-test. If the setup is not done properly it s likely to be an issue with pyros-setup. If the communication has some issue, it will be because of pyros.

I ll have a more detailed look at your issues sometime this weekend... On May 26, 2016 9:55 PM, "Dhiraj D" notifications@github.com wrote:

Checked this page https://pypi.python.org/pypi/pyros_setup/0.1.0 now trying to build pyros_setup catkin with catkin_pure_python

Catkin_make successful. But pyros-setup tests are still failing. I guess echo.py doesn't exist yet. And so a node is not being setup. Is there anything I am missing something here?

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/asmodehn/pyros/issues/70#issuecomment-221862665

dhirajdhule commented 8 years ago

Thats great. Thanks. after installing catkin_pure_python and pyros_tests most problems were solved. There is only one issue with pyros-setup in source and pip installation as well.

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/flight/alphaws/src/pyros/src/pyros/rosinterface/pyros_ros.py", line 150, in run
    sys.modules["pyros_setup"] = pyros_setup.delayed_import_auto(distro='indigo', base_path=self.base_path)
  File "/usr/local/lib/python2.7/dist-packages/pyros_setup/utils.py", line 15, in newFunc
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyros_setup/__init__.py", line 86, in delayed_import_auto
    from .ros_setup import ROS_find_workspaces
ImportError: No module named ros_setup

I checked the file and it seems like you have already noted few problems there. I tried some random stuff to solve the import stuff but not success.

Another error is I guess dependent on this one.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/flight/alphaws/src/pyros/src/pyros/tests/testPyrosCtxServer.py", line 26, in testPyrosROSCtx
    with pyros_ctx(mock_node=False, base_path=os.path.join(os.path.dirname(__file__), '..', '..', '..', '..', '..')) as ctx:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/flight/alphaws/src/pyros/src/pyros/pyros_ctx_server.py", line 55, in pyros_ctx
    yield ctx(client=pyros_client.PyrosClient(client_conn))
  File "/home/flight/alphaws/src/pyros/src/pyros/pyros_client.py", line 60, in __init__
    raise PyrosServiceNotFound('msg_build')
PyrosServiceNotFound: msg_build
asmodehn commented 8 years ago

I edited your post for code sections.

pyros-setup pip package is not the latest from source (and might conflict with pyros-setup ros package...), so maybe lets focus on getting the source version to work for you first...

To run the self tests, please verify that you are using nose version 1.3.7. The default ubuntu trusty version is 1.3.1 and has some issue with importing modules...

Pyros is usable from indigo-devel, that is the version I use with rostful currently in my robot development for work, but the tests in indigo-devel branch are not fully working. I eventually found out a while ago that the 'usual ROS structure' for python packages path/src/package_name was making problem with nose importer feature. So I am fixing these in config_refactor branch, by changing pyros structure to usual python structure path/package_name. I will probably merge it sometime in the next few days/weeks. I am also fixing self tests there, so you might want to try that branch...

One of the recent feature still in development is the use of a configuration file (instead of specifying the path to your workspaces in code, you specify it in a config file). Pyros-setup already uses that config file pyros-setup.cfg. Pyros itself does only from config_refactor branch ). So if there is a import error with these version, it is likely because of the configuration file content.

Note in config_refactor branch all self tests are passing : https://travis-ci.org/asmodehn/pyros/builds/133680922, so you might want to compare what you are doing versus what travis is doing...

Let me know how this goes !

dhirajdhule commented 8 years ago

I upgraded nose to 1.3.7. Sourced pyros-setup again.
Switched to pyros config refactor branch. All self tests are passing there.

So with this I have pyros, pyros-setup with all tests passing. Great, seems like I am good to go now. When are you planning to merge config work to indigo-devel? Is there anything I can help with, any testing required? Moving ahead, I'm trying to put together pieces to listen to a ros topic from pyros pure python api. I'm trying to gather these pieces from self tests. So I would first go for importing ros environment (with turtlebot node running) using pyros-setup, then try listening to topics advertised, try calling services exposed by turtlebot, try to get/set params, etc. Pyros seems to be ready for this, please suggest if something could be useful and which I'm not seeing here. Thanks.

asmodehn commented 8 years ago

When I will have all (ROS & pip) packages made, I will probably need some help testing them all. I ll let you know.

As far as I know, everything should be fine for using pyros as you plan. That is what I usually do for tests, rostful, or celeros. You might have some troubles with pyros config file since that feature is pretty new, and getting all imports to work fine for multiple users (from python or from ROS) is tricky. But everything else is already working for me, so it should work for you as well.

Let me know if you find things that would be useful but that are currently missing.

The main limitation now is that pyros is only "client-server" where the python node is client and the ROS node is server. I plan to improve that in the coming months...

Enjoy !

dhirajdhule commented 8 years ago

updates: I am able to call ros services, poll ros topics using pyros_client (similar to testpyrosclient.py). I tried injecting data to existing ros topic but I got rosmsg type errors. Is there any way to specify rosmsg type when injecting to topic ( pyros_client.topic_inject() ) ?

I am doing setup by sending topics, services as kwargs during instantiation of Pyros_Ros() class. I will shortly move on to using config.py.

The next thing I am trying to figure out is, getting callbacks on top of ros topics. So that users can register callbacks for data in ros topics instead of manually polling frequently. I see some mentions of callbacks on ros topics in rosinterface.py. But not sure if it is exactly what I need. Have you already thought of something that could be useful here?

Also can you shed some light on how baseinterface, rosinterface, pyros_ros, pyros_client, pyzmp are related? Until now I have figured out that pyros_client is child class of pyros_ros() class. Pyros_ros() is taking care of setting up communication with ros and message conversion with the help of baseinterface, rosinterface and pyzmp. Am I going in the right direction?

asmodehn commented 8 years ago

When injecting to topic, the message will be converted automatically from json to ROS, but it has to be in the proper json format otherwise there will be errors... we probably need to figure out a better workflow sometime...

For callback, I think we should work on pyzmp to implement DDP in some kind of generic way. refer to https://github.com/asmodehn/pyros/issues/89 for this.

About pyros structure it s like : 

(processA usercode + pyros_client) -- ZMP --> (processB pyrosROS+rosinterface == rosnode) --> ROS

The rest is an inheritance structure (baseinterface, basenode, etc), but it makes things more complicated than I anticipated, so I am thinking to remove the inheritance hierarchy at some point...