Finalize tests with RSA

[tcalmant]

The rsa-integration branch contains the implementation of Remote Service Admin, contributed by @scottslewis. It is currently been tested with the XML-RPC provider and with local EDEF file.

The following test still have to be written:

• [x] Tests for the shell commands
• [x] Tests for the etcd discovery provider
• [x] Tests for the Py4J transport provider. This one will only work on Python 3.5+ for now as it relies on concurrent.futures.

Those tests and the documentation of the RSA package are mandatory before the release of iPOPO 0.8.0. Any help is welcome :)

[scottslewis]

Hi Thomas. I'll work on the etcd tests.

[scottslewis]

Hi Thomas. How would I run the tests locally? Are there some instructions for doing that? Thanksinadvance. Also: Is there some way to run specific tests only? (e.g. rsa tests, or subset)? I just tried Pydev Python Unit test and it seems to work. But of course I would like to do it how the build/you do it to verify.

[tcalmant]

Hi Scott,

To run tests, you can use nose (which I use) or py.test. You can install one (or both) of them with pip.

To run specific tests, just indicate the file containing them, e.g. nosetest tests/rsa/test_shell.py. I don't know about PyDev integration in the IDE, but it should be possible to run each test method individually. I don't have the command line syntax in mind...

In the case of etcd, it will be easier to start with a remote server. The use of a local sercer can be useful to avoid useless load to the server when testing with Travis, but it can be done later.

[scottslewis]

Hi Thomas,

I've been able to run the tests fine with pydev's junit test runner.

I've created publisher tests for etcd discovery, and started etcd listener tests. Do you have a way that you run/launch a separate python process (running another framework instance) for the multi-process tests? To test the etcd listener, there has to be another process out there publishing services and I don't know how this is most easily done across-os with python unit testing.

Of course one way is that I just start the publisher (I've written a very small main that just publishes an xmlrpc remote service to etcd) and let it stay running while the listener unit tests are running, but I don't know how this will interact with the travis build/test environment.

[tcalmant]

Hi Scott, That's great!

You can find samples of multi-processes testing in tests/remote/test_discovery.py and test_transports_http.py. They are based on the multiprocessing module, and use a Queue to communicate.

The idea is to declare a method to that will be executed in another process. To avoid problems, this child process must be started before the local framework (as it relies on fork(), a framework state could be shared). Then, you can control barriers in the remote process with a Queue, in both ways (to let the remote process continue and to get its results).

The code coverage plugin is able to work nearly seamlessly in these conditions (you might need to install coverage_enable_subprocess), and it is easier to use than the subprocess module.

[scottslewis]

Hi Thomas,

With this commit:

https://github.com/tcalmant/ipopo/commit/f40c29518c426b008317c2db46724048481f000e

I've added test_etcd_discovery.py and a support class (in endpoint_event_listener.py). The test_etcd_discovery tests both advertise and discover.

What tests would you prefer I work on next?

[tcalmant]

Hi Scott,

The test of etcd works fine on Python 3 :) I've updated the requirements.txt to have it working on Travis. There is an error on Python 2.7, but don't worry, I'll look into it.

The next big thing would be the Py4J transport, as I'm not fluent with it... Meanwhile, I'll try to work on the documentation.

[scottslewis]

RE: Py4j. The hardest thing about doing unit testing for the Py4j distribution provider will be the java/OSGi side. I've been doing testing of it, of course...by running a java/OSGi sample app in my repo here: https://github.com/ECF/Py4j-RemoteServicesProvider/examples and then running the python side (this ordering is ordained by Py4j itself, which treats the java side as the initial listener). For Python unit testing I suppose I can launch a java/OSGi process (Py4j does it for their unit testing) but that just launches a java vm...not OSGi framework. Any libs in python unit stuff to help with this? I would expect that WrappedProcess would be for external python processes only.

For the initial test, I might just copy needed java bundles from github to travis, launch example from unit test setup (somehow) and run ipopo sample...and check for basics of importing Py4j java service. I'm sort of torn as to how much effort to dedicate to running and documenting as example/sample/tutorial vs running in unit test environment.

[tcalmant]

OK I see the problem. In that case, it would be better to have a documentation on the iPOPO side; maybe we'll see how to run the tests with a set of bash scripts.

[scottslewis]

Yeah...I think tests via bash scripts would be better for the moment. What I propose is that we start from this page: https://wiki.eclipse.org/OSGi_Services_between_Python_and_Java ...which currently describes how to run/start Java side...and add to the samples documentation on the iPopo side...with links back and forth, of course. That way it can be used as both a test and example+tutorial for folks that might want to do their own Python.Java remote services. Also it will provide some incentive to me to simplify the deploy/installation on both Java and Python.

There is already code in samples and samples.rsa that I've been using for testing. That's the code on the wiki.eclipse.org page pointed to above.

I suggest that this be another sample/example rather than the only one on IPopo side. I'm willing to write the Python-side docs...or with you as you prefer.

[scottslewis]

Hi Thomas. I made some changes and simplifications to the Java-side code of the Py4j remote services provider. The summary is that I simplified things so that all that's needed to get a remote service exported on the java side, and a consumer also available on the java-side, is to start an osgi framework with the hello example bundles: https://wiki.eclipse.org/OSGi_Services_between_Python_and_Java

This will make it easier to use the 'hello' example service with the ipopo samples in unit testing (as well as sample). We will still have to launch a java process with correct set of bundles, but now that's all.

I suggest that we complete the samples documentation, and release 0.8.0 before the unit test using that sample is integrated to the automated tests...as that may take some doing (e.g. java launching, putting binaries in repo or getting them some other way, etc). I will help to add the junit test...but I would like to use the Photon release and python rsa, and py4j provider to get some attention in OSGi community to ipopo 0.8.0.

[tcalmant]

Hi Scott, OK for skipping unit tests to speed up the 0.8.0 release. So, the last step would be a small documentation on how to use RSA with iPOPO, or at least a link to the existing documentation of the wiki of the ECF project ?

I'm also waiting for @pisua to check if #99 is an iPOPO issue or not.

[scottslewis]

Hi Thomas

So, the last step would be a small documentation on how to use RSA with iPOPO, or at least a link to >the existing documentation of the wiki of the ECF project ?

Yeah, I was thinking of completing the tutorial here:

https://wiki.eclipse.org/Tutorial:_Python_for_OSGi_Services

with a link to a ipopo page(s) that describe how to run the samples.run_rsa_py4java.sh ipopo application (which sets up both a IHello consumer and optionally a python-based impl. And I could put at the beginning of the ipopo documentation a link to the Tutorial above...since something on the java side has to be started first (for the py4j sample).

If you haven't already/done/started such a sample I could do it over the next few days. If you've got a basic layout/structure for another (e.g. xmlrpc) sample, I'll follow the same general approach (modulo the links to the setup/run of Java side).

[scottslewis]

HI Thomas,

Do you have sample documentation for rsa? e.g. using xmlrpc distribution? I don't see it obviously at:

https://github.com/tcalmant/ipopo/tree/master/docs

but perhaps I'm looking in the wrong place.

If so, please point me at it and I'll create the sample documentation for the starting/running of the py4j sample...in combination (links) with the wiki page I've got above.

[tcalmant]

Hi Scott, I don't have a page ready for rsa, but you can use docs/refcards/remote.rst as a template.

I'll be able to work a bit on documentation on Thursday.

Le mar. 17 juil. 2018 à 18:42, Scott Lewis notifications@github.com a écrit :

HI Thomas,

Do you have sample documentation for rsa? e.g. using xmlrpc distribution? I don't see it obviously at:

https://github.com/tcalmant/ipopo/tree/master/docs

but perhaps I'm looking in the wrong place.

If so, please point me at it and I'll create the sample documentation for the starting/running of the py4j sample...in combination (links) with the wiki page I've got above.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/tcalmant/ipopo/issues/98#issuecomment-405648701, or mute the thread https://github.com/notifications/unsubscribe-auth/ABeB5nC2E1ok3qyqqOrRcD89CJErec8eks5uHhQLgaJpZM4U5rrZ .

[scottslewis]

Hi Thomas,

ok, I'll work on the python side of the py4j example using remote.rst as template.

[scottslewis]

Hi Thomas. I looked at rsa.rst you recently added. A couple of comments:

1) I think it's over stating it to say that the RSA implementation is highly tied to ECF/Java. That seems to imply that ECF is needed to use the Python RSA implementation...or even needs to be understood separately to use rsa in ipopo....which is not right at all. Everything can run with python only w/o ECF or OSGi/Java, and all of the main architectural concepts (RemoteServiceAdmin service, topology manager role, distribution and discovery providers) are from RSA rather than ECF...e.g. fig 122.1

https://osgi.org/specification/osgi.cmpn/7.0.0/service.remoteserviceadmin.html#d0e90795

It's true that he EndpointDescription meta-data has ECF meta-data (osgi intents, versions, ecf.* properties) but this is so that the Python and ECF/Java impls can immediately interoperate. For remote services it seems to me that's a nice advantage.

I also expect/hope that additional distribution and discovery providers be created...for python-only exporters and importers...and it's not impossible (or even difficult, I would say) to create dist providers that interoperate with other RSA impls.

Just to summarize: I'm fine with having links to ECF docs as you do, but I don't think it's necessary to imply that the ipopo rsa is dependent upon or even requires ECF.

2) You say that:

Python for OSGi Services, describing a non-iPOPO way to work with ECF

I don't really understand how this doc is a non-IPOPO way to work with ECF (?) I think it would be more accurate to say something like:

Python for OSGi Services: A tutorial for remote services between Java/OSGi and Python/ipopo

The existing wiki doc has a Java service exported to a Python consumer, but the reverse is already in the sample code...i.e. a Python service exported to Java...and I'm planning to add that to the tutorial as well.

So to coordinate further on docs: I was going to work further on the Python/ipopo side of the Python for OSGi Services tutorial. Would you prefer that I do something else? or continue work on that? I'm hopeful that I will be able to do that work Wed am pacific time.

[tcalmant]

Hi Scott,

1. I agree; I was more thinking about the tight relation in terms of concepts and architecture. The way

2. The link to the tutorial ( https://wiki.eclipse.org/Tutorial:_Python_for_OSGi_Services ) was to be given as an indication that you can work with ECF in Python without iPOPO (directly with osgiservicebridge). This can be useful in some projects, where iPOPO wouldn't work on all parts of an infrastructure, but could communicate through that link. I should have pointed directly to https://wiki.eclipse.org/OSGi_Services_between_Python_and_Java which matches the samples we have in the iPOPO repository.

I let you the work on the documentation, I can't work on iPOPO at work for the moment I still have some issues to fix on my spare time.

Thomas

Le mer. 25 juil. 2018 à 02:41, Scott Lewis notifications@github.com a écrit :

Hi Thomas. I looked at rsa.rst you recently added. A couple of comments:

1. I think it's over stating it to say that the RSA implementation is highly tied to ECF/Java. That seems to imply that ECF is needed to use the Python RSA implementation...or even needs to be understood separately to use rsa in ipopo....which is not right at all. Everything can run with python only w/o ECF or OSGi/Java, and all of the main architectural concepts (RemoteServiceAdmin service, topology manager role, distribution and discovery providers) are from RSA rather than ECF...e.g. fig 122.1

https://osgi.org/specification/osgi.cmpn/7.0.0/service.remoteserviceadmin.html#d0e90795

It's true that he EndpointDescription meta-data has ECF meta-data (osgi intents, versions, ecf.* properties) but this is so that the Python and ECF/Java impls can immediately interoperate. For remote services it seems to me that's a nice advantage.

I also expect/hope that additional distribution and discovery providers be created...for python-only exporters and importers...and it's not impossible (or even difficult, I would say) to create dist providers that interoperate with other RSA impls.

Just to summarize: I'm fine with having links to ECF docs as you do, but I don't think it's necessary to imply that the ipopo rsa is dependent upon or even requires ECF.

1. You say that:

Python for OSGi Services, describing a non-iPOPO way to work with ECF

I don't really understand how this doc is a non-IPOPO way to work with ECF (?) I think it would be more accurate to say something like:

Python for OSGi Services: A tutorial for remote services between Java/OSGi and Python/ipopo

The existing wiki doc has a Java service exported to a Python consumer, but the reverse is already in the sample code...i.e. a Python service exported to Java...and I'm planning to add that to the tutorial as well.

So to coordinate further on docs: I was going to work further on the Python/ipopo side of the Python for OSGi Services tutorial. Would you prefer that I do something else? or continue work on that? I'm hopeful that I will be able to do that work Wed am pacific time.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/tcalmant/ipopo/issues/98#issuecomment-407596016, or mute the thread https://github.com/notifications/unsubscribe-auth/ABeB5kJPwlBDNn6nfgmHj5Eael4bquucks5uJ76hgaJpZM4U5rrZ .

[scottslewis]

Hi Thomas.

I've modified/simplified this page significantly:

https://wiki.eclipse.org/OSGi_R7_Remote_Services_between_Python_and_Java

It now also links to this new iPOPO tutorial page:

https://github.com/tcalmant/ipopo/blob/rsa-integration/docs/tutorials/rsa_pythonjava.rst

This page describes how to launch/run on Python sample side...and what happens wrt Java<->Python service interaction

Next up: Tutorial on using Python RSA with xmlrpc provider (Python impl and consumer), some concepts documentation about iPOPO RSA...e.g. roles and associated APIs, and 1-2 short tutorials on rsa console commands.

[scottslewis]

Next up: Tutorial on using Python RSA with xmlrpc provider (Python impl and consumer), some concepts documentation about iPOPO RSA...e.g. roles and associated APIs, and 1-2 short tutorials on rsa console commands.

I've decided to work first on a concepts document about RSA in general...java and python. I don't think it's well-articulated in the OSGi specs what the main roles are (e.g. RSA service, topology manger, discovery and distribution providers, etc) and how they fit together. The spec...IMHO...is more intended for implementers (i.e. me/ECF), and what's needed is kind of an RSA user's guide...so I've started in on this:

https://wiki.eclipse.org/OSGi_Remote_Service_Admin_(RSA)_User_Guide

I would like to have both java and python examples in this user guide, as well as point to both java and python tutorials/samples.

After I get this doc at least started/outlined, I'll do the xmrpc provider tutorial/sample.

I'm going to try to complete both this coming week, as the following week I have a new gig that I have to take on.

I'm hopeful that we can get things to a releasable (0.8.0) state shortly after that so that I can promote it via blog posts, etc. Let me know what you think and what plans you have wrt more docs and release schedule.

Scott

[tcalmant]

Hi Scott,

I've got some time this weekend, so I've been able to fix the CI error in the etcd discovery tests for Python 2.7: they were caused by a name conflict: import etcd was importing the RSA module instead of the python-etcd library. The fix had to be rename the file discovery_etcd, for some reason the use the absolute_import future wasn't enough.

I'll try the RSA tutorial and check if everything works as intended, then I'll try to add more description in the reference card. I think that once we'll be OK for a release once all providers will be described, with their dependencies, configuration, limitations and a small sample.

Also, I have to add a unit test for the thread pool, it seems that a small part of it isn't tested.

I should be able to work on iPOPO between wednesday and friday next week. I'll have more spare time the last weeks of august (my work tasks will be of lower priority)

Thomas

[tcalmant]

Hi Scott,

I've just added a test for the Py4J provider, based on the code from the tutorial. It uses Karaf to run the Java part: it seems to work fine. I just have troubles with Travis-CI and Python 3.4 (seems like a pip compatibility error with pytest).

I think we're ready for the version bump to 0.8.0, right after the addition of the XML-RPC provider tutorial.

Thomas

[scottslewis]

Hi Thomas,

OK, sounds good. I'll try to work on the xml-rpc provider tutorial tomorrow (Friday) and this weekend. Perhaps we can release 0.8.0 sometime mid-next week.

Scott

On Wed, Aug 8, 2018 at 8:21 AM Thomas Calmant notifications@github.com wrote:

Hi Scott,

I've just added a test for the Py4J provider, based on the code from the tutorial. It uses Karaf to run the Java part: it seems to work fine. I just have troubles with Travis-CI and Python 3.4 (seems like a pip compatibility error with pytest).

I think we're ready for the version bump to 0.8.0, right after the addition of the XML-RPC provider tutorial.

Thomas

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tcalmant/ipopo/issues/98#issuecomment-411444870, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJw9Gu45vAQchGYt3Rhm5NLQ6fY5REFks5uOwH1gaJpZM4U5rrZ .

[cah-sachin]

Hey Guys, We are using pelix framework in our project. We have need now for device to device communication like Edge or discovery server and discovery client.

I was thinking to use Remote services of pelix. Current development above looks like interesting to me. Let me know if I am on correct path.

[scottslewis]

Hi Cah-Sachin

I would say you are probably on the right path. RSA adds a number of helpful things for remote services...e.g. a remote services mgmt agent, console commands, finer-grained control of export/import, an extensible topology manager.

The area where additional work is needed is in supporting other distribution and discovery providers. With RSA there are distribution providers for xmlrpc and py4j (interaction between java and python runtimes), and discovery providers for etcd (kubernetes), and py4j.

If you know you are going to need a specific distribution provider...e.g. based upon jsonrpc, or mqtt+json, or some other transport, or are intending to use/integrate with some specific discovery provider (e.g. zookeeper) then my and Thomas' efforts on additional distribution and discovery providers could be focused in those areas.

[scottslewis]

Hi Thomas,

On the rsa-integration branch, I've created a first-cut of an iPOPO tutorial that uses the helloimpl sample and the xmlrpc distribution provider. It's available here:

https://github.com/tcalmant/ipopo/blob/rsa-integration/docs/tutorials/rsa_xmrpc.rst

[tcalmant]

@cah-sachin Hi Cah-Sachin, iPOPO has 2 implementations of remote services:

• pelix.remote, the historic version: it intends to provides a quick & easy way to setup remote services in dynamic environments. Its main purpose is to let frameworks on small devices discover themselves in new environments without configuration (with the Multicast discovery) or to talk to each from remote places (using XMPP).
• pelix.rsa is the newcomer: it implements RemoteService Admin, which is defined in the OSGi specification. You can consider that it is the package to use now. It is new (so it might have some bugs) and will be the one we'll focus our effort on. Like Scott said, it doesn't have a lot of providers yet, but they'll come with future versions of iPOPO, according to the requests of iPOPO users.

Note that RSA will be released with iPOPO 0.8.0 (coming this week hopefully), and that, for legacy reasons, both RSA & legacy implementations will coexist in iPOPO 1.0.

@scottslewis That's great! I'll take a deeper look in the evening (UTC+2), but it seems good at first sight

[cah-sachin]

Thanks @scottslewis and @tcalmant . If you have any example of how to do device to device communication using any of above that will be very good example for me.

[scottslewis]

@cah-sachin On the rsa-integration branch...feel free to review/comment on https://github.com/tcalmant/ipopo/blob/rsa-integration/docs/tutorials/rsa_xmrpc.rst

It's in draft form as it was just created, but any comments would be much appreciated. The sample code in source...also on rsa-integration.

[scottslewis]

Hi Thomas,

I've made additions and edits to the xmlrpc tutorial, along with getting the name right:

https://github.com/tcalmant/ipopo/blob/rsa-integration/docs/tutorials/rsa_xmlrpc.rst

I think this is an ok first cut tutorial/sample using xmlrpc for Python<->Python remote services. I'll add using etcd discovery to export/import remote services either tonight (Mon) or Tues.

[tcalmant]

Hi Scott, I think we're good for the release, let me know if you find some things to review. The rsa-integration branch already has the 0.8.0 version set in all headers.

[scottslewis]

I just made a small change to the java/python tutorial: for consistency's sake I changed the tutorial title to: RSA Remote Services between Python and Java. I think 'RSA Remote Services' will be a more explainable way to distinguish in documentation from the existing 'Remote Services'. And hopefully RSA Remote Services will be clear enough for consumers. wdyt?

Other than that, I'm a +1 for 0.8.0 release.

[tcalmant]

Good idea. Also, using the RSA terminology will be easier to relate to the pelix.rsa package. I'll do a quick review of the code & doc this evening (CEST) and I'll do the release if everything is OK.

Sadly, the Travis-CI runs are constantly failing, sometimes due to a timeout in the Py4J test, sometimes even during the container initialization phase.

[tcalmant]

I found why the Travis-CI builds were failing (see #100). It seems to be an error in osgiservicebridge: a thread handled by the ThreadPoolExecutor of the Py4J provider might not terminate as intended and makes the shutdown of the thread pool wait indefinitely.

The fix on iPOPO side is to not join the threads in the pool when shutting it down. An issue will be opened on the osgiservicebridge project when I'll have more details.

I'll merge the rsa-integration branch with master and prepare a release (which should occur tonight, CEST). Therefore, this issue can now be closed.