Improvement

[acutaia]

Dear Thomas, it's been 9 months that I'm working on your framework for a university project. I'm using the asyncio in order to develop asynchronous code. I think that substituting the contextmanager of iPopo with an async contextmanager could improve the efficiency of iPopo and, making iPopo asynchronous will also improve the scalability. What do you think about it? If you think it is a good idea I can help you developing it.

[tcalmant]

Hi Angelo, So the idea would be to have something like a use_ipopo_async inheriting the contextlib.asynccontextmanager from Python 3.7, am I right ? (I'm not used to asyncio yet, I just used it a couple of times on snippets)

I was wondering for some time if I should do a specific version of iPOPO supporting only Python 3.6+ (with enums, typing, ...) but left it aside as it wouldn't bring that much benefits. In fact, I would have renamed the current master branch to v1, with a 1.0 release of the iPOPO package on PyPI. The new master would be for 2.x releases. That being said, the addition of async capabilities would make the 3.6+ (or 3.7+) branch more interesting to work on.

Note that it must be not change the behavior described in OSGi: bundle start/stop and components in/validation must remain blocking to ensure we don't have side effects.

[acutaia]

I think It's better using python >3.7 because the API of asyncio of python 3.7 are more performing than python 3.6 and also easier to use. The behavior described in the OSGi will not change, what I have on my mind is modifying the framework low level making it asynchronous with asyncio leaving unchanged the high level API, for example the EventAdmin will have the asynchronous method post() and the synchronous method send() which under the hood will be asynchronous and handled by asyncio. What do you think?

[tcalmant]

I think that would be a good idea! Some parts of the event handling and module loading of Pelix/iPOPO good benefit from that approach as well.

[acutaia]

Perfect. How can we organize to develop it? If you're interested I've studied asyncio using this book

[acutaia]

I think it could be changed every part of the code that has blocking IO with an asynchronous one, e.g. replacing the http request with aiohttp. I want to try to reduce the number of Threads of iPOPO making the asyncio EventLoop (the one running in the main Thread) handle the whole framework, the IO task and short time running task. The long running and the blocking code will be scheduled by the main event loop on other new EventLoop running in workers asynchronous thread that will close by themselves when they finish their job. I forgot to tell you that asyncio in python 3.7+ has also new API that are the new standard of asyncio so we'll have to use the suggested asyncio API in python 3.7+. I hope I've explain myself well. Let me know. PS: here there are all the code snippets of the book that I've posted before but their explanation is only on the book. https://github.com/Apress/asyncio-recipes

[tcalmant]

So the plan, so far, would be:

1. Create a v1 branch from the current state of master and release iPOPO 1.0.0 (I have to talk to some users before that)
2. Change the README, setup.py and all headers to bump the master branch to version 2.0.0. The name of package might change to ipopo2 to avoid dependency issues.
3. At this point, you can fork the project and start working on the asyncio conversion. I can also make you a contributor of this project to avoid the fork. We should then use issues on this project to synchronize the parts we update.

Some things to not forget:

• The code and tests will need a big review to remove the parts of if managing the Python 2+3 usage
• We can now upgrade the dependencies of ome utility bundles (e.g Paho for the MQTT service)

Note for readers of this issue: This doesn't mean the v1 will stall, just that their will now be 2 versions of iPOPO until the real death of Python 2 (around June 2024 for Red Hat). Also, Remote Services and Remote Services Admin can be used to make v1 and v2 instances communicate.

[acutaia]

Perfect. I will gratefully take this task as a contributor. I will wait for further feedback from you in order to start working on this project. How often do you think we should synchronize ? If anything I'm available for help.

[tcalmant]

I'll try to do steps 1 and 2 of the previous comment tomorrow (Monday), and I'll add you as contributor.

For the synchronization, I was thinking about something like creating a single issue for the conversion, and adding a comment with the name of the modules and package we intend to work on the next days.

[acutaia]

Perfect , I agree . Let's stay in touch

[scottslewis]

A naive question: What will be the expected user and developer benefits of using asyncio in ipopo?

[acutaia]

A naive question: What will be the expected user and developer benefits of using asyncio in ipopo?

The benefit will be a boost in scalability and performance because asyncio was written to avoid wasting CPU time on I/O bound tasks or waiting for Network connection. In fact, coroutines and event loops allow developers to execute code only when it's not waiting for I/O or Network connection and to yield control back for other tasks. In phew words the number of threads of iPOPO will be reduced to only a single main thread look at previous comment . If you need further explanation just tell me

[tcalmant]

A naive question: What will be the expected user and developer benefits of using asyncio in ipopo?

Hi Scott, Like Angelo said, the main advantage of asyncio is to be more efficient handling I/O bound tasks, but also on some waiting APIs (locks, ...).

I would also point out that to gain an overall efficiency using asyncio, the whole framework must be ready to work in asynchronous mode (while keeping the critical sections synchronous, e.g. bundle activation, ...).

Finally, another advantage is the upgrade of the minimal version of Python supported by iPOPO v2 to Python 3.7, which will allow the use of new APIs, to remove compatibility tricks and also to ensure that the module/bundle loading process is up to date (the current one is deprecated in the official API).

[tcalmant]

I've added the v1 (new default branch) and v2 branches to the repository. @Angeloxx92 is now a contributor of the project.

[acutaia]

I've added the v1 (new default branch) and v2 branches to the repository. @Angeloxx92 is now a contributor of the project.

Perfect. Shall we use this issue #108 to synchronize or open a new one?However I'll post Tuesday (tomorrow) the bundle's name on which I'll start to work because I want to organize a good action plan

[tcalmant]

A new issue would be better I think.

Le lun. 6 janv. 2020 à 14:48, Angelo Cutaia notifications@github.com a écrit :

I've added the v1 (new default branch) and v2 branches to the repository. @Angeloxx92 https://github.com/Angeloxx92 is now a contributor of the project.

Perfect. Shall we use this issue #108 https://github.com/tcalmant/ipopo/issues/108 to synchronize or open a new one?However I'll post Tuesday (tomorrow) the bundle's name on which I'll start to work because I want to organize a good action plan

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tcalmant/ipopo/issues/108?email_source=notifications&email_token=AALYDZVWMMTLT7D26CFMBSDQ4MZEHA5CNFSM4KCNQM7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFPRXA#issuecomment-571144412, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALYDZVOQGI2NN6GYQGZONTQ4MZEHANCNFSM4KCNQM7A .

[acutaia]

I've just created the new issue