library update/development status and plans

[carlodri]

Hi @codypiersall, first of all thank you for your great work on this library!

I was wondering about your plans for actively developing and updating it, because if not I'm thinking about starting a new project inspired by your work. What are your thoughts? 😊

Thank you again!

[Novakov]

I would like to suggest moving this library under https://github.com/nanomsg organization. That will move maintenance burden from @codypiersall to more people allowing for faster releases, especially when new Python version is released.

@codypiersall has done amazing job by building this library and it is understandable that over time priorities change and it is no longer possible to devote lots of time.

[carlodri]

@Novakov thanks for your comment, yes I agree on your proposal, and I join the kudos to @codypiersall for the huge effort to build this amazing library!

[carlodri]

By the way, there is already an abandoned project there: https://github.com/nanomsg/nnpy

[olivierpelet]

Hi @carlodri,

It seems a good idea to start a new project, based on excellent @codypiersall work. I've started to apply some patches few months ago on my side (here) and I will be happy to contribute to the new project. There are also other users who forked pynng and applied some patches (for instance here), they can be included as well.

Some folks also added MQTT support to NNG (https://github.com/nanomq/NanoNNG), this could be a first feature in the roadmap (or not 😀)

[carlodri]

Thanks for chiming in @olivierpelet!

I'm more keen to start a project from scratch, as opposed to forking/moving this one, WDYT? For example, one thing I would do is avoid building nng and mbedtls from within the setup script but leave them as an external dependency with appropriate version constraints. Obviously we can do it on a fork, but maybe it would be cleaner to start from a fresh repo, possibly with a different project name (nng-python? pynng2?).

[olivierpelet]

Yes starting from scratch seems good to me. It means more work but at the end more maintainable code.

Concerning the code, do you plan to keep a similar architecture (ex: one big Socket class, async using nng AIO, support for trio/curio/asyncio) or do you already have a new architecture in mind ?

Or maybe you want to first focus on packaging / support for newer python version, with minor changes to existing code?

[Novakov]

@carlodri @olivierpelet Would you consider contacting nanomsg organization (I'm just pynng user, not associated with that org) and maintaining repository there?

pynng works great for me an my only issue is lack of wheels for new Python releases.

[carlodri]

@Novakov I think @djc is the person to contact here (sorry for dragging you in @djc!), at least judging from the nanomsg organization owners 🙂

@djc what are your thoughts on this?

[djc]

I think it's more of a question for @gdamore -- I haven't been active in the nanomsg space for a long time now.

[codypiersall]

Sorry for being late to the party here.

@carlodri Absolutely go for it if you're interested in forking/making a new project. I do intend to release new wheels as new Python releases come out, but it is unfortunately just in the "eventually" category of things on my to-do list. If you do fork or do another project from scratch I would prefer you choose a name other than pynng2, just because it could lead to user confusion; but if you really like that name jgo crazy.

FWIW the reason that nng and mbedtls are not external dependencies is that it was a pain point for trying to install in Windows in nanomsg-python, and led to runtime errors. There's all the normal tradespace issues with static linking vs dynamic linking and I chose the static linking route, but dynamic linking certainly has other advantages.

@Novakov Sadly I forget about releasing new wheels until I use a new Python version that doesn't have them available. Not the best maintenance strategy, I'll admit.

[Novakov]

@codypiersall Thanks for replaying :) Would you consider moving this project under nanomsg organization? That will give more people write access and remove some maintenance burden from you?

[codypiersall]

Yeah, absolutely.

[gdamore]

So @codypiersall - the easiest way to handle this is for you to "Transfer Ownership" of that repo to the nanomsg organization. HOWEVER, please be aware -- as I am the owner of that org, that this requires two things:

1. You are trusting my oversight as BDFL for the nanomsg project.
2. You agree to abide by the CoC: https://github.com/nanomsg/nng/blob/master/CODE_OF_CONDUCT.adoc

Once you do that, I'll make sure you retain maintainership rights over the moved repo, adding you to the nanomsg org. You'll have the right to select additional co-maintainers, subject to 1 and 2 above.

This does btw remind me that I think I need to put into place some kind succession plan for the nanomsg org itself. I will give it some thought.

[mlasch]

It would be great to see this project alive again. As mentioned above, we have some patches we'd like to contribute. In addition, it would also be great to have new "official" releases with binary packages. If the project would move to the nanomsg organization how would releasing on pypi work in the future?

As for a complete rewrite, I am a bit skeptical. I would rather incrementally improve/modernize the setup and build process in the current project.

[Novakov]

@mlasch I suggested moving this project to nanomsg in hope that will allow more maintainers to be involved and not relay solely on @codypiersall availablity. I work for company that uses pynng in several projects and spending some time on simple maintenance (like releasing new wheels or bumping nng version) is something that we (as company) can afford.

[codypiersall]

I found some time and motivation to work on this library again, mostly thanks to my wife and tiny, adorable little 6-month-old baby going to bed so early and giving me some unexpectedly wide open evening time. (If you didn't come here for unexpected personal details you are in the wrong place.) I'm going to close this issue.

[carlodri]

@codypiersall thank you, that is great news! And a baby going to sleep early is also wonderful news 🤩!

[ThomasMarwitzQC]

FWIW the reason that nng and mbedtls are not external dependencies is that it was a pain point for trying to install in Windows in nanomsg-python, and led to runtime errors. There's all the normal tradespace issues with static linking vs dynamic linking and I chose the static linking route, but dynamic linking certainly has other advantages.

@codypiersall did these runtime errors occur every time? And was it just enough to run the example listed in the README?

I ask this because on conda-forge we want to try un-vendoring the dependencies for windows build (https://github.com/conda-forge/pynng-feedstock/issues/12 and https://github.com/conda-forge/pynng-feedstock/issues/26). With a minimal reproducible example, I could pay special attention to this RuntimeError possibility.

[codypiersall]

did these runtime errors occur every time?

@ThomasMarwitzQC The old nanomsg-python library was structured such that if it could not find the nanomsg library at run time, it would fail the first time you tried to use it—IIRC nanomsg-python had a build-time option to statically link against the nanomsg library, but would fall back to a ctypes-based way of calling into the library, which would fail at runtime.

Since pynng uses cffi to generate bindings to the nng and mbedtls, our failure mode would be at import time; running import pynng would utlimately trigger an attempt to search for the shared libraries; if those couldn't be found, an exception would be raised.

It is not really technically difficult to use dynamic linking instead of static linking; the problem is that a big design goal of pynng was that it would be able to just be plain ol' pip install-able; the most straightforward way at the time was to statically link. I was under the impression that shared libraries could not be vendored along with wheels, but I am not totally sure that that is true.

If something needs to change in pynng to make packaging stuff better for Conda, I am open to PRs, as long as pip install pynng continues to work for the non-conda case.

[ThomasMarwitzQC]

@codypiersall Ok, many thanks for your thorough elaboration.

In conda-forge, I currently solved the un-vendoring for non-win systems by applying a patch. If the mbedtls library becomes available for windows, I will try to un-vendor for win as well. If that works, I might create a PR to get rid of this patch in the long run :)