search

zeromq / malamute

The ZeroMQ Enterprise Messaging Broker
Mozilla Public License 2.0
323 stars 77 forks source link

Problem: I want to use malamute in python #115

Open asokoloski opened 8 years ago

asokoloski commented 8 years ago

So I'm planning on making api models and generating bindings with zproject. Any reason that I shouldn't, or things to watch out for?

hintjens commented 8 years ago

It should already be working, in bindings/python. On 21 Jan 2016 15:02, "Aaron Sokoloski" notifications@github.com wrote:

So I'm planning on making api models and generating bindings with zproject. Any reason that I shouldn't, or things to watch out for?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/malamute/issues/115.

asokoloski commented 8 years ago

I'm confused -- malamute/bindings/python is empty. Are we talking about different things?

asokoloski commented 8 years ago

I started working on it a little bit but then I remembered to bring it up here first. If someone has already generated them then I'm glad I didn't continue.

hintjens commented 8 years ago

Ah, we need to define API models for the classes. We could generate them via zproto... On 21 Jan 2016 15:31, "Aaron Sokoloski" notifications@github.com wrote:

I'm confused -- malamute/bindings/python is empty. Are we talking about different things?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/malamute/issues/115#issuecomment-173585476.

asokoloski commented 8 years ago

Someone made a script in zproject called mkapi.py about a month ago. I've been looking at it and it seems to be pretty close to what we need. It still requires some manual tweaks afterward, as it's a lot easier to generate C code from a model than to generate a model from C code. Shall I get started?

asokoloski commented 8 years ago

Ok, I'm a bit new to C4, but I guess this counts as this problem being acknowledged. I'll go ahead. Maybe I've been erring on the side of being too cautious, and I shouldn't take up too much of your time discussing this.

hintjens commented 8 years ago

You don't need anyone to acknowledge a problem; simply stating it in your patch is enough.

I'd try mkapi, on the client API and see what it gives. The right solution is to generate the APIs via zproto, which has all the necessary info. I can do that later. You can already get started.

asokoloski commented 8 years ago

Ah, ok. I get it. Maybe generating python bindings to malamute isn't even necessary, if I can use zproto to generate the codec with python. That's if I understand how this works. Anyway, thanks for the help -- I'll look into that approach.

I'll leave the issue open until it's ready.

hintjens commented 8 years ago

zproto could also generate a Python codec, yes. Might be nicer than wrapping the mlm_client API. Either will work. The nicest would be a pure Python ZMTP stack... :)

On Thu, Jan 21, 2016 at 4:55 PM, Aaron Sokoloski notifications@github.com wrote:

Ah, ok. I get it. Maybe generating python bindings to malamute isn't even necessary, if I can use zproto to generate the codec with python. That's if I understand how this works. Anyway, thanks for the help -- I'll look into that approach.

I'll leave the issue open until it's ready.

— Reply to this email directly or view it on GitHub https://github.com/zeromq/malamute/issues/115#issuecomment-173615115.

asokoloski commented 8 years ago

Hmm, I've looked into this a bit. The zproto-generated python codec is pretty bare-bones, so I decided to look into what you described about generating APIs with zproto in order to bind to the client class with the full state machine.

It seems like there's a bit of a clash / redundancy. Zproto generates client / server header files from the state machine model. But Zproject also generates the @interface section of the same header files, if it's given a CLASS API model in order to generate bindings.

Zproto could ONLY generate the CLASS API model and let zproject turn that into a header, but is that the right way to go? At the very least, it makes zproto depend on zproject, which is probably bad unless we gain a lot from it. On the other hand, having two separate implementations of header-generating-gsl is bad not just because it's redundant, but also because it's confusing and you'd get the headers toggling back and forth slightly when generated by one or the other method, which are bound to be inconsistent.

Does this make sense at all? Or am I missing something important?

hintjens commented 8 years ago

zproto in any case depends on zproject as that's a basic tool for all our C projects. So the correct solution IMO is for zproto to generate an API file instead of the include file it generates today. You could override this with a switch (gsl -zproject:no myproto.xml).

On Fri, Jan 22, 2016 at 3:15 PM, Aaron Sokoloski notifications@github.com wrote:

Hmm, I've looked into this a bit. The zproto-generated python codec is pretty bare-bones, so I decided to look into what you described about generating APIs with zproto in order to bind to the client class with the full state machine.

It seems like there's a bit of a clash / redundancy. Zproto generates client / server header files from the state machine model. But Zproject also generates the @interface https://github.com/interface section of the same header files, if it's given a CLASS API model in order to generate bindings.

Zproto could JUST generate the CLASS API model and let zproject turn that into a header, but is that the right way to go? At the very least, it makes zproto depend on zproject, which is probably bad unless we gain a lot from it. On the other hand, having two separate implementations of header-generating-gsl is bad not just because it's redundant, but also because it's confusing and you'd get the headers toggling back and forth slightly when generated by one or the other method, which are bound to be inconsistent.

Does this make sense at all? Or am I missing something important?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/malamute/issues/115#issuecomment-173931571.

asokoloski commented 8 years ago

Ok, cool. You mean the switch would just disable the api generation, not that it would revert to the existing behaviour, right?

hintjens commented 8 years ago

I mean people who absolutely refuse to use zproject could use the switch to work the old way. For instance if they are using zproto in an existing, non zproject case.

On Fri, Jan 22, 2016 at 4:20 PM, Aaron Sokoloski notifications@github.com wrote:

Ok, cool. You mean the switch would just disable the api generation, not that it would revert to the existing behaviour, right?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/malamute/issues/115#issuecomment-173948155.

asokoloski commented 8 years ago

Ah, ok. From what you said above I thought you meant that you'd always need zproject, but if that's not the case then sure, let's not break existing code.

I'm not sure I'll have enough time to spend on this to do the full conversion myself, especially given my relative inexperience with GSL. As much as I'd like to do it the right way, I may just focus on getting the python bindings working using the mkapi.py approach as you originally suggested.

Thanks for taking the time to explain everything.

hintjens commented 8 years ago

Using mkapi.py was your idea :-)

I'd honestly take the risk to generate the API files; it will be educational and give you the best results. Also, if you get started and get stuck just pass it to me and I'll finish it for you.

On Fri, Jan 22, 2016 at 4:30 PM, Aaron Sokoloski notifications@github.com wrote:

Ah, ok. From what you said above I thought you meant that you'd always need zproject, but if that's not the case then sure, let's not break existing code.

I'm not sure I'll have enough time to spend on this to do the full conversion myself, especially given my relative inexperience with GSL. As much as I'd like to do it the right way, I may just focus on getting the python bindings working using the mkapi.py approach as you originally suggested.

Thanks for taking the time to explain everything.

— Reply to this email directly or view it on GitHub https://github.com/zeromq/malamute/issues/115#issuecomment-173951041.

asokoloski commented 8 years ago

Sorry, just I meant your suggestion to leave the hard part to you :)

Ok. I think it still makes sense for me to start at the end, generating python bindings from a hand-tuned API model and writing a higher-level wrapper for them, as using python's ctypes will also be educational :) and that will make malamute usable in python faster. Then I'll take another look at making zproto generate the API -- knowing how it is supposed to work should help a good bit.

How do these sorts of bindings generally get distributed? Does the hand-written python wrapper code live in a separate project, with the low-level binding copied over from zeromq/malamute?

asokoloski commented 8 years ago

You know what, no worries -- let's deal with that when I get to that point :) Still got a ways to go.

hintjens commented 8 years ago

Depends how complex the hand-written part is. I'd start it in the Malamute project and if it grows to become worth splitting off, to do so.

On Fri, Jan 22, 2016 at 5:03 PM, Aaron Sokoloski notifications@github.com wrote:

Sorry, just I meant your suggestion to leave the hard part to you :)

Ok. I think it still makes sense for me to start at the end, generating python bindings from a hand-tuned API model and writing a higher-level wrapper for them, as using python's ctypes will also be educational :) and that will make malamute usable in python faster. Then I'll take another look at making zproto generate the API -- knowing how it is supposed to work should help a good bit.

How do these sorts of bindings generally get distributed? Does the hand-written python wrapper code live in a separate project, with the low-level binding copied over from zeromq/malamute?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/malamute/issues/115#issuecomment-173962155.