Add HTTPS support

[ghost]

It appears from the documentation that there is no support for HTTPS. Is this so?

[dsuch]

Hey @pruggera,

you're right that 1.1 has no HTTPS. This is something I was already discussing with several people and I understand it's needed and it will be done - I want it as well.

(As a side note - yep, if something is not documented it means it's not available, i.e. there aren't any major features that are ready to be used but are undocumented, there may be some smaller or trivial ones here and there but in that situation it's usually a matter of improving the docs)

Proper HTTPS would need at least:

• Certificate-based security akin to other already existing security definitions - so you could use assign a security definition to an HTTP channel, and one would be able to state that such and such combination of fields must exist or that a certificate must be signed off by such and such CA. This would mean new GUI forms, CLI and API.
• The ability to assign/update certificates to servers (may require restarting the whole server, full zato stop && zato start cycle)
• Some sort of warnings displayed when a user logs in and it's less 180/120/60/30/15/etc. days to expiration dates

Nice to have in the first version but may wait a bit:

• The ability to assign/update certificates to servers with restarting each gunicorn worker serially one by one (this would be done automatically, users wouldn't even know anything, but this would make sure the server will always have a worker to process the messages)

I don't remember what the current state of SSL in gevent/gunicorn is - we are currently at 0.13.8/0.16.1 respectively.

https://github.com/zatosource/zato/blob/master/code/versions.cfg

But if adding SSL needs upgrading to newer versions we'll naturally do it.

What do you think about it? Would that suffice for your needs? Can you think of other things you'd like to see?

Also, if you need to have it absolutely right now and you can put nginx in front of Zato I can work with you on how to validate incoming requests basing on request's HTTP headers - this needs a bit of coding (like 10-40 lines of code) and keeping config in Redis but is easily doable.

Thanks!

[ghost]

We're building an HR event service using BizTalk and I was just interested in seeing what it would take to do the same thing in Zato. We've been working on this for a month. If I could complete this service using Zato in a few days I would campaign to bury BizTalk. But this project needs HTTPS and REST. So I'll wait until Zato has those feature and then give it a shot.

On Tue, Jul 16, 2013 at 3:20 PM, Dariusz Suchojad notifications@github.comwrote:

Hey @pruggera https://github.com/pruggera,

you're right that 1.1 has no HTTPS. This is something I was already discussing with several people and I understand it's needed and it will be done - I want it as well.

(As a side note - yep, if something is not documented it means it's not available, i.e. there aren't any major features that are ready to be used but are undocumented, there may be some smaller or trivial ones here and there but in that situation it's usually a matter of improving the docs)

Proper HTTPS would need at least:

-

Certificate-based security akin to other already existing security definitions - so you could use assign a security definition to an HTTP channel, and one would be able to state that such and such combination of fields must exist or that a certificate must be signed off by such and such CA. This would mean new GUI forms, CLI and API.

The ability to assign/update certificates to servers (may require restarting the whole server, full zato stop && zato start cycle)

Some sort of warnings displayed when a user logs in and it's less 180/120/60/30/15/etc. days to expiration dates

Nice to have in the first version but may wait a bit:

• The ability to assign/update certificates to servers with restarting each gunicorn worker serially one by one (this would be done automatically, users wouldn't even know anything, but this would make sure the server will always have a worker to process the messages)

I don't remember what the current state of SSL in gevent/gunicorn is - we are currently at 0.13.8/0.16.1 respectively.

https://github.com/zatosource/zato/blob/master/code/versions.cfg

But if adding SSL needs upgrading to newer versions we'll naturally do it.

What do you think about it? Would that suffice for your needs? Can you think of other things you'd like to see?

Also, if you need to have absolutely right now and you can put nginx in front of Zato I can work with you on how to validate incoming requests basing on request's HTTP headers - this needs a bit of coding (like 10-40 lines of code) and keeping config in Redis but is easily doable.

Thanks!

— Reply to this email directly or view it on GitHubhttps://github.com/zatosource/zato/issues/79#issuecomment-21078615 .

[dsuch]

Sure, I understand it. For SSL, can you please tell me if features I mentioned are what you need? Can you suggest anything else?

As for REST - outgoing RESTful connections are discussed in #45. This will be added soon (along with RESTful channels too) Can you please tell me what your preferred Python & REST library is though? Can you recommend nice and Pythonic examples perhaps?

[ghost]

The features you mention are spot on. The only other thing I can think of is HTTP Strict Transport Security needs to turned on. But I suspect you have that covered. The serious REST services I am writing currently are in Workday Studio (HR SaaS) . Although I have one in Python to call Mapquest's geocoding REST service which uses urllib.

The other issues around SSL have to do with exposures due to variations in packet size as described in http://research.microsoft.com/pubs/119060/WebAppSideChannel-final.pdf

[dsuch]

Hi @pruggera,

I haven't read that paper nor heard of Workday before but will read it and look the latter up so maybe I'll have some more questions still.

But for now I'm not sure what you'd need HSTS for? What sort of clients are you working with? I mean, my idea was that you either have HTTP or HTTPS and that's it. There's no way nor need to downgrade or upgrade protocols or to advertise that HTTPS should be used. At least that is what I thought :-)

Can you please tell me more about your need for HSTS and your clients apps? You cannot trust them? You don't know who they are? Just something that would let me understand it better why you need it here at all. I think I'm cool with adding it just would need to understand your situation closer.

Thanks!

[ghost]

It would be cool to be able to specify padding in an ESB that would pad the packets to X to foil packet size analysis.

We trust our vendor's SaaS systems. But we don't trust the internet with its many dynamically changing intermediate hops. HSTS is another step in the arms race for more secure communications. HSTS says all communications on that connection can only use HTTPS. This prevents SSL stripping attacks. This not a "must have" but would be a feature that most ESB's don't have yet.

[dsuch]

I read the paper and can see the rationale behind packets padding.

As for HSTS, I'll be sure to work on it, thanks for suggesting it. I may get in it touch with you when I actually start the work.

I didn't realize you'd be thinking of exposing an ESB directly to the net - I'd rather suggest putting a reverse SSL proxy - such as lighttpd or nginx - in front of it that would serve as another layer of security. What do you think about it?

[ghost]

That's a good question! And one I will ask this week of the integration group of which I am part.

We are now evaluating Mule for the role ESB. I wish we could put this off for a year. Zato might be a viable candidate by then.

[dsuch]

That's a good question! And one I will ask this week of the integration group of which I am part.

Yep. The thing is, the integration layer - if turns out successful - will with time handle more and more traffic related to various parts of your business, or it will integrate different applications unforeseen right now. It's good to insulate it from the net even if only for separate SSL layer and request throttling (which Zato has [1]) in case of any DDoS.

I wish we could put this off for a year. Zato might be a viable candidate by then.

@pruggera, I can see you're a Python person - do you think you could leave me your email address, mine is dsuch at zato.io - it would be cool to get in touch with you in some time with questions about what you actually found missing in Zato, what you'd like to see or would rather if was different or maybe you'll just like to chat about integrations in general.

This is the only way for me to have a picture of what people need and as you can see, I'm very interested in putting Python in use in new areas. Many thanks!

[1] https://zato.io/blog/posts/x-http-request-throttlingg-rate-limiting.html

[ghost]

I'll be in touch through email.

[ibeex]

as from 0.17.0 / 2012-12-25 gunicorn support SSL http://docs.gunicorn.org/en/latest/news.html#id7

[dsuch]

I've had a look at what gevent does with regards to SSL.

• They ship of copy of Python's ssl.py modified to suit gevent's needs (haven't checked yet to what extent it's modified, presumably only as much as it's needed to make it cooperative)
• Ultimately, everything boils down to using http://docs.python.org/2.7/library/ssl.html#ssl.wrap_socket after obtaining a socket, so it's nothing fancy

I don't know if this is enough for your needs @pruggera. wrap_socket does offer a knob to turn in the way of the cipher_list parameter which is one of https://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT but as far as padding to N bytes goes, this would have to be done on application side and would need custom clients to understand it, don't you think?

[ghost]

Sounds good about gevent. The padding of records is a policy level selection that needs to done just before encryption. On Sep 21, 2013 4:36 AM, "Dariusz Suchojad" notifications@github.com wrote:

I've had a look at what gevent does with regards to SSL.

• They ship of copy of Python's ssl.py modified to suit gevent's needs (haven't checked yet to what extent it's modified, presumably only as much as it's needed to make it cooperative)
• Ultimately, everything boils down to using http://docs.python.org/2.7/library/ssl.html#ssl.wrap_socket after obtaining a socket, so it's nothing fancy

I don't know if this is enough for your needs @pruggerahttps://github.com/pruggera. wrap_socket does offer a knob to turn in the way of the cipher_list parameter which is one of https://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT but as far as padding to N bytes goes, this would have to be done on application side and would need custom clients to understand it, don't you think?

— Reply to this email directly or view it on GitHubhttps://github.com/zatosource/zato/issues/79#issuecomment-24860884 .

[dsuch]

@pruggera - let's confirm we're thinking of the same thing :-)

When you say 'message padding' I understand it that a business payload is padded to N bytes so this forms a sort of a protocol to follow.

For instance, let's assume there's a business payload 40kB in size and you'd like for each message to be padded to 100kB.

The message Zato produces to a requesting app would consist of several elements

• business payload size
• padding size
• business payload
• padding

This would be the 100kB worth of data that would be sent over TLS to a requesting application thus concealing the actual size of the business payload.

Is that what you're thinking of too?

[ghost]

Yes, and here's my thinking behind this: One of the main goals of SOA is to separate the Service interface (represented by the contract and associated metadata) from the Service implementation (represented by the underlying code that the interface abstracts). A Service contract defines the relationship between a Service provider and a Service consumer, and the requirements for engaging in an exchange of information. A Contract is comprised of metadata and Policies. Policies drive the behavior of Services and their compositions. Policies are maintained separately from Service contracts because in large organizations the people that manage Service policies is separate from the people that manages the Service implementations or the Service contract metadata. And Policies are likely to change even when the Service contract metadata stay the same.

Padding is a Policy so that when the padding rules need to be changed the change can impact all contracts that have that padding policy attached.

On Sun, Sep 22, 2013 at 11:49 AM, Dariusz Suchojad <notifications@github.com

wrote:

@pruggera https://github.com/pruggera - let's confirm we're thinking of the same thing :-)

When you say 'message padding' I understand it that a business payload is padded to N bytes so this forms a sort of a protocol to follow.

For instance, let's assume there's a business payload 40kB in size and you'd like for each message to be padded to 100kB.

The message Zato produces to a requesting app would be consist of several elements

• business payload size
• padding size
• business payload
• padding

This would be the 100kB worth of data that would be sent over TLS to a requesting application thus concealing the actual size of the business payload.

Is that what you're thinking of too?

— Reply to this email directly or view it on GitHubhttps://github.com/zatosource/zato/issues/79#issuecomment-24887868 .

[dsuch]

Padding is a Policy so that when the padding rules need to be changed the change can impact all contracts that have that padding policy attached.

Good stuff.

I'm wondering if you can suggest any other policies to add in addition to that one?

I'm just thinking about GUI navigation right now, whether the one you mention would be better served from Security -> Policies -> Padding or Services -> Policies -> Message padding.

You know, where to add really, if it makes sense to add stuff to Services or just to Security.

Other thing is, we can certainly update the Python client https://zato.io/docs/progguide/clients/python.html so it honours the policy but clients in other languages would have to cope with it on their own, following the docs, I guess?

[ghost]

Policies, Contracts, and Service Implementations have a many to many to many relationship.

Policy menu could have policies by class (Access control, Authentication, Message security (padding is one of these), QoS, Reliable message, Routing, Logging, and Mediation.

All objects need to be versioned with the ability to package a set of related objects.

Service manager has a REST interface so clients can read the policies and contracts.

[dsuch]

Right, I get it now.

So except for message padding, certain aspects of authorizations and the REST interface - this all can be done right now but it's not called policies. This also means that making it read 'Policies' would me a major GUI overhaul which is unlikely to happen, I'm afraid, given that you can do the same only the name is missing.

This is similar to http://eaipatterns.com where Zato already has many of them but they have casual names.

Perhaps this can be explained in a series of articles in the docs? I'm thinking of both EAI patterns and policies. Sort of 'how to achieve such and such policy/make use of an integration pattern using tools Zato offers out of the box'?

[ghost]

Best practices is to use a standard vocabulary. But who's vocabulary? Robert Doigneau or Thomas Erl?

On Wed, Sep 25, 2013 at 4:54 AM, Dariusz Suchojad notifications@github.comwrote:

Right, I get it now.

So except for message padding, certain aspects of authorizations and the REST interface - this all can be done right now but it's not called policies. This also means that making it read 'Policies' would me a major GUI overhaul which is unlikely to happen, I'm afraid, given that you can do the same only the name is missing.

This is similar to http://eaipatterns.com where Zato already has many of them but they have casual names.

Perhaps this can be explained in a series of articles in the docs? I'm thinking of both EAI patterns and policies. Sort of 'how to achieve such and such policy/make use of an integration pattern using tools Zato offers out of the box'?

— Reply to this email directly or view it on GitHubhttps://github.com/zatosource/zato/issues/79#issuecomment-25080015 .

[dsuch]

Robert Doigneau or Thomas Erl?

How about Zato's? :-)

My idea is for Zato to always be lightweight and not follow J2EE servers that are strictly following the letter of various specifications in a way that makes them pure horror UI-wise.

I know it's not what you're suggesting but I'd like to emphasize it that when a, say, single checkbox will do, let's use it. The idea behind it can be a high-level architectural concept but that can be explained in the docs only.

What do you think?

[dsuch]

OK, just a follow up on SSL.

Turned out I had to contribute a bit of SSL options to gunicorn. I've got it ready in my fork

https://github.com/dsuch/gunicorn/commit/16f9974 https://github.com/dsuch/gunicorn/commit/4c617c2

and will talk it through with gunicorn's devs soon.

All looks good then and I'm adding it to Zato next.

[dsuch]

OK, in 5b36c3b I've just pushed changes that add TLS with optional client certificate validation.

There is no GUI for assigning client certificates to particular services yet and I'll add it in another ticket.

A thing that was also added was storing the certificate, as a dictionary and in DER along with its fingerprint in WSGI environment under keys:

• zato.tls.client_cert.dict
• zato.tls.client_cert.der
• zato.tls.client_cert.sha1

This all will be properly documented before releasing 1.2, naturally.