GemfireSink throws no security properties provided.

[sabbyanandan]

From @rahulvijay24feb on May 25, 2017 20:50

Hi,

When i try using Gemfire sink from the starter applications, I get the below exception. Do we have an option for providing this? Would there be any additional parameters apart from username,password inside this file ? 2017-05-25T16:45:22.739-04:00 [APP/PROC/WEB/0] [OUT] [info 2017/05/25 20:45:22.739 UTC tid=0xb9] Error prefilling connections : com.gemstone.gemfire.security.AuthenticationRequiredException: No security- properties are provided 2017-05-25T16:45:22.739-04:00 [APP/PROC/WEB/0] [OUT] com.gemstone.gemfire.security.AuthenticationRequiredException: No security- properties are provided 2017-05-25T16:45:22.740-04:00 [APP/PROC/WEB/0] [OUT] at com.gemstone.gemfire.internal.cache.tier.sockets.HandShake.readMessage(HandShake.java:1456)

Copied from original issue: spring-cloud/spring-cloud-dataflow#1518

[sabbyanandan]

@rahulvijay24feb: This looks like the breaking change wrt to security that was introduced in GF 9.x release. What version of Gemfire are you using?

[rahulvijay24feb]

I am using gemfire as part of the pivotal cloud cache on PCF. It is 9.0+ version of gemfire.

Pivotal Cloud Cache Version v1.0.2 Current Pivotal Cloud Cache details:

Version: v1.0.2 Release Date: 3rd May 2017 Software Component Version: GemFire v9.0.2 Compatible Ops Manager Version: v1.9.11 Compatible Elastic Runtime Version: v1.9.11 vSphere Support: Yes Azure Support: Yes GCP Support: Yes OpenStack Support: Yes AWS Support: Yes IPSec Support: No

[sabbyanandan]

The gemfire source/sink isn't compatible with PCC yet. There's upstream compatibility needed from Spring Data Gemfire and we are waiting on that. It is compatible, however, if you're running it against standalone GF 9.x cluster. cc:/ @markpollack @jxblum

[rahulvijay24feb]

Oh :(

[rahulvijay24feb]

Hello,

I was trying to create a custom sink to connect to gemfire instead using starter app. I was succesfully able to run similar implementation without the concept of sink inside PCF where as get the same error as above for my custom gemfire sink.

2017-06-02T12:32:10.176-04:00 [APP/PROC/WEB/0] [OUT] Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gemfireCache': FactoryBean threw exception on object creation; nested exception is com.gemstone.gemfire.security.AuthenticationRequiredException: No security-* properties are provided

Do you see a work around option using SCDF which I can live with for now ?

[sabbyanandan]

Hi, @rahulvijay24feb. My previous comment was specifically wrt to the "PCC tile" - it is still not compatible and we are working on it. However, the Gemfire sink is compatible with Gemfire 9.0 release. The latest Bacon.RELEASE includes the compatibility. It is not clear whether you are using the latest release, so let's start there.

[rahulvijay24feb]

As i have previously mentioned, I am using a custom sink built with below dependency.

UTF-8 UTF-8 1.8 **Dalston.RELEASE**

<dependencies>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-gemfire</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.6.3</version>
    </dependency>
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

[sabbyanandan]

The OOTB sink should work directly with Gemfire 9.0 - have you tried it? (you can download it from here)

I don't get the reason why you have to create a custom sink to write to Gemfire. If you are strictly trying to make it work on PCC, there are several upstream enhancements that the Spring team is working on and it is not ready yet. @markpollack might have other insights to share re: PCC.

[rahulvijay24feb]

@sabbyanandan : Unfortunately, that is exactly my issue , sink to a gemfire on PCC. The logs i have given previously relate to a customsink connecting to a PCC.

[jxblum]

Hi Guys-

Sorry for the late response everyone. I have been tied up in several other concurrent things.

So, the GemFire Cluster/Service in PCF, and specifically for PCC, requires authentication OOTB; this much we know. What is not apparent is how to configure a Spring Boot app using PCC to connect to, behind the scenes, Pivotal GemFire, particularly when auth is involved.

Much like a pushed Spring Boot app using PCC, Spring Cloud Data Flow's GemFire Sink is just a "GemFire Cache Client" (or a ClientCache).

So, what does this mean?

Well, in addition to the SCDF options (or properties) exposed to configure this (under-the-hood) ClientCache instance, GemFire Pool, you can also include additional "GemFire properties".

DISCLAIMER: I know that what I am about to say is not the preferred Spring [Cloud DataFlow] way to do this, but is a "temporary workaround" until I get the Spring Data GemFire bits in order, a WIP and my utmost, highest priority.

So, knowing that the SCDF GemFire Sink is just a ClientCache we can use this fact and provide additional "properties" recognized by GemFire itself on the classpath (can be in the root of a deployed app JAR, for instance).

Essentially, I think all that needs to be done is...

1. Add a gemfire.properties file to the CLASSPATH (or in the root of an app deployed JAR file) with the following properties...

locators = <host[port] of the PCC service GemFire Locator; I think there maybe 2; either Locator endpoint will work; only 1 is technically needed, but if you prefer to specify both, then the property syntax for this property is: host1[port1], host2[port2]>
security-username = <PCF/PCC service username>
security-password = <PCF/PCC service password>
security-client-auth-init = <Fully qualified class/method name to a static factory method used to create an instance of an implementation of GemFire's [AuthInitialize](http://gemfire-90-javadocs.docs.pivotal.io/org/apache/geode/security/AuthInitialize.html) interface>

2. As mentioned in 1. above, for the security-client-auth-init property, you must implement a custom GemFire AuthInitialize interface. This is really not as difficult as it sounds and, fortunately, I have such and example here. The factory method is here. And the configuration for this using the GemFire properties noted above (along with security-username and security-password, are here).

These references above are part of a test class I wrote testing GemFire auth between a client and server Spring app (well, in this case, the test is a cache client and it contains the configuration necessary to launch a "secure" GemFire Server manually, which is used by the test cache client during the test run). You don't really need to worry about the server-side (with GemFire's Authenticator interface impl and all that jazz), but it will kind of give you basic understanding of what went into securing the GemFire-based PCC service and what it is sort of doing to authenticate a GemFire cache client (like a Spring Boot app, or SCDF's GemFire Sink) pushed to PCF using PCC.

Again, GemFire (clients and peers) will always look for certain files (e.g. properties) and such on the classpath (or sometimes, working directory of the JVM process) in well-known locations, regardless of the wrapping/adapting technology (Spring Boot, SCDF, Spring Data GemFire (SDG), non-Spring, or otherwise), and when GemFire finds these artifacts, it will apply them.

More information on securing GemFire and authentication in particular can be found here.

Finally...

DISCLAIMER: I have not had an opportunity to try this myself. However, I cannot see why this would not work and I do think it is worth a try.

Also know that I am working to improve this situation and make it more Spring friendly. I am driving several initiatives forward in this regard, w.r.t. Spring specifically.

First, is to make changes directly in Spring Data GemFire itself.

This involves improving on the existing @EnableSecurity annotation I introduced to SDG already to now, actually provide a default implementation of GemFire's AuthInitialize interface on-behalf of the user so that users will not have to implement this interface themselves. All users will need to provide is well-known username/password properties, and like Spring Boot's "auto-configuration" support when, these properties are "present", SDG will apply the necessary configuration (which includes the default implementation of the AuthInitialize interface).

For the interested reader, I blogged about the new Security support in SD Geode (also applies to Spring Data GemFire now (as of 2.0, current is 2.0.0.M3) as well), here. The examples for this blog are here. The test class uses Spring Profiles to showcase the many different ways that GemFire Security can be easily configured using SDG's new @EnableSecurity annotation. For instance, here is the client.

Again, a user must provide an implementation of AuthInitialize interface for now, but that is going to change. NOTE: this AuthInitialize impl in this example is quite a bit more involved, but that is not strictly necessary. Its impl was to support my examples.

Second, I am planning to add more "auto-configuration" support when Spring Boot is in the mix to make all the PCF PCC services discoverable and automatically authenticated/setup when pushing Spring [Boot] (or SCDF, or plain SDG) apps.

Anyway, I hope this helps.

I truly apologize for the inconvenience and we working hard to make sure the experiences is first class and second to none.

Please feel free to follow up here with feedback or additional questions if you decide to try the workaround noted above, in case of any issues.

Regards, John Blum SDG Lead

[rahulvijay24feb]

@jxblum : Thank you very much for the informative post :+1: Will keep it posted with my results.

[rahulvijay24feb]

@jxblum : I was finally able to create/execute a custom gemfire sink with a custom implementation of AuthInitialize and security-client-auth-init token ! Will upload and sharing the link for the sample shortly.

[jxblum]

Hi @rahulvijay24feb - Glad to hear you got it working; yay! I am making progress on @EnableSecurity; Will have good news to share with everyone, hopefully soon. Looking forward to seeing your sample.

Cheers, John

[rahulvijay24feb]

@jxblum : I uploaded the sample below simplifying my usecase: https://github.com/rahulvijay24feb/CustomGemfireDBSink

Eagerly waiting for enhanced EnableSecurity :smile:

[sabbyanandan]

Hi, @jxblum. When you get a chance, it'd be great to get an update on this. (see also: spring-cloud-stream-app-starters/gemfire#9)