Kafka Ver. 0.11.0.1 uses OracleJDK

[JiniousChoi]

The default Kafka version 0.11.0.1 utilizes JAVA(TM) internally, it seems. So, I got wondering if you have any issue with upgrading the version of Kafka, say, to 3.4.x or to 3.5.x. Because both of those docker images use OpenJDK I think no one would need to worry about the license issue.

So, my question is, in a nutshell, do I need to change the version of Kafka on my own if I want to keep it free, using Openwhisk as a whole?

$ docker exec -i kafka0 java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

[dgrove-oss]

I think upgrading to a more recent Kafka release (eg. 2.2.x) would be a worthwhile project. I don't know how much or little effort this would take. Someone would need to take a careful look at the release notes since 0.11.x and see what changes were made in the clients and in the configuration details of the docker image.

[JiniousChoi]

@dgrove-oss I am in a position where I need to upgrade it to a recent release due to the license issue. Since it seems that it's me who wants this change the most why don't I have a chance to take a look at it and proceed with the upgrading process?

[dgrove-oss]

Sounds good; thanks @JiniousChoi!

[style95]

@JiniousChoi If you look into it, it would be greatly appreciatable.

I think a few Kafka libraries are being used in some places such as KafkaMessagingProvider and MessageFeed. (Since I didn't look into it deeply, there can be more components which dependent on Kafka.)

We need to figure out such components are compatible with the latest Kafka. I think we may need to upgrade those libraries as well.

[JiniousChoi]

Sounds good! Then I'll look into the release notes for Kafka soon

[JiniousChoi]

Also, I have a question. I read some hints that the team would use Vagrant for the development of Openwhisk at this README.md. It is still valid? If so, would you use it for integration tests as well? Some recommendations would be appreciated.

[style95]

In the upstream Travis build, we are deploying OpenWhisk natively.

[JiniousChoi]

@dgrove-oss Then how about the local test? Do you use Vagrant to keep the environment the same, controlled across all the developers? Or do you just let them develop on whatever host machine they have?

[rabbah]

Likely there are many camps on this. Some like kube some native ansible and some compose to give an idea. My suggestion: use the standalone controller it’ll be easier and faster for you. We maintain the vagrant box so infrequently. I’ve been increasingly switching from ansible to standalone and like it. https://github.com/apache/openwhisk/blob/master/core/standalone/README.md

[dgrove-oss]

I think Vagrant is the least used environment. Perhaps it is time to remove it from the documentation.

[rabbah]

less is more :)

[JiniousChoi]

That's what I was asking for! Thanks a lot :)

[JiniousChoi]

And, I'd love to read through a guiding document for Openwhisk developers if there is any. Neither CONTRIBUTING.md nor Recommended-Reading-for-OpenWhisk-Developers wiki seems comprehensive, judging by not including the local environment as you guys recommend.

[style95]

I think Vagrant is the least used environment. Perhaps it is time to remove it from the documentation.

+1 on this.

And I think it would be also great to change our QuickStart guide to the one based on standalone deployment. https://github.com/apache/openwhisk#quick-start

It's easier to setup and use.

[JiniousChoi]

@rabbah What happens if I run java -jar openwhisk-standalone.jar without '--kafka' switch? It seems that '--kafka' creates a Kafka docker container. But the standalone still works without the '--kakfa' switch. How does that work if there is no embedded Kafka running? I tried and looked into the code to understand but it's not so clear to me.

@rabbah Here is the real question. Can I use external Kafka, not the embedded one, when using standalone Openwhisk? I tried overriding the default standalone.conf with my own 'my-standalone.conf' as following:

include classpath("standalone.conf")

whisk {
  kafka {
      hosts = "ow-kafka-upgrade.example.com:9093"
  }
}

But it didn't work. So I am looking into the source code a bit but it's hard to tell due to the complex config system, whether this 'whisk.kafka.hosts' is in effect on the Controller code.

FYI, I am trying to set up a cozy build-and-test environment for this issue 4717, hence trying this standalone environment as you suggested.

[chetanmeh]

What happens if I run java -jar openwhisk-standalone.jar without '--kafka' switch?

Then OpenWhisk runs in lean mode (#4216) without Kafka. With --kafka it uses embedded kafka. See #4628 for details on what all was done for this mode. Note that it uses Kafka 2.1.1. So basic stuff works as expected with this Kafka version.

To make standalone connect to existing running Kafka would need some work as current approach was implemented with embedded mode and has to adapt the listeners to workaround the Docker networking. May be try launching Standalone without --kafka switch and instead set following system properties

-Dwhisk-config.kafka.hosts=localhost:9093 -Dwhisk.spi.MessagingProvider=org.apache.openwhisk.connector.kafka.KafkaMessagingProvider
-Dwhisk.spi.LoadBalancerProvider=org.apache.openwhisk.standalone.KafkaAwareLeanBalancer

Change the port as per your setup. I have not tried it so cannot comment if it works as expected!

[JiniousChoi]

@chetanmeh Thanks for your detailed tips! I'll look into the #4628 and the system.properties hack. Also, I'll share the result.

[JiniousChoi]

testSystemBasic.zip

I ran the SystemBasic test against my working system on Vagrant(Ubuntu16.04) with the following command:

./gradlew :tests:testSystemBasic -Dwhisk.auth="23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP" -Dwhisk.server=https://localhost -Dopenwhisk.home=`pwd`

And I have 2 failed tests.

Is it me doing something wrong? I'd like to know a proper test environment where I can see them all passed, 100%.

Do I need to see all-green in my test result before a PR? Please guide me on the minimum requirement for PR.

[rabbah]

There are some tests that are known to fail sporadically. Unless you suspect your change is the reason for the failing tests, I would open the PR and let Travis determine status more thoroughly.

[JiniousChoi]

4802