search

amino-os / Amino.Run

Amino Distributed OS - Runtime Manager
Apache License 2.0
29 stars 12 forks source link

Verify Deployment on Kubernetes #338

Open dhzhuo opened 5 years ago

quinton-hoole commented 5 years ago

@DonghuiZhuo This belongs in https://github.com/Huawei-PaaS/DCAP-Sapphire-Examples, right?

quinton-hoole commented 5 years ago

Added to Milestone Release Candidate 1, removed BEFORE_RELEASE label.

ackeri commented 5 years ago

From what I can see, the Readme.md in DCAP-Sapphire-Examples/k8-deployment doesn't make sense anymore. No relevant gradlew to run. The DCAP-Sapphire repository also doesn't have any of the relevant build targets. This is broken at least from a usability perspective.

quinton-hoole commented 5 years ago

Thanks @ackeri ! Great to see you're back in the saddle.

@DonghuiZhuo This one needs some answers from you, please see above.

dhzhuo commented 5 years ago

@Preethikathamuthu Can you please verify and update the documents accordingly? Thanks

Preethikathamuthu commented 5 years ago

@DonghuiZhuo yes.deployment is not working.will fix that.Thanks

Preethikathamuthu commented 5 years ago

k8s deployment for minnietwitter is not working because now it has been completely changed to a pure android application.Previously, in minnietwitter-app.yaml file, to run the app, TwitterWorldGenerator class was specified, where java main method was present, so it was running like a java application in k8s. Now, we need to set up emulator and android platform in k8s to run the app. So, can we do k8s deployment for minnietwitter java app only or do we need to deploy android minnietwitter app also?

Preethikathamuthu commented 5 years ago
quinton-hoole commented 5 years ago

Wonderful! Thanks @Preethikathamuthu

Preethikathamuthu commented 5 years ago

@quinton-hoole-2 @DonghuiZhuo Do we need k8s deployment for both Minnietwitter java application and android application?

quinton-hoole commented 5 years ago

The client-only side (whether it's running on Android or not) does not, for now, need to be deployed using kubernetes. But oms and the kernel server do. And there needs to be a test demonstrating that those client's can run against the kubernetes-deployed oms and ks.

Make sense? On Mon, Nov 5, 2018, 05:07 Preethikathamuthu <notifications@github.com wrote:

@quinton-hoole-2 https://github.com/quinton-hoole-2 @DonghuiZhuo https://github.com/DonghuiZhuo Do we need k8s deployment for both Minnietwitter java application and android application?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Huawei-PaaS/DCAP-Sapphire/issues/338#issuecomment-435867302, or mute the thread https://github.com/notifications/unsubscribe-auth/ApoAeMimZ9U3PNeWgXddXucMhAEsEYLYks5usDf6gaJpZM4X2uO4 .

Preethikathamuthu commented 5 years ago

yeah @quinton-hoole-2 Thanks

quinton-hoole commented 5 years ago

@Preethikathamuthu @VenuReddyHuawei Here is the document about android emulator networking that I mentioned on the call. It might help you to resolve the networking issue that @Preethikathamuthu mentioned:

https://developer.android.com/studio/run/emulator-networking

Also, here is some info on Kubernetes networking, in case you need it. In summary, depending on how your kubernetes cluster's networking is set up, you may need to expose the OMS and Kernel server via a load balancer with a public IP address:

https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types https://kubernetes.io/docs/concepts/cluster-administration/networking/

Preethikathamuthu commented 5 years ago
quinton-hoole commented 5 years ago

Thanks @Preethikathamuthu . We'll look into it.

Preethikathamuthu commented 5 years ago
sungwook-moon commented 5 years ago

It will be great if we can test with multiple servers (e.g., each POD or container hosting the kernel server). DHT+Consensus or DHT+MasterSlave in two regions by DHT and two replicas by followed DMs will require 4 hosts, and that may give us good demonstration of capability.

SrinivasChilveri commented 5 years ago

Basically if we deploy app also as pod then everything goes smooth as expected.

But as we want application as standalone then all the stubs(oms/ks), group policy stub etc needs to have proper endpoints & accessible from the application. so we have changed the code to support the same...
In oms we changed below code (Given Node IP & Given proper port ) and added the k8s service for the different exported objects...

System.setProperty("java.rmi.server.hostname", args[0]); OMSServer omsStub = (OMSServer) UnicastRemoteObject.exportObject(oms, 0); KernelServer localKernelServerStub = (KernelServer) UnicastRemoteObject.exportObject(localKernelServer, 0);

so that app which has oms stub can able to invoke the methods properly. Minnie twitter app working as expected. We need to analyse properly all other side impacts , other issues with these changes etc then we can change the code (specially giving the node IP for the stubs).

OMS ready with omsStub:Proxy[OMSServer,RemoteObjectInvocationHandler[UnicastRef [liveRef: [endpoint:127.0.0.1:38866,objID:[247c1d3f:1672a771d16:-7fff, 2374890344654643044]]]]]localKernelServerStub:Proxy[KernelServer,RemoteObjectInvocationHandler[UnicastRef [liveRef: [endpoint:127.0.0.1:38866,objID:[247c1d3f:1672a771d16:-7ffe, 334361290202287981]]]]]

Currently we have not changed the code properly, we just changed the scripts, partially code patch to est & make it work for the Mini twitter application. Mini Twitter app is working with these changes.

We need to come up with proper solution. (may need to use LB /Ingress & external DNS etc) Any other suggestions are welcome.

Preethikathamuthu commented 5 years ago

PR : https://github.com/Huawei-PaaS/DCAP-Sapphire-Examples/pull/31

prostil commented 5 years ago

Need to test with more than 2 pods

Preethikathamuthu commented 5 years ago

With multiple kernel servers, an issue with resolving dns name into IPAddress occurs. Functionality works fine when tested by adding entries in /etc/hosts. will fix that issue shortly

Preethikathamuthu commented 5 years ago

Deployment Scenario:

Issue:

RootCause:

Temporary fix:

AmitRoushan commented 5 years ago

Following is my findings for issue mentioned by @Preethikathamuthu Currently kernel server uses InetSocketAddress object to register itself with OMS. The object maintains only IP address and port information. With same object, OMS try to get hostname (host.getHostName) of registered kernel server.

Now InetSocketAddress.getHostName always return host name if InetSocketAddress object has hostname with it otherwise it tries to do DNS lookup with available IP address. If even DNS lookup fails , it returns textual representation of IP address.

Currently when OMS + kernel server runs in same VM or in different VM , DNS is not set so it always uses textual representation of IP address as hostname (loopback when 127.0.0.1 is used) and hence it work fine .

But if OMS + Kernel server runs behind any NATed environment or any environment where DNS is setup locally (like in Kubernetes cluster where KubeDNS is used ) , OMS may endup resolving wrong hostname which might not be accessible from outside.

So to make OMS and Kernel server available in above mentioned environment, Amino can expose a command line argument --hostname which can be used to set hostname in InsetSocketAddress object and java.rmi.server.hostname so that it always return set hostname across Amino platform. Similar approach is adopted for JAVA RMI as well.

@quinton-hoole @VenuReddy2103 @maheshrajus @Vishwa4jeet Please let me know your opinion