search

binocarlos / powerstrip-weave

A Powerstrip plugin that runs weave inside a container and ensures that containers are connected to the weave network before running their entrypoints.
36 stars 3 forks source link

binocarlos/wait-for-weave is exiting without any logs #13

Closed sbhaskaran closed 9 years ago

sbhaskaran commented 9 years ago

docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e1d06951ebf3 zettio/weave:git-8e65c7127d82 "/home/weave/weaver 12 minutes ago Up 12 minutes 0.0.0.0:6783->6783/tcp, 0.0.0.0:6783->6783/udp weave

c6fc0d9f544e binocarlos/wait-for-weave:latest "/home/weavewait/wai 12 minutes ago Exited (0) 12 minutes ago weavew ait
21f2ceedf2b2 binocarlos/powerstrip-weave:latest "/srv/app/run.sh lau 12 minutes ago Up 12 minutes 80/tcp powers trip-weave

I am trying to run powerstrip-weave on latest CoreOs.

In older versions its complaining "clinet and sever version did not match" can you please guide me how to rectify this?

binocarlos commented 9 years ago

Hi - I've included a client only docker binary into the image so it can start the weave and wait-for-weave containers that you've listed. The client and server version did not match error you see was before this change`.

Do you have powerstrip itself running? Also - I'm assuming that you have an adapters.yml file somewhere?

sbhaskaran commented 9 years ago

Thanks for your reply.. and fixing the version issue.

But now the containers are failing to start. I am following the doc in gthub to setup using the latest versions

  1. docker run -d --name powerstrip-weave \ --expose 80 \ -v /var/run/docker.sock:/var/run/docker.sock \ binocarlos/powerstrip-weave launch

    binocarlos/wait-for-weave:latest is exiting without any logs.

and

docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9ec33884e3c9 zettio/weave:git-8e65c7127d82 "/home/weave/weaver 2 minutes ago Exited (1) 2 minutes ago weave
a781ce6bac3d binocarlos/wait-for-weave:latest "/home/weavewait/wai 2 minutes ago Exited (0) 2 minutes ago weavewait
9417f890a798 binocarlos/powerstrip-weave:latest "/srv/app/run.sh lau 2 minutes ago Up 2 minutes 80/tcp powerstrip-weave

zettio/weave exitedd saying

docker logs 9ec33884e3c9 weave 2015/03/24 18:44:03.537380 Command line options: map[iface:ethwe name:7a:ef:8c:e0:16:8c nickname:dockerhost wait:20] weave 2015/03/24 18:44:03.537425 Command line peers: [] weave 2015/03/24 18:44:23.669367 Unable to find interface ethwe

does it needs to have a ethwe setup before hand?

cat ~/powerstrip-demo/adapters.yml endpoints: "POST //containers/create": pre: [weave] "POST //containers//start": post: [weave] "POST //containers/*/restart": post: [weave] adapters: weave: http://weave/v1/extension

here is the docker daemon.

root 1231 1 89 Mar20 ? 3-12:06:09 /usr/bin/docker --daemon --storage-driver=btrfs --host=fd:// --insecure-registry="0.0.0.0/0"

binocarlos commented 9 years ago

Hey - so can I check which kernel version you have?

Can you paste the results of uname -a

The reason I ask is this thread on the weave issues: https://github.com/zettio/weave/issues/470

Also - can I check the format of your adapters.yml

endpoints:
"POST //containers/create":
pre: [weave]

Should be

endpoints:
"POST /*/containers/create":
pre: [weave]
sbhaskaran commented 9 years ago

@binocarlos Here is the kernel version(3.17.2). 3.17.2 #2 SMP Thu Dec 11 02:25:47 EST 2014 x86_64 Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz GenuineIntel GNU/Linux

Also I can run weave binary on the host and make it assign IP for the containers. Also I double confirmed my adapter.yml , the above was a copy paste error.

I will try getting into shell and running weave using powerstrip-weave container as you described in #11

sbhaskaran commented 9 years ago

I still did not get it working and when I tried to run wait-for-weave in bin bash it says

interface ethwe not found after 10 seconds

where is it looking for interface?

I tried the binary too same result.

stage # ./wait-for-weave interface ethwe not found after 10 seconds

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e29d093f7237 ubuntu:14.04 "/bin/bash" 5 minutes ago Up 5 minutes mad_mestorf
1d9416bb067d zettio/weave:0.9.0 "/home/weave/weaver 6 minutes ago Up 6 minutes 0.0.0.0:6783->6783/udp, 0.0.0.0:6783->6783/tcp weave
root@docker98 stage #

I have weave (started thorugh binary weave launch)and ubuntu running , ubuntu can get the Ip from weave. so it looks like the host as such is capable of creating the interface.

I was going through run.sh and found that cmd-launch(){ docker run --name weavewait -e "WAIT_FOR_WEAVE_QUIT=yes" binocarlos/wait-for-weave cmd-weave launch "$@" node /srv/app/index.js }

can you please help me understand what it will do? specially WAIT_FOR_WEAVE_QUIT=yes?

also I went ahead and did the rest of the steps even though the wait-for-weave is not running and here is the logs from

zettio/weave

server listening on port: 80

undefined:1 client and server don't have same version (client : 1.16, server: 1.15) ^ SyntaxError: Unexpected token c at Object.parse (native) at /srv/app/actions/start.js:34:30 at /srv/app/dockerclient.js:15:7 at ConcatStream. (/srv/app/node_modules/concat-stream/index.js:32:43) at ConcatStream.emit (events.js:117:20) at finishMaybe (/srv/app/node_modules/concat-stream/node_modules/readable-stream/lib/_stream_writable.js:460:14) at afterWrite (/srv/app/node_modules/concat-stream/node_modules/readable-stream/lib/_stream_writable.js:340:3) at /srv/app/node_modules/concat-stream/node_modules/readable-stream/lib/_stream_writable.js:327:9 at process._tickCallback (node.js:448:13)

and clusterhq/powerstrip:latest

2015-03-26 03:09:37+0000 [HTTP11ClientProtocol,client] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fad1feabd88> 2015-03-26 03:09:37+0000 [DockerProxyClient,client] "172.17.42.1" - - [26/Mar/2015:03:09:37 +0000] "POST /v1.15/containers/create HTTP/1.1" 404 65 "-" "Docker-Client/1.3.3" 2015-03-26 03:09:37+0000 [DockerProxyClient,client] Stopping factory <powerstrip.powerstrip.DockerProxyClientFactory instance at 0x7fad1feb3f38> 2015-03-26 03:09:37+0000 [HTTPChannel,1,172.17.42.1] Starting factory <powerstrip.powerstrip.DockerProxyClientFactory instance at 0x7fad1f251320> 2015-03-26 03:09:56+0000 [DockerProxyClient,client] "172.17.42.1" - - [26/Mar/2015:03:09:55 +0000] "POST /v1.15/images/create?fromImage=binocarlos%2Fpowerstrip-weave-example&tag=late" 2015-03-26 03:09:56+0000 [DockerProxyClient,client] Stopping factory <powerstrip.powerstrip.DockerProxyClientFactory instance at 0x7fad1f251320> 2015-03-26 03:09:56+0000 [HTTPChannel,2,172.17.42.1] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fad1f251368> 2015-03-26 03:09:56+0000 [HTTP11ClientProtocol,client] Starting factory <powerstrip.powerstrip.DockerProxyClientFactory instance at 0x7fad1f264560> 2015-03-26 03:09:56+0000 [HTTP11ClientProtocol,client] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fad1f251368> 2015-03-26 03:09:56+0000 [DockerProxyClient,client] "172.17.42.1" - - [26/Mar/2015:03:09:55 +0000] "POST /v1.15/containers/create HTTP/1.1" 201 90 "-" "Docker-Client/1.3.3" 2015-03-26 03:09:56+0000 [DockerProxyClient,client] Stopping factory <powerstrip.powerstrip.DockerProxyClientFactory instance at 0x7fad1f264560> 2015-03-26 03:09:56+0000 [HTTPChannel,3,172.17.42.1] Starting factory <powerstrip.powerstrip.DockerProxyClientFactory instance at 0x7fad1f264440> 2015-03-26 03:09:56+0000 [DockerProxyClient,client] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7fad1f264878> 2015-03-26 03:09:56+0000 [DockerProxyClient,client] Stopping factory <powerstrip.powerstrip.DockerProxyClientFactory instance at 0x7fad1f264440> 2015-03-26 03:09:56+0000 [HTTP11ClientProtocol,client] while running chain Traceback (most recent call last): Failure: twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]

sounds like the version issue is back.

binocarlos commented 9 years ago

Hey - so from this code:

cmd-launch(){
  docker run --name weavewait -e "WAIT_FOR_WEAVE_QUIT=yes" binocarlos/wait-for-weave 
  cmd-weave launch "$@"
  node /srv/app/index.js
}

docker run --name weavewait -e "WAIT_FOR_WEAVE_QUIT=yes" binocarlos/wait-for-weave runs a container that quits immediately. The point is that other containers can now have a --volumes-from weavewait option set and they will have access to the wait-for-weave binary from inside.

This is so we can start any container but block its entrypoint until the weave interface is connected.

cmd-weave launch "$@" uses the included client-only docker binary to talk to docker and start the weave container.

node /srv/app/index.js starts the powerstrip adapter.

Conclusion - its OK that wait-for-weave is not running, the container just needs to exist to allow --volumes-from

So - we have a version problem - can you:

Let me know if pulling powerstrip-weave grabs a later image or not.

Thanks!

sbhaskaran commented 9 years ago

@binocarlos

docker version Client version: 1.3.3 Client API version: 1.15 Go version (client): go1.3.2 Git commit (client): 5dc1c5a OS/Arch (client): linux/amd64 Server version: 1.3.3 Server API version: 1.15 Go version (server): go1.3.2

Git commit (server): 5dc1c5a

docker pull binocarlos/powerstrip-weave:latest Pulling repository binocarlos/powerstrip-weave 33746eea6148: Download complete 511136ea3c5a: Download complete 3b3a4796eef1: Download complete 50ec2d202fe8: Download complete 1073b544a1cb: Download complete 6a7520a7fd8c: Download complete 8d2c59617a00: Download complete dbe39e1288de: Download complete 2913aa111764: Download complete 16c185acbd34: Download complete 1549a2ff9d8d: Download complete 5c56afc37b51: Download complete bf59c8ed38cf: Download complete ea711f4feff1: Download complete 79868bd755bd: Download complete 16229b065aec: Download complete 8a660bc2d559: Download complete 6e9603505e52: Download complete 31b54e80a26d: Download complete a19542006097: Download complete 1c9ac61e378f: Download complete 538e59084b49: Download complete 515636cf430d: Download complete Status: Image is up to date for binocarlos/powerstrip-weave:latest

sounds like the image I have is latest. It did not pull any new layers.

binocarlos commented 9 years ago

Hey so something funky is going on for us to get the message:

client and server don't have same version (client : 1.16, server: 1.15)

where are we getting client: 1.16 from - the results of docker version from the static docker binary inside the container is:

Client version: 1.3.1
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 4e9bbfa-dirty
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

We can test this real quick with the command:

docker run -ti --rm --name powerstrip-weave \
    --expose 80 \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --entrypoint="/bin/bash" \
    binocarlos/powerstrip-weave

Once inside the container - try doing docker ps -a - if this works then the error above is coming from a strange place :-)

sbhaskaran commented 9 years ago

@binocarlos

that works . here is the output

docker run -ti --rm --name powerstrip-weave --expose 80 -v /var/run/docker.sock:/var/run/docker.sock --entrypoint="/bin/bash" binocarlos/powerstri- root@7367145d2843:/usr/local/bin# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7367145d2843 binocarlos/powerstrip-weave:latest "/bin/bash" 5 seconds ago Up 4 seconds 80/tcp powerstrip-we 109e585d9c5f zettio/weave:0.9.0 "/home/weave/weaver 3 hours ago Up 3 hours 0.0.0.0:6783->6783/tcp, 0.0.0.0:6783->6783/udp weave
root@7367145d2843:/usr/local/bin# docker version Client version: 1.3.1 Client API version: 1.15 Go version (client): go1.3.3 Git commit (client): 4e9bbfa-dirty OS/Arch (client): linux/amd64 Server version: 1.3.3 Server API version: 1.15 Go version (server): go1.3.2 Git commit (server): 5dc1c5a root@7367145d2843:/usr/local/bin# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7367145d2843 binocarlos/powerstrip-weave:latest "/bin/bash" 41 seconds ago Up 40 seconds 80/tcp powerstr df1a263e3059 zettio/weavedns:0.9.0 "/home/weave/weavedn 3 hours ago Exited (2) 3 hours ago weavedns 109e585d9c5f zettio/weave:0.9.0 "/home/weave/weaver 3 hours ago Up 3 hours 0.0.0.0:6783->6783/tcp, 0.0.0.0:6783->6783/udp weave
cd704996cac1 binocarlos/wait-for-weave:latest "/home/weavewait/wai 7 hours ago Exited (0) 7 hours ago weavewai root@7367145d2843:/usr/local/bin#

will any other logs /info help to troubleshoot further?

by the way thanks for quick replys :)

sbhaskaran commented 9 years ago

where can I get docker client only binaries? also does that means if the docker version is upgraded powerstrip also will need to be updated to use with newer version?

binocarlos commented 9 years ago

Hey right - I'm an idiot - I've just worked out where the 1.16 is coming from:

link here

The docker cli has no way of doing a request to `/images/*' and so I've had to do these requests over HTTP - I had erroneously left 1.16 in the version number.

I've downgraded this to 1.15 and added a note to the Readme explaining that you need a docker server API of at least 1.15

To get a client only docker binary - you compile docker from source with some flags set - I have a script for this here

The reason to include a client only binary in the container was that on fedora it needed devicemapper to be dynamically linked - making the docker a client only version meant that this was not a problem anymore.

So - do a docker pull binocarlos/powerstrip-weave and it should pull some layers and then try again and fingers crossed the version problem has gone away.

The powerstrip adapter can stay the same if docker updates its api - the key is that the client only binary (and node.js HTTP requests) hit an api version less than the installed docker server.

This means the docker server can get bumped to like 1.25 and we still hit 1.15 from our client binary + HTTP requests.

Let me know how it goes :-)

sbhaskaran commented 9 years ago

great @binocarlos it solved the version issue. I did launch a mysql container and it got the IP. I will further experment with with other images and see how it goes. Does

thanks for all your support. Appreciate it.

sbhaskaran commented 9 years ago

I tried to to run the power binocarlos/powerstrip-weave-example:lates and it failed saying

docker logs powerstrip

015-03-26 12:01:20+0000 [HTTP11ClientProtocol,client] while running chain Traceback (most recent call last): File "build/bdist.linux-x86_64/egg/treq/client.py", line 44, in connectionLost

      File "build/bdist.linux-x86_64/egg/treq/content.py", line 35, in connectionLost

      File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 382, in callback
        self._startRunCallbacks(result)
      File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 490, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
        raise ValueError("No JSON object could be decoded")
    exceptions.ValueError: No JSON object could be decoded

2015-03-26 12:01:20+0000 [HTTP11ClientProtocol,client] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7f3474206ef0>

same happening with my images too. I was able to run them wit weave binary.

But the mysql is running fine CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c29ebf489df2 mysql:5 "/entrypoint.sh mysq 21 minutes ago Up 21 minutes 3306/tcp mysql1

f7780252cffa mysql:5 "/entrypoint.sh mysq 52 minutes ago Up 52 minutes 3306/tcp mysql

Is there any particular requirement for the image to run on powerstrip-weave?

@binocarlos

binocarlos commented 9 years ago

@sbhaskaran hey - there isn't any particular requirement of an image to run no

Can you paste the results of docker logs powerstrip-weave?

sbhaskaran commented 9 years ago

@binocarlos It does not print any logs . Probably I will have to run in debug mode? Pasting the out put for both weave and powerstrip-weave. will try to run in debug mode and update.

docker logs powerstrip-weave 7b7057b942a5bd8dff1c4d6dbabc99642516ef32943a10193e6f703533c3086d server listening on port: 80

docker logs weave weave 2015/03/26 11:04:35.694597 Command line options: map[iface:ethwe name:7a:a2:5c:d7:a9:7f wait:20] weave 2015/03/26 11:04:35.694627 Command line peers: [] weave 2015/03/26 11:04:36.704133 Communication between peers is unencrypted. weave 2015/03/26 11:04:36.704208 Our name is 7a:a2:5c:d7:a9:7f weave 2015/03/26 11:04:36.730395 Sniffing traffic on &{66 65535 ethwe ce:55:f6:72:a3:e6 up|broadcast|multicast} weave 2015/03/26 11:04:36.730570 Discovered our MAC ce:55:f6:72:a3:e6 weave 2015/03/26 11:04:37.353211 Discovered local MAC 1e:da:85:14:f4:32 weave 2015/03/26 11:04:37.642719 Discovered local MAC 7a:a2:5c:d7:a9:7f weave 2015/03/26 11:11:09.819217 Discovered local MAC 06:e9:0e:ab:65:5f weave 2015/03/26 11:14:36.731733 Expired MAC ce:55:f6:72:a3:e6 at 7a:a2:5c:d7:a9:7f weave 2015/03/26 11:15:36.731986 Expired MAC 1e:da:85:14:f4:32 at 7a:a2:5c:d7:a9:7f weave 2015/03/26 11:21:36.738377 Expired MAC 06:e9:0e:ab:65:5f at 7a:a2:5c:d7:a9:7f weave 2015/03/26 11:41:33.271307 Discovered local MAC 7a:c6:ed:b6:74:1f weave 2015/03/26 11:46:18.378243 Discovered local MAC 66:5f:f9:49:e6:50 weave 2015/03/26 11:51:36.750298 Expired MAC 7a:c6:ed:b6:74:1f at 7a:a2:5c:d7:a9:7f weave 2015/03/26 11:56:36.751278 Expired MAC 66:5f:f9:49:e6:50 at 7a:a2:5c:d7:a9:7f

sbhaskaran commented 9 years ago

This is what I saw in debug mode

Cannot start container a69e0500fa1cec4db0d6a1ec5ac49809cfc3cc74f47ab5d4a5377503d873ed5b: exec: "/home/weavewait/wait-for-weave": stat /home/weavewait/wait-for-weave: no such file or y

where is it looking for the file any idea?

Version: 1 Type: pre-hook Request:

Method:      POST
URL:         /v1.15/containers/create

{ "Hostname": "", "Domainname": "", "User": "", "Memory": 0, "MemorySwap": 0, "CpuShares": 0, "Cpuset": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "PortSpecs": null, "ExposedPorts": {}, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "WEAVE_CIDR=10.255.0.51/8" ], "Cmd": [ "bash", "/srv/app/run.sh", "hello", "world" ], "Image": "binocarlos/powerstrip-weave-example", "Volumes": {}, "WorkingDir": "", "Entrypoint": [ "/home/weavewait/wait-for-weave" ], "NetworkDisabled": false, "OnBuild": null, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LxcConf": [], "Privileged": false, "PortBindings": {}, "Links": null, "PublishAllPorts": false, "Dns": null, "DnsSearch": null, "ExtraHosts": null, "VolumesFrom": [ "weavewait:ro" ], "Devices": [], "NetworkMode": "bridge", "CapAdd": null, "CapDrop": null, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "SecurityOpt": null } }

Version: 1 Type: post-hook Request:

Method:      POST
URL:         /v1.15/containers/11804864616c765731b6bf9f169bec272ff865d5d4ab129c7c7420d7fa885ad9/start

{ "Binds": null, "ContainerIDFile": "", "LxcConf": [], "Privileged": false, "PortBindings": {}, "Links": null, "PublishAllPorts": false, "Dns": null, "DnsSearch": null, "ExtraHosts": null, "VolumesFrom": null, "Devices": [], "NetworkMode": "bridge", "CapAdd": null, "CapDrop": null, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "SecurityOpt": null }

Response: }

Response:

ContentType: text/plain; charset=utf-8
Code:        500

Cannot start container 11804864616c765731b6bf9f169bec272ff865d5d4ab129c7c7420d7fa885ad9: exec: "/home/weavewait/wait-for-weave": stat /home/weavewait/wait-for-weave: no such file or y

and the docker run will be stuck and had to kill though another seession. that too kill -9. something fishy?

binocarlos commented 9 years ago

Hi - so, its failing to find /home/weavewait/wait-for-weave - this is defo a problem and will cause the container to die (as you can see from the log)....

The idea with powerstrip-weave is that weave attach runs after the container has started. This means there is a period of time during which the process is running but there is no weave network.

powerstrip-weave rewrites the entrypoint of the container to say wait for the weave network to be connected and then run the process

It does this by changing the Entrypoint to be:

"Entrypoint": [
"/home/weavewait/wait-for-weave"
],

and the Cmd (the args passed to the entrypoint) to be:

"Cmd": [
"bash",
"/srv/app/run.sh",
"hello",
"world"
],

wait-for-weave is a statically compiled golang binary that waits for the weave network to be connected before executing any of its arguments as a command.

So wait-for-weave bash /srv/app/run.sh hello world will run the command bash /srv/app/run.sh hello world after the weave network is connected.

The question is - where does it get access to the wait-for-weave binary?

If you notice the adapter has added a --volumes-from argument to the container create step:

"VolumesFrom": [
"weavewait:ro"
],

This means the container can now access /home/weavewait/wait-for-weave because its made available from the weavewait container (the one that exists immediately) - its only job is provide other containers with a --volumes-from to access the wait-for-weave binary.

So - question - can you see the weavewait container when you do docker ps -a? It wont be running but should be present.

If you can then for some reason it is not being made available to our container. Can you try something like this:

$ docker run -ti --rm --volumes-from=weavewait:ro ubuntu /bin/bash
12345$ ls -la /home/weavewait

You should see the wait-for-weave binary in that folder?

Thanks!

sbhaskaran commented 9 years ago

running docker run -ti --rm --volumes-from=weavewait:ro -e "WEAVE_CIDR=10.255.60.79/24,10.255.0.1/8" ubuntu /bin/bash works fine.

I did a fresh install of every thing including OS and with out using --volumes-from=weavewait I got everything working fine.

huge thanks for supporting this case and sharing your knowledge. @binocarlos . It was pleasure working with you.

sbhaskaran commented 9 years ago

Closing this case as issue is solved.