Moving from deprecated nats repository go-nats-streaming to stan.go

matthiashanel commented 4 years ago

Description

This allows use of a more recent version of the nats-streaming client. The change renames the package in go files and Gopkg.toml. Also includes corresponding changes in Gopkg.lock and vendor directory. Fixes #87

Motivation and Context

[x] I have raised an issue to propose this change (required)

How Has This Been Tested?

I ran some of the tests using minikube. Connecting to minikube docker eval $(minikube -p minikube docker-env) queueworker-dep.yml has been altered to use the local image:

        image: openfaas/queue-worker:latest-amd64
        imagePullPolicy: Never

I ran unit tests in this repo without issues.
Tested if the new version of the library is being used
- in a checkout of faas-netes modified nats-dep.yaml to expose the monitoring endpoint by adding -m=8222 to nats-streaming-server args
- forwarded the port kubectl port-forward nats-5b46fc6cf9-qb8g4 8222:8222 -n openfaas &
- used curl curl http://localhost:8222/connz to obtain the connection's client version "name": "faas-worker-queue-worker-769cdb957b-2jjbs", ... "version": "1.9.2"
Tested that async function invocation is still working
- Deployed cows function
- invoked it asynchronously: curl http://127.0.0.1:31112/async-function/cows -d "{}"
- Before and after invocation Inspected logs 'kubectl logs cows-78f4dccc84-prf2n -n openfaas-fn' and made sure that to observe a new Forking fprocess. line.

Relevant curl output (please not faas-publisher-gateway... version will change once if references a newer version of nats-queue-worker)

curl http://localhost:8222/connz
Handling connection for 8222
{
  "server_id": "NDUNEVJZU5AHFV5AAWEYQP6YKSU547I3BHXOTJEQM67LMV27CXLBG5NK",
...
  "connections": [
   ...
    {
...
      "in_bytes": 8844,
      "out_bytes": 1373,
      "subscriptions": 5,
      "name": "faas-worker-queue-worker-769cdb957b-2jjbs",
      "lang": "go",
      "version": "1.9.2"
    },
    {
...
      "in_msgs": 419,
      "out_msgs": 419,
      "in_bytes": 9999,
      "out_bytes": 292,
      "subscriptions": 4,
      "name": "faas-publisher-gateway-55d6db96cd-zrcwh",
      "lang": "go",
      "version": "1.7.2"
    }
  ]
}

Relevant output for async function invocation.

> kubectl port-forward svc/gateway -n openfaas 31112:8080 &
Forwarding from 127.0.0.1:31112 -> 8080
Forwarding from [::1]:31112 -> 8080
> kubectl port-forward svc/gateway -n openfaas 31112:8080 &Handling connection for 31112
Handling connection for 31112
> kubectl logs cows-78f4dccc84-prf2n -n openfaas-fn | grep Forking | wc -l
       8
> curl http://127.0.0.1:31112/async-function/cows -d "{}"
Handling connection for 31112
> kubectl logs cows-78f4dccc84-prf2n -n openfaas-fn | grep Forking | wc -l
       9
>

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I've read the CONTRIBUTION guide
[x] I have signed-off my commits with git commit -s
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

alexellis commented 4 years ago

Hi, I missed this PR but it's on my radar now. Thank you for contributing 👍

Alex

alexellis commented 4 years ago

Hi @matthiashanel, I merged this PR and then ran dep ensure, but afterwards the build failed:

main.go:19:2: cannot find package "github.com/nats-io/stan.go" in any of:
        /Users/alex/go/src/github.com/openfaas/nats-queue-worker/vendor/github.com/nats-io/stan.go (vendor tree)
        /usr/local/go/src/github.com/nats-io/stan.go (from $GOROOT)
        /Users/alex/go/src/github.com/nats-io/stan.go (from $GOPATH)

Could you advise?

Gopkg.toml now looks like:


[[constraint]]
  name = "github.com/nats-io/stan.go"
  version = "0.6.0"

[[constraint]]
  name = "github.com/openfaas/faas"
  version = "0.10.2"

[prune]
  go-tests = true
  unused-packages = true

[[constraint]]
  name = "github.com/openfaas/faas-provider"
  version = "0.7.0"

matthiashanel commented 4 years ago

Hey @alexellis, I can't seem to reproduce the issue. Where did you run the build? In docker as part of make?

I checked out the current version: It built with makefile -> worked dep ensure basically changed the reference to pick up the most recent checkin. built again -> worked. Reverted changes and checked out my commit. (just to make sure your later changes didn't fix it) It build with makefile without problems Then I did a bo build without problems. To be sure my version had the right dependencies I ran dep ensure again, no changes.

> git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
> git log | grep Hanel
Author: Matthias Hanel <mh@synadia.com>
    Signed-off-by: Matthias Hanel <mh@synadia.com>
> make build | tail
 ---> Using cache
 ---> ab983e70389b
Step 23/24 : COPY --from=golang /go/src/github.com/openfaas/nats-queue-worker/app    .
 ---> Using cache
 ---> 819ed76b59fc
Step 24/24 : CMD ["./app"]
 ---> Using cache
 ---> e564ba1e3f9f
Successfully built e564ba1e3f9f
Successfully tagged openfaas/queue-worker:latest-ppc64le
> dep ensure
> git diff
diff --git a/Gopkg.lock b/Gopkg.lock
index 180a16d..568ebb9 100644
--- a/Gopkg.lock
+++ b/Gopkg.lock
@@ -76,6 +76,14 @@
   revision = "9ce928bc82cbb2642e6d534f93a7904116179e6c"
   version = "0.7.0"

+[[projects]]
+  digest = "1:2165477f55a57e56283780cfb43604682766219d0e316277f436e3d6b6328912"
+  name = "github.com/openfaas/nats-queue-worker"
+  packages = ["nats"]
+  pruneopts = "UT"
+  revision = "1f4e16e1f7afe1fbd464fcaa8a7dabaa3e4ef0bb"
+  version = "0.10.1"
+
 [[projects]]
   branch = "master"
   digest = "1:cd7e85fc3687e062714febdee3e8efeb00a413a2a620d28908fd0258261d2353"
@@ -94,6 +102,7 @@
     "github.com/nats-io/stan.go",
     "github.com/openfaas/faas-provider/auth",
     "github.com/openfaas/faas/gateway/queue",
+    "github.com/openfaas/nats-queue-worker/nats",
   ]
   solver-name = "gps-cdcl"
   solver-version = 1
> make build | tail
 ---> Using cache
 ---> ab983e70389b
Step 23/24 : COPY --from=golang /go/src/github.com/openfaas/nats-queue-worker/app    .
 ---> d3560d0aca7a
Step 24/24 : CMD ["./app"]
 ---> Running in d64a15f65523
Removing intermediate container d64a15f65523
 ---> e7b9cad5d499
Successfully built e7b9cad5d499
Successfully tagged openfaas/queue-worker:latest-ppc64le
> git log | head
commit 1f4e16e1f7afe1fbd464fcaa8a7dabaa3e4ef0bb (HEAD -> master, tag: 0.10.1, origin/master, origin/HEAD)
Author: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
Date:   Wed Apr 22 12:42:15 2020 +0100

    Update queue types

    Includes QueueName

    Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>

> git co Gopkg.toml
Updated 1 path from the index
> git co Gopkg.lock
Updated 1 path from the index
> git co 4cdd2b89da7b8d3692f3d112a81b4b1f8a996808
Note: checking out '4cdd2b89da7b8d3692f3d112a81b4b1f8a996808'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 4cdd2b8 Moving from deprecated nats repository go-nats-streaming to stan.go
> make build | tail
 ---> Using cache
 ---> 97be8b6e0b0b
Step 22/23 : COPY --from=golang /go/src/github.com/openfaas/nats-queue-worker/app    .
 ---> d61d6185e946
Step 23/23 : CMD ["./app"]
 ---> Running in 90ff4df3a68b
Removing intermediate container 90ff4df3a68b
 ---> 0608c0f278fe
Successfully built 0608c0f278fe
Successfully tagged openfaas/queue-worker:latest-ppc64le
> git log | head -n 14
commit 4cdd2b89da7b8d3692f3d112a81b4b1f8a996808
Author: Matthias Hanel <mh@synadia.com>
Date:   Tue Mar 31 16:14:02 2020 -0400

    Moving from deprecated nats repository go-nats-streaming to stan.go

    Fixes #87
    This allows use of a more recent version of the nats-streaming client.
    The change renames the package in go files and Gopkg.toml.
    Also includes corresponding changes in Gopkg.lock and vendor directory.

    Signed-off-by: Matthias Hanel <mh@synadia.com>

commit 3bffe0ddab520bb410363cd885f880ef9e804d51
> go build
> dep ensure
> git diff
>
>cat Gopkg.toml | tail -n 15[[constraint]]
  name = "github.com/nats-io/stan.go"
  version = "0.6.0"

[[constraint]]
  name = "github.com/openfaas/faas"
  version = "0.10.2"

[prune]
  go-tests = true
  unused-packages = true

[[constraint]]
  name = "github.com/openfaas/faas-provider"
  version = "0.7.0">

alexellis commented 4 years ago

I got around this by running go get, perhaps some kind of local conflict between Go modules and dep?

matthiashanel commented 4 years ago

most likely

openfaas / nats-queue-worker