matryer / vice

Go channels at horizontal scale (powered by message queues)
https://medium.com/@matryer/introducing-vice-go-channels-across-many-machines-bcac1147d7e2
Apache License 2.0
1.54k stars 79 forks source link

Some messaging systems behave as work queues and others as pub/sub system. #19

Closed HeavyHorst closed 7 years ago

HeavyHorst commented 7 years ago

1: pub/sub (every receiver get the message):

2: work queue (only one receiver gets the message):

I think we should put all transport implementations in the second mode to mimic the default go channel behaviour.

GeorgeMac commented 7 years ago

I made a suggestion in #10 that you actually treat every call to Receive as a subscription. As it is trivial in Go to then take the channel and use it as a work queue by sharing that channel. Where as, if you treated Receive as work-queue like, then every consumer would get the same channel and wouldn't be able to choose to have subscription semantics. Well... without then copying each message and fanning out themselves.

HeavyHorst commented 7 years ago

At the moment every call to Receive("something") would get the same channel in every implementation. So if we are using just a single transport on a single node we have already work queue semantics.

But if we create more than one transport or let our program run on more than one node we get sometimes pub/sub and sometimes work queue behaviour.

Receiver.go

package main

import (
    "fmt"

    vn "github.com/matryer/vice/queues/redis"
)

func main() {
    nt := vn.New()
    defer nt.Stop()

    rc := nt.Receive("test123")

    for i := 0; i < 5; i++ {
        v := <-rc
        fmt.Println(string(v))
    }
}

Sender.go

package main

import (
    "fmt"
    "time"

    vn "github.com/matryer/vice/queues/redis"
)

func main() {
    nt := vn.New()
    defer nt.Stop()

    sc := nt.Send("test123")

    for i := 0; i < 5; i++ {
        sc <- []byte(fmt.Sprintf("%d Hallo", i))
        time.Sleep(1 * time.Second)
    }
}

Redis:

screenshot1

NSQ:

screenshot2

GeorgeMac commented 7 years ago

I suppose vice is intended to be a queue and not pub sub. So queue semantics by default makes sense over subscription.

HeavyHorst commented 7 years ago

I agree

matryer commented 7 years ago

Right, Vice is a queue. It's tough to think of a way to properly test that stuff; so I think we just have to be careful about it in the implementations.

Do we need to tweak anything here?

HeavyHorst commented 7 years ago

The nats implementation just needs to use QueueSubscribe instead of Subscribe. I'm sure redis can also be used as a work queue.

HeavyHorst commented 7 years ago

Solved with #29 and #28