search

mmmries / nats.ex_streaming

A NATS Streaming client for Elixir
3 stars 4 forks source link

Basic PUB/SUB functionality (cluster id option) #6

Closed costa closed 4 years ago

costa commented 4 years ago

I've seen a vague note on this project readiness and no description of a basic PUB/SUB functionality. I wonder if it is available already as I need to use NATS streaming with Erlang/Elixir.

Thank you.

costa commented 4 years ago

@mmmries Can you please tell what problems prevent this to be actually published (on hex), I might take a look?

mmmries commented 4 years ago

Hi @costa, sorry I didn't see this issue 6 days ago when you opened it. I think the basics of publishing and subscribing are working (see the test/end_to_end_test.exs file). I ended up not using this in a project at work and then heard about NATS Jetstream which will end up being the built-in way of handling persistent streams in NATS. So I haven't pushed this project forward mostly because I was going to wait for Jetstream to stabilize and then add support for that in the main nats.ex project

costa commented 4 years ago

Thanks @mmmries for your reply. If the basics work here, and there's no leaks, it's good enough for me, I'll appreciate it if you publish it to hex so the deps will work appropriately. I will check it right away — functionally at least — and will let you know, so you may take it down if it's no good at all.

Re: JetStream, yes, I've seen the project and heard from the guys, but "Technical Preview" readiness does not inspire confidence in me, let alone the fact I'll be in the same trouble of looking for the clients for the platforms I'm integrating. I've originally chosen NATS Streaming (apart from it being lightweight) because it can connect (most of the) popular language platforms, and unfortunately, I doubt I Jetstream will be a decent option for this soon.

mmmries commented 4 years ago

Hey @costa your selection of NATS streaming sounds like a solid choice for the requirements you are dealing with. I hope this project is useful to do along those lines. I won't be adding any significant resources to this project in the near-term, so if it's useful to fork this project and extend it please feel free.

I went ahead and downloaded a 0.0.1 version so you can test it out: https://hex.pm/packages/nats_streaming

mmmries commented 4 years ago

If it looks good for your usage I'll take a look at a couple of outstanding PRs and potentially release a 0.0.2 version.

costa commented 4 years ago

Great, thank you, however, right out of the box it yields

==> nats_streaming
Compiling 5 files (.ex)

== Compilation error in file lib/gnat/streaming/protocol.pb.ex ==
** (UndefinedFunctionError) function Gnat.Streaming.Protocol.StartPosition.key/1 is undefined (module Gnat.Streaming.Protocol.StartPosition is not available)
    Gnat.Streaming.Protocol.StartPosition.key(0)
    lib/protobuf/dsl.ex:33: anonymous fn/2 in Protobuf.DSL."MACRO-__before_compile__"/2
    (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3
    expanding macro: Protobuf.DSL.__before_compile__/1
    lib/gnat/streaming/protocol.pb.ex:170: Gnat.Streaming.Protocol.SubscriptionRequest (module)
could not compile dependency :nats_streaming, "mix compile" failed. You can recompile this dependency with "mix deps.compile nats_streaming", update it with "
mix deps.update nats_streaming" or clean it with "mix deps.clean nats_streaming"

within my elixir:1.9 container.

I'm going to see if your mix test works there.

costa commented 4 years ago

@mmmries to follow up, it compiles within that container successfully, runs and gets to :connectionrefused (which is expected and it is further than I got), so I fear something might be out of order with the package specification (note that it says gnats_streaming in README, for instance).

mmmries commented 4 years ago

Oh, that makes sense. I changed the name to nats_streaming right before I published it (the name gnat is a pun we put into the main client, but I think it makes more sense to keep the name simple. I forgot to change the rest of the naming structure of the library to match. I'll take a look at it tonight. If you want to test in the meantime, feel free to pull the code down locally and test it with {:nats_streaming, path: "../nats.ex_streaming"} in the meantime

costa commented 4 years ago

I haven't finished the integration attempts, but this is what I have to report so far:

mmmries commented 4 years ago

Hey @costa I just merged the two oustanding PRs that improved the options you can pass when setting up a subscription and I fixed all the naming in the library to use Nats.Streaming as the namespace so that it matches the nats_streaming library name on hexpm.

I also setup an example repo that uses the library that you can see here: https://github.com/mmmries/nats_streaming_example

Most of the work in in the application.ex module which starts the supervision tree. You can see the diff here: https://github.com/mmmries/nats_streaming_example/commit/765a6abf049985baf8710855dd1069edb0291ffb

The basic idea is to have a supervised connection, a supervised client and a supervised subscription. These are all supervised separately so that you can show how to share them within your application structure (ie maybe a single Gnat connection that is used for a mix of different kinds of work, or a connection per subscription).

Once these are setup you can start publishing messages and the subscription will your a function each time a message is received (it does this in a TaskSupervisor so each of them can crash or succeed independently)

costa commented 4 years ago

Thank you @mmmries, this sounds great, I will check it right away, although the cluster name issue is a suspicion still, considering https://github.com/mmmries/nats.ex_streaming/blob/e07f4f1990d84c39be8a6018ff80dddaaeb151ab/lib/nats/streaming/client.ex#L160

costa commented 4 years ago

@mmmries I have verified there's a problem with the hardcoded cluster id, when i switched my nats-streaming to the 'test-cluster' id, my code started working without changes (with your fixed-deps 0.0.2). I will conclude my primitive integration tests soon and see if there's anything else. Thank you.

costa commented 4 years ago

Hey @mmmries, I'm happy to have just completed the functional testing of my integration and I've found no other issues this far. I'm "polishing" the solution still wrt supervision and client (seq) persistence, but I see no substantial complications with those. I will make this my project code available to you later on, if interested. Now I've not been familiar with NATS internals and I'm not sure the cluster-id is a NATS or NATS-Streaming thing, so, please advise, if it's just here, I might be able to come up with a PR. Thanks.

mmmries commented 4 years ago

🎉 that's great to hear @costa! I think think that the cluster ID is only needed in that one place within the client.ex module. I did a quick test locally of starting the nats-streaming-server -cid booyah to run the streaming servers with a cluster id of "booyah" and then I changed the end_to_end_test.exs to pass [connection_name: :streaming_connection, cluster_id: "booyah"] and it looks like everything was working as expected.

Do you want to take a quick look at #7 and see if that looks good for you? Or you can even point your mix deps at it {:nats_streaming, github: "mmmries/nats.ex_streaming", branch: "configurable_cluster_id"} and test it out with your setup.

costa commented 4 years ago

@mmmries Yes, the branch worked perfectly for me and it can definitely close this issue.

I thank you for your support and, since it's on a major data flow of my integration, I will continue covering it with system/load tests. Once I reach a certain project system maturity, I will share the use case with pleasure. Have a nice weekend.

mmmries commented 4 years ago

I released the change for a configurable cluster_id as 0.0.3. I hope your project goes well!