project-codeflare / rayvens

Rayvens makes it possible for data scientists to access hundreds of data services within Ray with little effort.
Apache License 2.0
43 stars 7 forks source link

HTTP/WS Server as source? #78

Closed autolyticus closed 1 year ago

autolyticus commented 1 year ago

Hi! First of all, thanks for open-sourcing this project. It's super cool! I feel really fortunate running across this project as I was planning to work on something very similar, and I hadn't realised what a complex undertaking it was going to be, so I feel relieved that something like what I envisioned is already out there.

I've just been playing around with it and I had a naive question regarding how to set up an endpoint and to use Rayvens as a server (ws:// or wss:// preferred, but http is fine as well) for listening for publishers from outside the cluster. For eg. Ray provides the Ray serve API which can be used to set up routes and pass on parameters to other actors.

If I'm understanding correctly, it seems that Camel-K itself comes with support for a websocket/http server with components, however, my naive approach at trying to implement this as a generic source didn't quite work out

source_config = dict(kind='generic-source',
                     uri="websocket://0.0.0.0:9292")
(StreamActor pid=572) [Kamel subprocess] Caused by: org.apache.camel.NoSuchEndpointException: No endpoint could be found for: websocket://0.0.0.0:9292, please check your classpath contains the needed Camel component jar.

I'm sure I must be missing something, but even if the code was correct, does rayvens come with support for exposing ports/services based on this as an ingress resource? Or do I have to manually configure this on k8s?

Is there a recommended way to set up a server for listening to events other than polling as a client? Is this even supported or am I barking up the wrong tree?

autolyticus commented 1 year ago

Whoops, opened a duplicate due to network problems. Duplicate of https://github.com/project-codeflare/rayvens/issues/77