POC for Webhook Integration with GitHub or Another Service Using QWATCH

pratikpandey21 commented 2 months ago

Description

Create a Proof of Concept (POC) demonstrating the integration of DiceDB with GitHub or another external service using webhooks to showcase real-time reactivity with the QWATCH command. The POC should be a concrete example illustrating how to set up and use DiceDB to receive and process webhook events, leveraging QWATCH to push client updates based on changes to the resultset triggered by these events.

POC Requirements:

Integration with GitHub/Another Service that supports Webhook integrations:

Set up a webhook to listen for specific events (e.g., push events, pull requests) from GitHub or another service.
Configure DiceDB to receive these webhook events(using JSON.INGEST HTTP Endpoint) and store or process the incoming data appropriately.

Demonstrate Real-Time Reactivity with QWATCH:

Use the QWATCH command to monitor changes in the data stored by the webhook events.
Push real-time updates to clients whenever the results of a query affected by these events change.

Example Placement and Documentation:

Place the POC in the examples directory of the repository.
Provide a comprehensive README file that includes:
1. Step-by-step instructions for setting up the integration.
2. Configuration details (e.g., setting up webhooks in GitHub, configuring DiceDB).
3. A description of the example, its use case, and how to run it.
4. Similarity to the leaderboard-go example to maintain consistency in style and clarity.

pratikpandey21 commented 2 months ago

Depends on #552 and #553

aashraybhandar1 commented 2 months ago

can i pick this up

lucifercr07 commented 1 month ago

@aashraybhandar1 assigned, thanks for contributing.

lucifercr07 commented 1 month ago

@aashraybhandar1 Dependent issues are closed now, please let us know if any details required.

aashraybhandar1 commented 1 month ago

Hey @lucifercr07 sorry was on vacation, just catching up on stuff. So as per my understanding

We need to integrate github with Dice via webhooks
Use QWATCH to monitor changes to the data stored via the webhooks
Push real time updates in case change happens to the above mentioned data

Will this be a correct understanding, that the role of this demo will just be try to push the output of the QWATCH command back via the webhook.? A couple questions

Will we need to support multiple webhooks?
Do we need to setup some kind of persistence for the same or an inmemory operation mapping the multiple webhooks to the corresponding QWATCH would be sufficient

aashraybhandar1 commented 1 month ago

@lucifercr07 @pratikpandey21 would appreciate feedback on the previous comment. Thank You!

pratikpandey21 commented 1 month ago

Apologies for the delay @aashraybhandar1 .

Yes, your understanding is correct. If we can find a better application for webhook integration than GitHub, we can choose that as well.

Just to highlight, push real time updates is something that should be handled already by QWATCH.

And no persistence needed and one webhook should be enough for the demo.

arpitbbhayani commented 1 month ago

Hello @aashraybhandar1,

There has been no activity on this issue for the past 5 days. It would be awesome if you keep posting updates to this issue so that we know you are actively working on it.

We are really eager to close this issue at the earliest, hence if we continue to see the inactivity, we will have to reassign the issue to someone else. We are doing this to ensure that the project maintains its momentum and others are not blocked on this work.

Just drop a comment with the current status of the work or share any issues you are facing. We can always chip in to help you out.

Thanks again.

aashraybhandar1 commented 1 month ago

Hey @arpitbbhayani working on it haven't gotten much time yet but looking to make good progress over the weekend

aashraybhandar1 commented 1 month ago

Hey @lucifercr07 @pratikpandey21 so I was first trying to see how the QWATCH command works over HTTP. This is the steps I followed

Issued this curl curl --location 'http://localhost:8082/qwatch' \ --header 'Content-Type: application/json' \ --data '{"query": "SELECT $key, $value WHERE $key like '\''match:100:*'\'' and $value > 10 ORDER BY $value desc LIMIT 3"}' which set up a long lived HTTP connection in my postman
Issued the following command via CLI SET match:100:user:3 118

I see no response in my postman console when I follow this steps, I was expecting this value to be displayed there. Is my understanding incorrect? Thanks

lucifercr07 commented 1 month ago

@aashraybhandar1 try below once.

curl --location 'localhost:8082/QWATCH' \                      
--header 'Content-Type: application/json' \
--data '{
    "query": "SELECT $key, $value WHERE $key LIKE \"match:100:*\" AND $value > 100 ORDER BY $value DESC LIMIT 3"
}' --no-buffer

aashraybhandar1 commented 1 month ago

I have been exploring options for webhook integrations,

Github only has webhooks associated with actions against a repo. I don't know if using that workflow suits our usecases.
I tried exploring tools like IFTT,Zapier etc all have some kind of paid version for webhooks.
Found this tool call pipedream which allows building a webhook workflow for free. But currently when I process push events and then try to make a call to my localhost to the json.ingest endpoint to insert a value, the requests time out.

Do we explicitly need to set up a third party webhook? Can we explore the option of setting up a lightweight node server where we push events to. Does the below flow sound like something which would match the goals for this POC

Set up a long lived HTTP connection to listen to the QWATCH updates
Start up a node server and send sporadic requests to it updating data which the server processes and then calls dice db server with the json.ingest endpoint
Receive the updated value on our long lived HTTP connection and display the results on the terminal

@pratikpandey21 @lucifercr07 . Looking forward to your comments. Thank You!

aashraybhandar1 commented 1 month ago

@pratikpandey21 @lucifercr07 aplogies for the tag, doing incase you missed my previous comment. Thank You!

lucifercr07 commented 1 month ago

@aashraybhandar1 have you looked in Trello/Slack webhooks?

arpitbbhayani commented 3 weeks ago

Hello @aashraybhandar1,

There has been no activity on this issue for the past 5 days. It would be awesome if you keep posting updates to this issue so that we know you are actively working on it.

We are really eager to close this issue at the earliest, hence if we continue to see the inactivity, we will have to reassign the issue to someone else. We are doing this to ensure that the project maintains its momentum and others are not blocked on this work.

Just drop a comment with the current status of the work or share any issues you are facing. We can always chip in to help you out.

Thanks again.

DiceDB / dice