Develop IRC stream source for Spring Cloud Dataflow streams - 500$ #4

Closed evgenydmitriev closed 3 years ago

In GitLab by @penpyt on Jul 19, 2018, 15:24

Develop a ircstream source component, which can be easily integrated into Spring Cloud Dataflow streams.

Toolset: JAVA, Spring, Spring-dataflow, docker

If you want to lock this issue to make sure no one else is working on it, please comment below and send us your resume at careers@incasec.com. After your resume review, we'll add "in progress" tag and assign the issue to you. Upon request, we can also create an escrow job on one of the freelancer websites (Upwork, fl.ru, etc). All of this is optional - you can skip this step if you just want to show us the result.
You need to create a separate personal git project and provide the issue creator with access to your repository for code review.
Upon completion of the project, please add "release" branch, create merge request from master to release, assign the issue creator to it, and leave a comment here.
After resolving all our comments associated with the merge request, we'll release the payment, and move the project into our repository.

Component should

be dockerized.
communicate with a stream via RabbitMQ-broker.
be configured through spring-dataflow-dashboard GUI to receive initial input data: server, username, password, comma-separated list of channels.
yields new messages from those channels to self-output in [SCHEMA]() format.

following commands should be provided:

a command for building docker-image with the component;
a command for building application-metadata jar package companion.

Definition of done

all review discussions are closed.
docker-image with the component and the metadata-package are built successfully.

In GitLab by @durm on Sep 21, 2018, 17:35

changed title from {-irc-} to {+Develop IRC stream source for Spring Cloud Dataflow streams - 500$+}

In GitLab by @durm on Sep 21, 2018, 17:39

changed the description

In GitLab by @durm on Sep 21, 2018, 17:40

changed the description

In GitLab by @durm on Sep 21, 2018, 17:41

@kemi let's specify output format for ircsource too.

In GitLab by @durm on Sep 26, 2018, 13:59

changed the description

In GitLab by @aturok on Nov 7, 2018, 24:35

Work in progress here: https://github.com/aturok/ircstream

Status: runs in scdf, listens to IRC, spits messages to the stream, but: hardcoded parameters, no message structure, lots of things to refine and clean up, not dockerized

Next steps:

refine structure and Spring harness
define and wire-up parameters
define proper message structure

This one will be ready earlier than Tg

@durm could you please:

Confirm that we should have the following parameters here: IRC server, comma-separated list of channels, IRC credentials / nickname
Check whether we need anything special in IRC message structure when compared to Tg or I can simply use the Tg structure and adjust it where appropriate (e.g. IRC supports only u2u files exchange, which at the first glance makes media fields irrelevant here)
Provide the desired groupid (so that you don't have to mess with the package name)

In GitLab by @aturok on Nov 7, 2018, 22:15

@durm, I updated the repo. The thing is almost ready, just need to dockerize properly, consider spring-related improvements and ensure that the message structure and other details match your expectations.

Some questions to you in addition to the above ones:

Which kind of username do we want in author? Can chose at least from the login and the nickname of the user.
IRC doesn't have real message ids - they only plan to add it (see https://ircv3.net/specs/extensions/message-ids.html). Should we generate some unique uuid to substitute the message id?
In regards to docker, do we want to dockerize the entire project and its build process or just copy the packaged jar into it?

To help you deciding on the message schema, here's an example of a message that the app generates now:

{
    "id":"95",
    "date":"11/07/2018 19:59:50.055",
    "source":"IRC",
    "category":"#aturoktest",
    "channel_id":"19f30e0c-4e80-41b6-ad90-079281a44f42",
    "author":"~aturok",
    "content":"hello bots"
}

In GitLab by @evgenydmitriev on Nov 7, 2018, 23:17

Which kind of username do we want in author? Can chose at least from the login and the nickname of the user.

The one that the user can't change - probably login. We should still keep the nickname in some other field. @myrmecophagous where do you want it?

IRC doesn't have real message ids - they only plan to add it (see https://ircv3.net/specs/extensions/message-ids.html). Should we generate some unique uuid to substitute the message id?

@myrmecophagous do you need unique IDs created by a grabber? I don't really see the point.

In regards to docker, do we want to dockerize the entire project and its build process or just copy the packaged jar into it?

@durm @anshlykov

In GitLab by @anshlykov on Nov 7, 2018, 23:39

just copy the jar into the container.

In GitLab by @myrmecophagous on Nov 7, 2018, 23:58

@evgenydmitriev @aturok

Which kind of username do we want in author?

Let's keep both. I suggest putting login in author and nickname in user_name fields respectively.

do you need unique IDs created by a grabber?

I believe we need something to identify messages in a reliable way, something based on channel_id and date maybe?

In GitLab by @evgenydmitriev on Nov 8, 2018, 01:32

I believe we need something to identify messages in a reliable way, something based on channel_id and date maybe?

@myrmecophagous I believe we are going to have both channel name and timestamp that will help you create a unique fingerprint for each message

In GitLab by @myrmecophagous on Nov 8, 2018, 13:30

@evgenydmitriev True

In GitLab by @aturok on Nov 12, 2018, 21:09

@penpyt , @durm Folks, the merge request arrived - please review: https://gitlab.com/aturok/ircstream/merge_requests/1

I added just the two of you to the repo - please let me know if I need to add anyone else.

@evgenydmitriev @myrmecophagous I retained the id field in the message for now - it's the channel name and timestamp concatenated together. Please tell me if I should better drop it.

In GitLab by @anshlykov on Nov 13, 2018, 09:16

@aturok penput not working with us anymore. Please add me

In GitLab by @aturok on Nov 13, 2018, 13:35

@anshlykov done.

In GitLab by @evgenydmitriev on Nov 15, 2018, 14:50

@durm @anshlykov, what's up?

In GitLab by @durm on Nov 15, 2018, 15:04

@evgenydmitriev code-review is in progress in merge-request here

when @aturok will fix comments he will reassign merge-request to me or to @anshlykov

In GitLab by @durm on Nov 23, 2018, 19:20

closed

1712n / challenge

Develop IRC stream source for Spring Cloud Dataflow streams - 500$ #4