Twitter Accounts Source Integration - $250 #33

Closed evgenydmitriev closed 3 years ago

In GitLab by @evgenydmitriev on Aug 17, 2019, 20:20

Bounty Description

This task is for creating a CDC (Content Delivery Chain) Source that regularly collects, normalizes, and forwards the data to Nakamoto Terminal’s pipeline.
This source module will be used to improve agent data collection for Yupana (see internal source-integration issue)
Following this guide, you should modify the "twitter-account-task" project that was created specifically for this bounty issue to include the required functionality described below.
Please make sure your scraping method is robust and can withstand slight changes in the page layout.
When done, you submit your work by setting a deployment::ready tag in your merge request.

Functionality

Agent Metadata

ID
Twitter Name
Display Name
Verified Account (true/false)
Account Age
Location
Bio (with any linked social media accounts)
Number following/followed by
Any additional files if available
Lists of accounts following (we don't need the followers) - relationship information should probably send it as separate events

Resources

Other Agent Sources: medium
Starting list of Twitter accounts and agent tags
Current NLP module documentation
Guide on creating a CDC (Content Delivery Chain) Source
Bounty Program Wiki
Inca Digital Securities website
NTerminal website
Twitter API tools (Python implementation)

General rules

When you start working on the issue, please comment below to let everyone know that there might be potential competition.
When you are ready to submit your work, leave a comment in this issue with the link to your document and we'll get in touch with you regarding the bounty release or to help you make necessary changes to your submission.
We will pay the bounty as soon as we get a good quality submission that fulfills all of the requirements listed here.
By completing this project you agree to let Inca use any and all work submitted for any internal or external purposes. Inca reserves the right to use or not use any work submitted via this project.
For additional information about the Bounty program, please refer to our wiki page.

Background

Inca often uses "bounty projects" as introductory projects to vet potential employees or interns. These projects give interested individuals a chance to prove themselves, learn a bit about our company & products, and produce a useful result in the process. These projects are extremely independent and will require you to manage your own time and work process.

NTerminal is a data aggregation and analytics platform used for navigating the crypto-financial ecosystem. NTerminal's many data streams can be categorized into three general segments:

Financial data - Trade and order book data from exchanges and aggregation entities (price, size, trade-pair, volumes, etc.)
Natural Language data - Text-based data streams with keyword & sentiment analysis (social media posts, news articles, regulatory meeting minutes, etc.)
Technical data from Blockchains, mining pools, code fuzzing, github repositories, etc.

Don't hesitate to ask us questions by commenting in this issue or emailing us at bounty@incasec.com.

In GitLab by @evgenydmitriev on Aug 17, 2019, 20:28

@myrmecophagous add all your scripts/projects related to Twitter profiles in the Resources section

In GitLab by @myrmecophagous on Aug 18, 2019, 24:10

changed the description

In GitLab by @myrmecophagous on Aug 18, 2019, 24:10

Done

In GitLab by @evgenydmitriev on Aug 21, 2019, 01:42

assigned to @MarioIshac

In GitLab by @zfinzi on Aug 27, 2019, 05:14

Event Schemas

You will need to follow our internal documentation for how we structure Agent events. Here is the standard structure entitled ABM Agent.

You only need to include fields that apply to the source you are developing.
- For example you will need to add any location data provided by the Twitter API to the location field. However you will not need to add a revenues field as this will not apply to the Twitter API.
- For all additional fields that are specific to the Twitter API you can follow the schema I have outlined called ABM Twitter Account. This will be passed into the field agent_specific_attributes.
- You will need to add an agent_type field which should be either person or organisation based on the account.
- username can go under aliases and agent_specific_attributes.username and display name can go under full_name

For one twitter account following another, you will need to follow the ABM Relation schema as this will be a separate event.

Ping me if there are any questions about the schemas or any thoughts you may have.

In GitLab by @anshlykov on Sep 2, 2019, 19:29

changed the description

In GitLab by @anshlykov on Sep 2, 2019, 19:30

unassigned @anshlykov

In GitLab by @MarioIshac on Sep 8, 2019, 03:47

@zfinzi For Account Age under metadata, is that represented as the account's date of creation or a certain amount of time units from that date? If it's the latter, which time unit should I use?

In GitLab by @zfinzi on Sep 8, 2019, 04:53

@MarioIshac whichever is provided by Twitter API, if they provide a value in units of time for how old the account is then keep the field name as is. If the API provides a date of creation, then switch the field name to creation_date.

Let me know which one it is so that I can update spotlight documentation.

In GitLab by @MarioIshac on Sep 8, 2019, 09:50

The Twitter API provides a creation date, I'll switch the field to creation_date on my end as well.

In GitLab by @zfinzi on Sep 8, 2019, 10:01

Thanks, Stoplight schema has been updated.

In GitLab by @MarioIshac on Sep 9, 2019, 24:54

Update: This will take me a bit longer to finish (~2 days) because Twitter requires OAuth authentication even when accessing public data. I'll have to incorporate that into the trigger configuration's mapping rule.

In GitLab by @evgenydmitriev on Sep 18, 2019, 06:05

any progress?

In GitLab by @MarioIshac on Sep 18, 2019, 06:13

Still have to figure out why the GitLab CI configuration is invalid, after that I can assign the merge request.

1712n / challenge