Twitter Connector

[srenault-meeds]

Rationale

As we aim to provide gamification connectors, we need to add new connectors to the current one (meeds, github). Twitter connector seems to be the most relevant at the time so organizations can boost their project / employee advocacy -->

1. Functional Requirements

Top User Stories

As an admin, I can access the twitter connector from the page:

Then I have the list of events proposed for Twitter gamification connector:

to study

• Follow
• mention
• like
• rt
• repost
• tweet with a given hashtag
• account with x followers

To enable this connector, I would need to do as for any other connector (i.e GitHub):

• Enable profile connection to their account
• Add an account to watch

Impacts

NA

Gamification

New connector

Notifications

NA

Analytics

Usage of connector:

• Which event is triggered and rate of usage

Monitoring

• In case of error, let know the admin the error type, the response code and the duration
• To follow rates limitation, let know for a period number of requests for each event so the admin can monitor if if needed / OR proceed to get another plan

Unified Search

NA

2. Technical Requirements

Expected Volume & Performance

Terminology

• The same as described in Github connector

Security

• A Twitter oAuth ConsumerKey & ConsumerSecret key has to be provided by rewarding admin in a dedicated UI (General Settings Tab, for example). Those oAuth Keys have to be encrypted using CodecInitializer before storing them in the database. Those Keys will be used to validate the oAuth Token provided by the user from the Meeds Server to the Twitter Server.

• The generation of those API Keys has to be made manually by the Rewarding Administrator for now (even though we can make a UI to simplify its generation). In case, we want to facilitate the generation of those oAuth keys, a "Browser-To-Twitter" flow has to be used without implying Meeds server in the flow (using oAuth JS client-specific library).

• A Twitter API AccessToken & AccessTokenSecret key has to be provided by rewarding admin in a dedicated UI (General Settings Tab, for example). Those API Keys have to be encrypted using CodecInitializer before storing them in the database.

Extensibility

• The same as described in Github connector

Configurability

• The same as described in Github connector

Upgradability

NA

Existing Features

NA

Feature Flags

No feature flag is needed.

4. Software Architecture

Security

• The constant polling from the Twitter API should done every 15 min to check every available meeds event using AccessToken & AccessTokenSecret

• The Twitter API AccessToken & AccessTokenSecret has to be encrypted using CodecInitialize before storing it in the database using SettingService with Global context and TwitterApiKey Application Scope.

• The Twitter oAuth ConsumerKey & ConsumerSecret using CodecInitialize before storing it in the database using SettingService with User context and Twitter Application Scope.

REST endpoints

A rest endpoint to add for the management of the watched account by the rewarding admin

• /gamification/connectors/twitter/account

Access

All administration UIs have to be made extensible (injected using extensionRegistry) so that it prepares integration with future connectors.

Services & processing

• A periodic service that fetches data from the Twitter API every 15 minutes and then sends it as triggers.

• The trigger handling has to be made asynchronously through a queued executor. The queue has to hold the objects retrieved from Twitter API without parsing data.

• Re-use the framework for Twitter OAuth that is already developed in the social module and modify it to avoid storing keys inside the internal Meeds User Profile attribute that has to be protected against any access or modification made through the "/users" REST endpoint (even by a super administrator).

Data and persistence

• Twitter events will be stored in the existing GAMIFICATION_EVENTS gamification table
• User Twitter account will be stored in the existingGAMIFICATION_CONNECTOR_ACCOUNTS gamification table
• Twitter Connector Settings will be stored in the global setting
• Twitter account to watch will be stored in a new TWITTER_ACCOUNTS

[srenault-meeds]

To review please @margondicco

[margondicco]

ok RAS Go fonc

[srenault-meeds]

Thanks. Ready for tech spec @AzmiTouil @boubaker

[boubaker]

@AzmiTouil can you please add the Tec Spec for this connector. Thanks in advance for your help

[AzmiTouil]

Ready for tech review by DAO Members (eXo: @rdenarie)

[rdenarie]

When requesting an external service, we used to log a specific message format : example :

remote_service=serviceName operation=operationName parameters="..." status=ok duration_ms=3556

This is for a successful action, and we have other message for failed request. Can you add a requirement about these logs ?

They are very usefull for diagnostic purpose and can be managed by monitoring if needed.

For the rest I'm ok.

[boubaker]

remote_service=serviceName operation=operationName parameters="..." status=ok duration_ms=3556

We don't use this anymore in fact (deprecated strategy for statistics collections). We rely in fact on analytics for such data collection.

[rdenarie]

In fact, it is not for collecting data about the Twitter API call, but the target is to have logs in case of problem.

For example : The twitter api answer Error 500 for some reason. The UI Will only say that it is not working. Having this type of logs allows to make a quicker diagnotic of the problem as the error reason is loggued.

I'm not sure the analytics API is design for this type of requirement.

The requirement I wish to add is : Allow to see quickly that the external API have a problem, and why

[boubaker]

In fact, the functional administrator should have such information and knows about the API rate usage. so I would say that this is a typical usage of analytics and let the logging used for trace the exception if the platform has a bug. Like in Alchemy, you have the list of API calls with the error + response code + duration, which is designed in Analytics as builtin attributes. We used to have such traces before to collect data and display it using Grafana for ITOP, but now that we are making SaaS offers, a functional administator should have all tools to make the diagnosis himself without implying to have access to a support/ITOP guy who will tell him what's happening.

[rdenarie]

I'm ok if such information are collected with analytics like you explain. Could it be added in technical spec ?

[boubaker]

Could it be added in technical spec ?

In fact, it should be a functional requirements, we have a dedicated section for it. @srenault-meeds WDYT ?

[srenault-meeds]

Ok added accordingly.

[boubaker]

@rdenarie ok for you ?

[rdenarie]

ok, go tech

[AzmiTouil]

Ready for review by DAO Members (eXo: @rdenarie ) https://mips-ft.meeds.io/portal/meeds/

[rdenarie]

All PR validated

[AzmiTouil]

All PR validated

Thanks, all PRs merged.