Improve Orbit comprehension and settings

[ferdi05]

• [x] Understand and challenge how our current Orbits are designed (why someone is in a orbit)

[ferdi05]

• [x] Make sure that orbit size and presence are correctly being measured

[ferdi05]

A few remarks on Orbit:

• Definition of Orbit levels: uses percentile rathen that an absolute value of Love

[ferdi05]

• [x] Clean up and create the correct activities in Orbit settings (private link) as described in Orbit model.

[ferdi05]

I can't find a way to automatically track the Presence of all members of a specific orbit or of the community. I asked for help on Orbit's Discord.

[ferdi05]

I also asked how Reach is calculated, it's unclear

[ferdi05]

It looks like emoji reactions are not tracked by the Slack integration. I've asked for it on Orbit's Discord, but in the meantime, this limits our definition of a Slack active member.

[ferdi05]

Exploring MeiliSearch's private Cloudflare worker repo, I understood that the data sent to Segment comes from an Orbit report. Here's an example (private link) of what a typical report looks like.

[ferdi05]

I can't find a way to automatically track the Presence of all members of a specific orbit or of the community. I asked for help on Orbit's Discord.

@qdequele it seems doable using the List members in a workspace route. The response is a json that gives reach, love and orbit_level for each member. So it's one API call with a long json to process. It can be sliced in as many API calls as wanted to get smaller responses, by playing with the items and page parameters of the query.

Here what we want to do is: for each orbit_level (can be 1,2,3 or 4)     sum         for each member             love*reach

If the result can be sent to Notion or Google Sheet that would be great (until we find a way to send this to Amplitude).

Is this something that you can ask a Cloudflare worker to do? thanks

[qdequele]

@ferdi05 I finally did it. It was not simple :x https://analytics.amplitude.com/meili/chart/j7hv8zl?source=copy+url

[ferdi05]

Thanks @qdequele it seems that it did not work lately, could you check? Also is the NULL level for the whole community? thanks

[ferdi05]

@qdequele good news!! I have a new request to plot metrics coming from Orbit.

I'm trying to count the number of contributors to Meilisearch over time.

It won't be a number of contributors per month as this wouldn't make much sense: with a 2 month long release cycle, we may get a lot of contributions during the first half of the cycle, and less during the second half. So it would be the total number of contributors since the start.

This looks doable again with the List members in a workspace API route. Parameter would be:

• affiliation: member
• activity_type: pull_requests:merged
• start_date: 2018-05-14
• end_date: each month (like 2022-01-3) starting from 2019-04-30

See for instance this report (private link).

Can you help me to plot this in Amplitude? thanks

[qdequele]

@ferdi05 Yes, grab a time in my calendar :)

[ferdi05]

@qdequele as discussed I'm trying to get this work, but was unsuccessful so far. I have different figures on Amplitude and Orbit (private links), but I'm really out of my comfort zone with the Amplitude chart settings. Can you help? thanks

[ferdi05]

Just had a discussion with @qdequele and this seems undoable so far :(

[ferdi05]

On October 17, 2022, @Luna-meili and I met to discuss weights of Slack activities. Some integrations are not really behaving as we would expect them to.

Also we did not know how we should distribute the weight of activities:

• linearly distributed according to the engagement (may not be the right term, my maths courses were too long ago, but I mean following a straight line): activities with a lot of engagement has a bit more weight than those with some engagement and so on
• exponentially distributed: activities with more engagement have a much high weight than activities with small engagement

Find all those reflections on Notion (private link) (playing with a table on Notion is far more easy than on GitHub, but eventually we'll publish our weights here)

[ghost]

Following the meeting we had with @ferdi05, we came up with a list of questions we'd like to ask and points we wish to clarify with our Orbit account manager in order to improve our usage of the tool:

• We're not sure we understand very well how a community member goes from group 3 to 4 or from 4 to 3. Is there a threshold of Love ? From the FAQ:

  

• What is the best way to handle Potential (ex-presence) in the tool? How can we understand it? How to make the best of it? > No one really has this mesure, but people seem to be doing that and focus more on how to bring them from one to the next level

• When we set a graph using the "since" function, why doesn't it update automatically? How long until it updates itself? Is the current day not taken into account? How can it remain a "since X until today" everytime we visit the graph? > this is a bug, they can try to adjust it on their side

• On Slack

  • We don't really understand our Slack setup, what is the logic behind the (current) default setup? (eg: weight attribution - private link, why not all channels are integrated?) > https://orbit.love/knowledge-base/how-to-set-up-the-orbit-slack-integration

• On Reddit:

  • Can we include Reddit messages outside of our own subreddit r/meilisearch to be counted as community activities? (eg: if someone in r/rust mentions Meilisearch, how can we have that count?) > https://orbit.canny.io/integrations/p/reddit-keyword-search
  • We would like upvotes and downvotes to be counted as community engagement

• On Twitter:

  • Are the keyword and the mentions counted twice? > Jira is opened for it, it’s under investigation
  • Does each Tweet trigger an additional activity point? Can we see somewhere the addition of each point that creates the final community member score? > Yes each tweet will trigger additional activity. You can’t see how the score is attributed
  • Can 1 Tweet trigger multiple Love points? (eg: @ = 1 point and "Meilisearch keyword" = 1 point in the same tweet?)
  • Can we include likes and RT in our pointing system> Not sure
  • Can we differentiate (in the pointing system) between people that start a conversation and people who reply on Twitter? > Not sure

• How can we manually remove mentions we don't consider relevant from Orbit? Can we or must we go through support? how can we get rid of activities and members mistakenly added when using the integration with r/rust > support ticket needs to be open

• How would you weight each activity? Can we weight the same actions differently depending on the platform?

We've also received additional feedback from @curquiza that we will gladly pass along during our meeting:

• This request is a high priority: The possibility to create group of activity properties (e.g. all repos from the core team) to make comparison easier. Eg: she'd like to be able to separate the PR merged by the integration team from the PRs merged by the core team >No answer for this yet unfortunately, she will bring this back to the product team
• The possibility to overlay 2 graphs that share the same XY axis (eg: be able to compare the data of 2 different teams) >not possible now but it’s in their future plans; maybe we can find a workaround with Amplitude. She’ll ask internally
• This request is a high priority: The possibility to merge repos (eg: same repo but with a different name such as Tokenizer, tokenizer, charabia) in the same way we can merge profiles. She's happy to do this manually if need be. e.g. Core graph (private link) >-> No answer for this yet. She’ll investigate internally
• The possibility to have averages and median calculated automatically (eg: average number of merges PR by month OR average number of GitHub stars per month/per user)
• The addition of a third category called "ex-teammate" (currently there's only "members" and "teammate") when someone moves on from the company. She'd like to be able to adjust the dates of when the person was a teammate and when they became a member. The current setup falsifies her stats. > Not sure this will even be feasible

Feedback from the Orbit team about this section of questions: Maybe orbit isn’t the best tool for this type of usage. Orbit’s main function is around following Community activity rather than following employees contribution.

Similar point we also raised:

• Bug report: "since" behaviour in graphs is suboptimal as it doesn't update automatically. If you put "since" on October 18th, it'll stay on Oct 18th even if time goes on. > this is a bug, they can try to adjust it on their side

[ferdi05]

Thanks @Luna-meili and @curquiza for this amazing list. I edited it a bit to add some extra info.

I'd like to add something a bit anecdotal: we keep on getting the following error: Someone who set up the MeiliSearch community Slack App integration has deleted their account. Please [reset the Slack App integration](https://app.orbit.love/meili/settings/integrations) to restart imports.

[ghost]

I've updated my post with the answers from our account manager. You can identify the answers as I've written them as follows: >This is the answer of our account manager

[ghost]

We had a follow-up call with our Orbit account manager. Please find below the notes @ferdi05 took:

Amplitude has suggested giving us access to the raw data/data usable. We think this will solve some of the points raised around the stream of activities for the workspace. Cohort analysis becomes doable There's a possibility to do a SQL exercise It'll be possible to use queries to search through the data. There are 2 ways to handle the data:

• advanced way: push something to our data warehouse
• less advanced way: metabase and use custom dashboards This will be useful for grouping by activities and repos repos, which would solve the points raised by @curquiza coming soon: custom activities which represents a fancy tagging for activities This is all in the API.

There will be some updates on the Orbit model: change to a threshold base rather than on a percentage base to define why a person is in an Orbit, said threshold could be configurable to be adapted to Orbit user's needs.

What defines a Community member?

• contribution path: open a PR/an issue/code project-related tools Some new calculations will be done by the end of the year on how activities are weighted. It might be the case activities are grouped together rather than simply looking at the weights of them. Regarding guidance on how to weights activities, there's a table in the documentation

We put in a public request for the handling of tweets and retweets: https://ideas.orbit.love/integrations/p/twitter-retweets

We've received a link explaining more in detail how activities are tracked: https://orbit.love/knowledge-base/integrations-historical-import-and-activities Reddit: putting activity to 0 should recalculate the love score to 0, could be a bug

We've also identified a bug with Reddit. Even despite having an activity level of 0, the score of Reddit users isn't at 0 in Orbit.

[ghost]

A new update was rolled out on Orbit and we need to understand the changes.

One of the main changes is around the mesurement of love. There's a new factor called "presence". Basically, the level of love of a person is judged based on their baseline. So for instance, if Person 1 is active 4x/week and then becomes active 2x/week the following month, their presence will drop. However, Person 2 is active 1x/week and then becomes active 2x/week the following month, their presence will rise. this is due to their difference in baseline.

This makes us wonder about the ability of comparing people to each other since everyone's baseline is personalised.

Secondly, we know some of our users disable the data tracking on their profiles, and we were wondering if Orbit could still track these profiles or not.

We also would like to understand better how Orbit handles Twitter in general, especially accounts of our team members.

Finally, we would like to get their insight on how they mesure the weight of activities within the new system.

I have a meeting tomorrow afternoon with our Orbit success manager.