search

ubiquity-os / plugins-wishlist

0 stars 5 forks source link

Developer Referrals #4

Open 0x4007 opened 8 months ago

0x4007 commented 8 months ago

In order to increase the size of our developer community, we should pay who refer developers who contribute. Let's call them DevRels.

The vision is to generate a new permit for the DevRel upon task completion.

Configuration Requirements

Implementation Overview

We can do this in a couple of parts:

  1. the new referral logs in with GitHub at work.ubq.fi?ref=[github user id of DevRel]
  2. we associate in a KV store the key being the DevRel GitHub ID and the value being an array of DevRel values. 
    type GitHubId = number;
type KV = { [key: GitHubId]: GitHubId[] };

This feels a little parasitic if a developer accidentally clicks a DevRel's link at some point in the future, especially if they start losing some of their rewards. I think we need some type of system to "undo" a referral. This doesn't feel like the right solution but I'm unsure what is.

Sybils

We can protect against sybils based on the age of a GitHub profile as well as a minimum XP requirement. For example, if the developer XP is low then the rewards are zero sum?

0x4007 commented 8 months ago

@gentlementlegen @chris-chris rfc

Make sure that we can toggle zero sum mode (subtract from developer reward to pay the DevRel.)

@whilefoo rfc I see you're working on implementing https://github.com/ubiquibot/permit-generation/pull/2 do you think its realistic to adjust somebody else's reward derived from another plugin (i.e. conversation-rewards)? I hope to make the permit generation architecture as modular as possible, but I also am concerned with unnecessary overengineering at this stage since we really should have been live about a month ago already.

chris-chris commented 8 months ago

Hey, I'm here :)

chris-chris commented 8 months ago

/start

ubiquibot[bot] commented 8 months ago 
! No price label is set, so this is not ready to be self assigned yet.
chris-chris commented 8 months ago

Haha, discussion should be the first, right?

chris-chris commented 8 months ago

@pavlovcik

Q. I understood the draft of the architecture, but I have a question about the policy. Can you specify the rules.

[ZeroSum Mode]

[Not ZeroSum Mode]

I want to specify the rules in detail. But the rules can be configurable in a flexible design.

Q. What is the source of the reward in the future? For example, if a company adopt the UniquityDAO into their organization, I think it would be unfair for the company pay for the DevRel activity.

chris-chris commented 8 months ago

@pavlovcik I would like to contribute to the developer onboarding tutorial document as well. It is the core step for the developer conversion rate.

chris-chris commented 8 months ago

If the referral reward is infinite time range, the developer would hate the zero-sum reward system.

chris-chris commented 8 months ago

So, which entity takes the biggest benefit of the DevRel program?

0x4007 commented 8 months ago

Q. Is it infinite time range? Or just 1 time?

I was thinking it could be nice to have a max per contributor. For example, you refer an amazing contributor and they keep completing tasks. We can make your maximum reward from that contributor like 5000 USD?

[Not ZeroSum Mode]

The company gives extra reward to the DevRel? Q. Is is called as percentage mode?

Yes I think its just like a "flat fee mode" (we can express this in the config as percentageReward: false)

Q. What is the source of the reward in the future? For example, if a company adopt the UniquityDAO into their organization, I think it would be unfair for the company pay for the DevRel activity.

Because we are trying to make the platform as decentralized as possible, the rewards should not come from us. However we may consider emitting some of our governance tokens especially for the early partner projects as an incentive to start using our system, as well as to grant influence for early adopters.

You bring up a good point about it being unfair, so the "fair" way would be to do zero sum I think. We can bootstrap non zero sum just in the beginning since we are the only "partner" using the system.

If the referral reward is infinite time range, the developer would hate the zero-sum reward system.

We can also consider a time limit although this seems pretty easily gamable. I think it should be a cap like 5000 USD per DevRel per referral.

So, which entity takes the biggest benefit of the DevRel program?

Every user in the system should benefit. Ubiquity, as a partner, like anyone else. However, we eventually will have access to detailed performance analytics which can be useful for our recruiting efforts. New engineer: they have more work opportunities. DevRel gets money for referring. Other companies get access to a larger developer pool.

Remarks

  1. We have plans to increase the rewards substantially and remove most of our base salaries for the core team (maybe have small retainers for miscellaneous help that isn't directly financially incentivized.) This will make it simpler to do zero sum and still pay a nice amount to the DevRel. Imagine $100 USD direct reward every time a contributor helps solve problems - seems like a substantial amount that shouldn't put too much of a dent in a developer's reward payouts if they are receiving in the range of $2000 for example.
  2. Lastly, I recommend you edit a single comment and post all your feedback there. Otherwise the conversation grows much longer than it needs to be regarding scrolling on the page with the separate comments and their padding/margins.
chris-chris commented 8 months ago

@pavlovcik Oh thank you for the feedback! Yeah, I got that in my mind! :)

chris-chris commented 8 months ago

Key considerations

  1. Pricing Model and Incentive Structure:
  1. Configuration and Flexibility:
  1. Implementation Considerations:
  1. Sybil Attack Prevention:
  1. Feedback and Continuous Improvement:
0x4007 commented 8 months ago

Engaging in market research or surveys within the developer community could provide insights.

Too slow I saw we test in production and tweak parameters.

Address potential gaming of the system by introducing mechanisms to undo accidental referrals or prevent misuse.

ChatGPT isn't specific enough lol but yes zero sum is the best solution for this, I think.

Establish a feedback loop with both developers and DevRels to gather insights on the program's effectiveness and areas for improvement.

Telegram group for DevRels

Infinite time range for rewards seems problematic. Instead, a cap ensures DevRels are rewarded substantially without creating perpetual obligations.

We can set a default of 3 months.

chris-chris commented 8 months ago

Yeah I love ChatGPT lol

chris-chris commented 8 months ago

I am looking to gain a deeper understanding of the code structure and the interactions between different modules within the UbiquityDAO's DevRel program. Knowing where to find the specific pieces of code I need to work on and how these parts interact within the larger system will be instrumental in my contribution. This insight is not only crucial for my immediate work but also for creating detailed onboarding documentation that will aid future contributors in navigating and understanding the system efficiently. My goal is to grasp the overall functionality of the system and document the onboarding process to ensure a smooth integration for new developers joining the project.

0x4007 commented 8 months ago

I am looking to gain a deeper understanding of the code structure and the interactions between different modules within the UbiquityDAO's DevRel program. Knowing where to find the specific pieces of code I need to work on and how these parts interact within the larger system will be instrumental in my contribution. This insight is not only crucial for my immediate work but also for creating detailed onboarding documentation that will aid future contributors in navigating and understanding the system efficiently. My goal is to grasp the overall functionality of the system and document the onboarding process to ensure a smooth integration for new developers joining the project.

@eresdev perhaps you can point out the bit on work.ubq.fi and @gentlementlegen perhaps you might know from the perspective of plugins

EresDev commented 8 months ago

I am looking to gain a deeper understanding of the code structure and the interactions between different modules within the UbiquityDAO's DevRel program. Knowing where to find the specific pieces of code I need to work on and how these parts interact within the larger system will be instrumental in my contribution. This insight is not only crucial for my immediate work but also for creating detailed onboarding documentation that will aid future contributors in navigating and understanding the system efficiently. My goal is to grasp the overall functionality of the system and document the onboarding process to ensure a smooth integration for new developers joining the project.

You can see a list of available bounties in the devpool at work.ubq.fi from multiple applications of Ubiquity DAO.

Each item in the list provides a task description, the incentive amount, priority, and urgency level. It also provides a link to the source GitHub issue. You can go through all the tasks, and find the one that suits your experience. Make sure it is not assigned to anyone else on Github, and start working on it. Ubiquibot will help you through the entire process, from starting the task to getting paid. As you already typed /start command to assign you a task, you can also type /help to see other available commands.

Start by forking a repository of interest and see how can make changes to it to achieve the goal described in the issue. Once, you think the work is ready for submission, you can open a pull request. Do not forget to include a QA with each pull request to show how your changes are working. Your pull request will go through a review before being merged to the production.

Every application you see here is constantly being improved and enhanced. Including work.ubq.fi itself. If you are interested in any listed issue of work.ubq.fi itself, you can see its repository. It is a simple typescript application with some CI/CD Github workflows and tests. A readme file is included with each repository to help you get started.

gentlementlegen commented 8 months ago

I think we need at least the condition that the new person you referred completed a task before you get any incentive, to avoid abuse. Then maybe we can think of tiers like "completed 1 task, 5, 10 etc." and make the incentive increase based on this. It's a pretty common scheme for referrals.

Tech wise, we cannot use Cloudflare KV for this as it is not suited for relational queries or search (think of it as a .env file). We would need a fully fledge database. If you really wanna go full native Cloudflare tools, they provide D1 for this. The problem is it doesn't have any ORM not Typing capabilities. Otherwise, we can keep it in Supabase since it also handles our login and would be the logical place to store referral information. If the problem comes from the fact that Supabase is hosted outside of Cloudflare, they provide a worker implementation.

0x4007 commented 8 months ago

@EresDev I'm talking about the Google Analytics system you implemented and details around how we track DevRels. We already have a program overview at https://dao.ubq.fi/devpool-flow

I think we need at least the condition that the new person you referred completed a task before you get any incentive, to avoid abuse.

I've implemented something at my old agency at the past that would add to our team's payroll if they met some leads out at conferences which seems fairly similar. They would enter their email in a webapp and we would enrich it/look up data on it (return their LinkedIn, Twitter, Facebook etc.) Also if it was not a real email it would bounce. This was how we were able to determine fake emails or not. In addition we would track GPS location and timestamp so I was able to plot the business development team's efforts with live analytics. It was a nifty project.

The point of me highlighting all that is we can validate if they are a "real profile" in an opaque way (maybe not open source it so it is more difficult to game.) but at a high level we can set arbitrary checks like 1. age of profile 2. amount of contributions in the last year 3. followers/following 4. enough filled out information on their profile (i.e. Twitter, organization they work at etc.)

Lastly we had an audit tool which would let me review the data. Given that the bizdev team were all people I knew in real life/personal friends, and all the email validation tools (honestly, we can do the same with the GitHub profile information- Supabase gives us their emails) we never had a problem with sybils.

The only problem I see is that some folks on the team, including myself, have anonymous emails for our GitHub profiles so it is not 100% reliable unfortunately.

This is at top of mind because I was updating our pitchdeck and wanted to share some background on how the idea of UbiquiBot came about. Here's a fun little video that shows a little of the old system back from 2016.

EresDev commented 8 months ago

@EresDev I'm talking about the Google Analytics system you implemented and details around how we track DevRels.

Currently, Google Analytics tracks common events like page views, and first visits on every page load of work.ubq.fi. It tracks a devrel referral when they logins to work.ubq.fi using devrel's referral link e.g. https://work.ubq.fi/?devRel=EresDevRel using their Github.

I see things are going to get a little more complex and I recommend moving to better data storage. My experience with Google Analytics is that it is only good for its exact use cases. Any deviation from the exact use-case, it gets difficult to produce fast results. However, it can still stay to provide the exact analytics it is suitable for e.g. page views, user interaction with the app. But anything custom should move to better data storage.

What we have right now with Google Anlytics can stay, but I see the incentive system would need access to data stored on Analytics and probably would end up changing what we store there.

The point of me highlighting all that is we can validate if they are a "real profile" in an opaque way (maybe not open source it so it is more difficult to game.) but at a high level we can set arbitrary checks like 1. age of profile 2. amount of contributions in the last year 3. followers/following 4. enough filled out information on their profile (i.e. Twitter, organization they work at etc.)

This has the potential to stop real profiles too. Completion of a task before getting incentivized sounds like a great idea. The devrel incentive can start to go down with each task a developer completes. 10% to the devrel on the first task, 9% on the second, and so on.

0x4007 commented 8 months ago

10% to the devrel on the first task, 9% on the second, and so on.

I think this approach is pretty cool. One consideration though is that I've seen the highest quality candidates usually start with tiny tightly scoped tasks and then work their way up to more complex tasks. This also means smaller to larger pricing. I'm just trying to visualize in my mind if the rewards will be worth it for the DevRel's especially if we go zero sum. Imagine a 100 USD total reward and they get 10 USD. And then eventually, for example, on the fifth task, they get a 500 USD reward * 5% = 25 USD. With this scenario and some simple linear regression, I estimate that the DevRel would obtain a max of ~250 USD per DevRel.

Tech wise, we cannot use Cloudflare KV for this

@gentlementlegen I figured we can do key = GitHub user ID and value = array of GitHub user IDs. See my specification above. I just want to ensure that its as simple as possible to maintain our plugins, and setting up a Supabase instance seems significantly more involved than a KV store.

chris-chris commented 8 months ago

@pavlovcik

  1. Incentive Structure: Designing a decreasing incentive model based on task completion, which aligns rewards with the developer's engagement level.

Let's design the incentive structure to be as flexible as possible, allowing for easy adjustments whenever necessary. This way, we can solve potential problems by simply deciding how to tweak the levers. The key is to ensure that we can modify the incentives without major overhauls, adapting quickly to feedback and evolving requirements.

  1. Data Management: Shifting from Google Analytics for custom tracking needs to a more robust and flexible system like Supabase or a similar platform that allows us to manage custom data sets effectively.

Having spent years addressing technical debt, I deeply understand the impact of flawed technical decisions. While a Key-Value implementation might seem simple, I've observed that as documents accumulate, the system's data transparency and architecture can become increasingly difficult to manage.

When it comes to selecting a database, I believe we should make a careful and informed choice. Even if we opt for a No-SQL solution, it's crucial to have automated schema management and documentation tools in place to ensure the system remains manageable and scalable.

I would like to recommend to introduce flyway for Supabase for schema history management.

https://dev.to/sruhleder/how-to-migrate-supabase-databases-with-flyway-github-actions-2ani

I prefer using Supabase since it is PostgreSQL based DB especially it's good for user management features.

EresDev commented 8 months ago

I'm just trying to visualize in my mind if the rewards will be worth it for the DevRel's especially if we go zero sum.

If the amount is low for devrel, it can be more than 10%, like 20% or so.

Here is another idea.

A referral system works great when both parties have something to gain and nothing to lose. This is possible if the pricing of the tasks is managed in the background. It should not look like the devrel is getting a cut, instead you are getting a little extra. This has the potential to attain devrel with a large audience like developers with large followings on youtube, and other social media who are always looking for affiliates.

So if a task is 100$. Let's say devrel is supposed to get like 20%. Put an extra 5% fee for devrel promotion program. Now, the visible price of the task is 75$. If the developer is from a devrel, they get 5$ extra, and devrel gets 20$. In case the task is done by a developer who is not from a devrel, all 25$ goes to devrel promotion program pool, and it can be used for other purposes like rewarding the best-performing developers or devrels.

This will require separating the tasks for these developers. And because now the devrel's work has become much much easier, their commission can be much much lower like 1-2%.

0x4007 commented 8 months ago

@chris-chris:

  1. I deleted the summary because I don't think it makes sense to add summaries in an active discussion. Its my responsibility as the author to summarize once the specification has been finalized. I also have a plugin on the wishlist specifically for this.

Let's design the incentive structure to be as flexible as possible, allowing for easy adjustments whenever necessary. This way, we can solve potential problems by simply deciding how to tweak the levers. The key is to ensure that we can modify the incentives without major overhauls, adapting quickly to feedback and evolving requirements.

  1. We need concrete technical proposals and analysis rather than high-level commentary. Please focus on sharing specific, actionable ideas taking into consideration their pros/cons.

@EresDev:

It should not look like the devrel is getting a cut, instead you are getting a little extra.

I think this is a really good idea.

Implementation Strategies

More Config Params To All Related Plugins

A partner must manually calculate and sync the subtracted amount from any plugin affected. Taking the assistive-pricing plugin as an example: we can manually tweak the assistive-pricing config with a "flat rate" or percentage" subtraction of the final computed reward amount[^1^].

Branching/Labeling For Special Accommodations

In case the special accommodations over complicate all of our related plugin code, we could branch assistive-pricing and make a developer-referrals branch/label but doesn't seem very elegant. i.e. in the config 

      uses: ubiquibot/assistive-pricing@developer-referrals

It is easy to keep in sync with latest changes because you can pull from main occasionally within the plugin repo.

Plugins Dynamically Working Together

A more complex solution would be that plugins can dynamically resolve changes to eachothers' configs. The more I think about this, the more it seems to be a massive project. I am not sure how we can have a separate dedicated plugin modify the behavior of another. @whilefoo rfc

[^1^]: But one thing I realize is rewards accrued from other plugins like conversation-rewards is not being taken into account. However, assignee reward is the only one that sets clear expectations from the get-go so it makes sense to manage those expectations proactively over any of the other rewards. This is a problem to solve for another day, when we make a chrome extension that will overlay the pending conversation rewards etc on a conversation.