Closed gentlementlegen closed 2 months ago
@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.
interface PluginOutput {
comment: string; // html comment
rewards: Rewards; // { "whilefoo": "500", "token": "0x0" } etc
}
/start
Deadline | Mon, Apr 1, 7:36 PM UTC |
Registered Wallet | 0x2F05fD58023B0a95d1866aa0A3b672cEf05945c5 |
/wallet 0x0000...0000
if you want to update your registered payment wallet address.This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.
This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.
I think you should fork from and overtake that second pull due to us being behind schedule
@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.
interface PluginOutput { comment: string; // html comment rewards: Rewards; // { "whilefoo": "500", "token": "0x0" } etc }
there are a couple of options:
In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits. I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.
To me 1 is the most straightforward to do for few reasons:
3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result.
there are a couple of options:
- we let the conversation-rewards plugin generate and post the comment
I think the most pure architecture would be that plugins can NOT inherit (write) authentication (only read if possible) of the kernel. As a consequence, no plugin should be able to post directly any issue. Ideally it should only be the kernel with a direct interface to issues. Plugins should just output comment HTML and the kernel can post it all in a single comment at the end of the webhook event invocation chain.
- we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it
Could be interesting to have a dedicated plugin to handle commenting, but because this seems like such an essential capability, I would more carefully consider the pros/cons of including this within the kernel.
comment:
and then we don't have the comment
capability available (the comment plugin is not included in the config) then why is this a standard output property required on every plugin?
- we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.
In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits. I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.
I'm not concerned about latency now. Besides we can port any plugin to Cloudflare Workers (or combine several inside a single Worker) when we are ready to fix latency issues. To be honest though, except for setting labels, I have no problem with latency (like with permit generation) so there's no need to overengineer yet. I also love the GitHub Actions logs being available for easy debugging, and the fact that they are super generous with the compute.
Given the entire conversation and all the considerations, I definitely think we should do option 3.
This separates concerns the best, which will allow our plugin ecosystem to grow the fastest.
I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires window
instance to be instantiated which is annoying on node
based projects.
Having a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.
My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the conversation-reward
that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.
I think each plugin should output JSON not html as it is not reliable to parse nor manipulate
I know JSON makes things more complicated than it needs to be with serialization/transport. Then we have problems like this https://github.com/ubiquibot/conversation-rewards/issues/18. If the comment property is ONLY meant to be used for GitHub comments, then transporting HTML only is the most sensible. Besides, there's no other metadata thats necessary within the comment
property, which is the main point of sending a JSON object (it has multiple properties.)
As a somewhat side note: there should also not be any variables inside of the HTML. We could look it as like server-side-rendering (or in this case, plugin-side-rendering) handle producing the final HTML output and then output it as a single string of html entities.
requires window instance to be instantiated which is annoying on node based projects.
We are currently using mdast
in @ubiquibot/comment-incentives
to generate virtual DOMs. No window
needed.
My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the conversation-reward that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.
If we agree that this is considered as technical debt, and that we are accruing this so that we can get back on schedule, ok.
If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format. The new comment reward actually does instantiate a DOM through JSDOM to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.
Biggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.
If we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise.
If you want to manipulate and convey data, HTML really is not made for this.
Going back to my "plugin-side-rendering" mention, the data manipulation happens inside of the logic of the plugin. Then when the plugin is finally done with all of its compute, it emits a single string in the comment
output property. This string is the final, rendered HTML.
The comment
output is not intended to be consumed by other plugins. In most cases, the kernel will concatenate all the comment
outputs from every plugin and post a single comment at the end. Plugins will consume the metadata
property output which will include raw values to work with. I forget where I originally proposed this, but imagine something like:
type HtmlString = string;
type RequestedPermits = { username: string; amount: string; token: string; }[]
interface PluginOutput {
metadata: Record<string, any>;
comment: HtmlString;
rewards: RequestedPermits;
}
But then how do we consider the formatting of that output?
Practical case: we want to post a comment when a user queries a wallet. That comment is 'user name': 'wallet 0x0' Kernel calls the comment plugin, saying that it wants a comment to be posted. Should the Kernel send the rendering it wants, should the comment plugin transform the data to HTML?
Then, comment-reward wants to post the results. Should ask the Kernel to call the comment plugin, but then formatting is different. Should the Kernel notify the comment plugin that it wants a different output formatting? Should the Kernel compute beforehand the HTML and send it to the comment plugin?
Might be something I don't grasp there. Because I do understand your use case but it seems to be very deterministic on what is the purpose of the plugins which kinda defeats the purpose of having plugins, looks more like a monolithic approach to me
But then how do we consider the formatting of that output?
The proposed comment
output is intended for ease of composability for MOST situations, basically that the output from one plugin would just be appended to the final rendered comment. For simple plugin configurations (plugins not modifying the output results of others) this seems like straightforward architecture.
However, as we know, there will be situations where a subsequent plugin will consider the results of a previous plugin, which means it would need to change the comment that is rendered.
In this situation the subsequent plugin should clobber the output of the previous plugin. It is now clear to be that this will be a new challenge to express to the kernel that it should ignore the comment output of a previous plugin. I suppose it would be straightforward in the metadata using the plugin ID i.e. { "silences": ["ubiquibot/conversation-rewards"] }
.
Kernel calls the comment plugin,
I think that comments should be handled from within the kernel. There should not be a separate comment plugin. Read my explanation here.
For completeness of my previous comment:
type PluginId = string; // i.e. `ubiquibot/conversation-rewards` `ubiquibot/assistive-pricing`
interface PluginInput {
metadata: Record<PluginId, any>;
context: GitHubEventContext;
}
Notice that we should not pass in the HTML of other plugins. Instead, just the metadata ("computed") values from the previous plugins.
This can work, but we skyrocket coupling and to me defeat purpose of plugins that should be unaware of each other. If any plugin has to understand the result of a previous plugin, it means these plugins have necessarily to co-exist so basically they become a single plugin with no purpose to split them.
so basically they become a single plugin with no purpose to split them.
I understand your concern and I would need to put more thought into composability. Maybe the comment
proposal is bad after all due to your point. It's only useful with simple plugins, but for more complex ones your point is valid.
I need help to think through how any partner in the future can create new plugins that modify
I don't like the idea of having a single monolithic rewards generation plugin that wraps all the ways possible to compute rewards (i.e. time estimation, comment rewards, review rewards etc)
@pavlovcik To mitigate that that's why inside the comment reward itself I also integrated that Module
principle so code is not coupled tightly and easy to build on. There is as usual pros and cons to both approaches (splitting or not) but biggest pro is that comments get evaluated once in the same spot, so we save calls to OpenAPI and speed up the process. Also makes it only one configuration file in one location. We need to think about our best options there.
! action has an uncaught error
I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is ISSUE | REVIEW
with the level MEMBER | CONTRIBUTOR
etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.
I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is
ISSUE | REVIEW
with the levelMEMBER | CONTRIBUTOR
etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.
I see, so you're suggesting that we must annotate each comment as well in order to properly handle the scoring at the end?
Off hand I think there's:
I suppose we need a better naming convention for the pull related ones. They are considered as separate entities according to the GitHub api. They require different methods to obtain both types.
Agreed, I think currently there are 3 possible things to annotate on the comments:
SPECIFICATION
for the issue itself, TASK
for the related PR fixing it, or simply COMMENT
MEMBER
or CONTRIBUTOR
for the status of the memberREVIEW
or ISSUE
for where the comment was addedI think this shall cover all cases.
Consider calling it "contributor" and "collaborator" as that is how it is presented on the GitHub APIs as I recall.
Also I think you forgot about the "review comments"
/stop
+ You have been unassigned from the task
/start
Deadline | Thu, Apr 18, 7:17 AM UTC |
Registered Wallet | 0x0fC1b909ba9265A846b82CF4CE352fc3e7EeB2ED |
/wallet 0x0000...0000
if you want to update your registered payment wallet address.+ Evaluating results. Please wait...
View | Contribution | Count | Reward |
---|---|---|---|
Issue | Comment | 9 | 139.9 |
Review | Comment | 1 | 1.6 |
View | Contribution | Count | Reward |
---|---|---|---|
Issue | Specification | 1 | 23.6 |
Issue | Task | 1 | 400 |
Issue | Comment | 9 | 180.4 |
Issue | Comment | 9 | 0 |
Review | Comment | 2 | 8.6 |
Review | Comment | 2 | 4.3 |
View | Contribution | Count | Reward |
---|---|---|---|
Issue | Comment | 1 | 21.3 |
Comment | Formatting | Relevance | Reward |
---|---|---|---|
> @whilefoo rfc on how we can deal with comment outputs. Perhaps... | 21.3li: count: 3 score: "3" words: 72 code: count: 1 score: "1" words: 0 | 0.67 | 21.3 |
In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754
The same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned here
The module should be: