65536 GitHub Comment Character Limit

0x4007 commented 7 months ago

This run seemed to have failed on the transmission step at the very end because according to the logs it even generated the permits, but then failed to post back to the original repository issue comments.

A GitHub comment size is max 65536 characters long. Since the conversation incentives were so long, perhaps it makes sense to truncate the permit comment's metadata (we use HTML comments to comment out an embedded JSON object) or to somehow compress it further.

https://github.com/ubiquibot/comment-incentives/actions/runs/7896867759/job/21551519371


Error: HttpError: Validation Failed: {"resource":"IssueComment","code":"unprocessable","field":"data","message":"Body is too long (maximum is 65536 characters)"}

0x4007 commented 7 months ago

@rndquu would you mind handling this? Getting payouts to work is a top priority and this seems like there's many ways to go about handling this.

The vision is to be able to interoperate bot plugins with a standardized comment metadata schema. The problem is that this counts against the character limit. Perhaps we can filter out the _url related properties in the GitHub object embedded in the metadata. GitHub includes a ton of URLs as part of any object entity like a GitHub Issue object or a GitHub Comment object.

Beyond that, I would prefer to not have to compromise for edge cases with SOME permit comments not having metadata. It seems like that will lead to headaches down the line for future tooling if its not consistent.

gitcoindev commented 7 months ago

I may be able to fix this today, will try to set up and reproduce on my forked repository.

molecula451 commented 7 months ago

you think you can have this one @gitcoindev ?

gitcoindev commented 7 months ago

you think you can have this one @gitcoindev ?

Hi @molecula451 I am a bit stuck on the setup on my fork, it seems it needs to be run from org and depends on other config repositories. I tried to run locally from my local .env but in the end it needs github actions. If you feel you can fix this one quickly with QA please go ahead. I will experiment on my setup anyway today but feel free to pick it up -)

molecula451 commented 7 months ago

we're all debugging on this one, so that's what im asking i think we're all can come with a good solution

gitcoindev commented 7 months ago

Ah sure. I also see @pavlovcik has been experimenting in parallel. I hope we can get it fixed together somehow.

0x4007 commented 7 months ago

I'm setting up jest to mock the incoming event object

0x4007 commented 7 months ago

It is somewhat stable (I discovered a bug!) so I'm working on that here https://github.com/ubiquibot/comment-incentives/pull/21

gitcoindev commented 7 months ago

Btw. I am not sure if I can see in the 'matrix' correctly today after I spent quite some time looking at logs -) but for me the root cause seems to be that the comment that is going to be inserted is too large because of another error.

I see that relevance-scoring thrown

Failed to average samples {
  error: TypeError: Cannot read properties of undefined (reading 'map')
      at averageSamples (/home/runner/work/comment-incentives/comment-incentives/src/handlers/issue/relevance-scoring.ts:170:8)

and in turn it created a cascade of trace dumps from failed generate-erc20-permit-signature.ts . This in total is larger than 65536 and it fails to insert such huge comment.

gitcoindev commented 7 months ago

I also downloaded json and wrote a simple script that calculated comment body lengths:

So as you can see even the sum of characters in all comments in this issue is less than 65536, which also is in favor of the theory that the new comment to be inserted is too large due to error traces.

molecula451 commented 7 months ago

30186? then is almost half of the acceptable character limit

0x4007 commented 7 months ago

Btw. I am not sure if I can see in the 'matrix' correctly today after I spent quite some time looking at logs -) but for me the root cause seems to be that the comment that is going to be inserted is too large because of another error.

I see that relevance-scoring thrown
Failed to average samples {
  error: TypeError: Cannot read properties of undefined (reading 'map')
      at averageSamples (/home/runner/work/comment-incentives/comment-incentives/src/handlers/issue/relevance-scoring.ts:170:8) 
and in turn it created a cascade of trace dumps from failed generate-erc20-permit-signature.ts . This in total is larger than 65536 and it fails to insert such huge comment.

Yea the bug I identified was that I accidentally coded it to check if the price labels ARRAY exists, instead of checking that the length is zero. That seemed to throw that map error that I would see so commonly in the logs.

In this specific scenario, the test issue has no labels and I think the code was not designed to handle that. Still getting it stable, I'm at permit generation part now (missing private key in my env vars!)

https://github.com/ubiquibot/comment-incentives/blob/77ba9b0fc95924930eaeaa3cbfe689bef26ab4f8/src/handlers/issue/assignee-scoring.ts#L6-L75

which also is in favor of the theory that the new comment to be inserted is too large due to error traces.

This is a good theory!

0x4007 commented 7 months ago

This is a pretty solid co-op campaign

0x4007 commented 7 months ago

Hey guys, I got it to work! Can somebody work on generating a new X25519_PRIVATE_KEY for the ubiquibot organization? We need to share it for development purposes.

Generate a new key
Encrypt a new evmPrivateKeyEncrypted (no funds!)
Add the shared test key to the comment-incentives readme.

https://github.com/ubiquibot/comment-incentives/pull/21/commits/567419d9688e92edf698f64c697f1a7cafe1d02e

ubiquibot[bot] commented 7 months ago

+ Evaluating results. Please wait...

ubiquibot[bot] commented 7 months ago

[ 104.2 WXDAI ]

@pavlovcik

Contributions Overview

View	Contribution	Count	Reward
Issue	Specification	1	23
Issue	Comment	6	77.4
Review	Comment	2	3.8

Conversation Incentives

Comment	Formatting	Relevance	Reward
This run seemed to have failed on the transmission step at the v...	23 code: count: 1 score: "1" words: 0	1	23
@rndquu would you mind handling this? Getting payouts to work is...	27 code: count: 1 score: "1" words: 1	0.765	27
I'm setting up jest to mock the incoming event object...	2.2	0.565	2.2
It is somewhat stable (I discovered a bug!) so I'm working on th...	4.6	0.62	4.6
> Btw. I am not sure if I can see in the 'matrix' correctly toda...	23.6 a: count: 1 score: "1" words: 2 code: count: 1 score: "1" words: 0 hr: count: 1 score: "1" words: 0	0.78	23.6
This is a pretty solid co-op campaign...	1.6	0.825	1.6
Hey guys, I got it to [work](https://github.com/ubiquibot/produc...	18.4 a: count: 1 score: "1" words: 1 li: count: 3 score: "3" words: 24 code: count: 3 score: "3" words: 4	0.83	18.4
> code looks good, can you paste us some sort of debugging outpu...	2.4 a: count: 1 score: "2" words: 2	0.65	2.4
Let's give it a test @molecula451 ...	1.4	0.36	1.4

[ 5.5 WXDAI ]

@molecula451

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	3	4.2
Review	Comment	1	1.3

Conversation Incentives

Comment	Formatting	Relevance	Reward
you think you can have this one @gitcoindev ?...	0.8	0.41	0.8
we're all debugging on this one, so that's what im asking i thin...	2.4	0.54	2.4
30186? then is almost half of the acceptable character limit...	1	0.685	1
code looks good, can you paste us some sort of debugging output/...	1.3	0.68	1.3

[ 31.95 WXDAI ]

@gitcoindev

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	5	31.95

Conversation Incentives

Comment	Formatting	Relevance	Reward
I may be able to fix this today, will try to set up and reproduc...	1.9	0.7	1.9
> you think you can have this one @gitcoindev ? Hi @molecula4...	7.4	0.61	7.4
Ah sure. I also see @pavlovcik has been experimenting in paralle...	2	0.575	2
Btw. I am not sure if I can see in the 'matrix' correctly today ...	11.35 code: count: 1 score: "0.25" words: 0	0.695	11.35
I also downloaded [json](https://api.github.com/repos/ubiquity/u...	9.3 a: count: 1 score: "0.25" words: 1 code: count: 1 score: "0.25" words: 0	0.655	9.3

ubiquibot[bot] commented 7 months ago

@pavlovcik the deadline is at 2024-02-16T21:20:06.472Z

ubiquibot[bot] commented 7 months ago

+ Evaluating results. Please wait...

0x4007 commented 7 months ago

+ Evaluating results. Please wait...

Looks like it happened again. I'll use this exact issue to test and push another fix asap!

molecula451 commented 7 months ago

btw judging the gitcoindev message i think the character set it's manipulatible say if i post a comment that long loooking like a drop down table the bot could think that perhaps my contribution comment is that helpful because of longer?

need to round well

molecula451 commented 7 months ago

indeed it happened

image_2024-02-16_132738180

0x4007 commented 7 months ago

Here is the full transmission for the broken comment:

1708104501473.html.zip

Some initial observations:

metadata can be minified (not pretty printed)
wordScoreCommentDetails counting every single word in the conversation is pretty insane. We should probably exclude that.
Conclusion is just to remove the metadata of the totals object. That will certainly take out most of the metadata.

ubiquibot-dev[bot] commented 7 months ago

[ 330.3 WXDAI ]

@pavlovcik

Contributions Overview

View	Contribution	Count	Reward
Issue	Specification	1	23
Issue	Task	1.00	200
Issue	Comment	8	101.6
Issue	Comment	8	0
Review	Comment	2	3.8
Review	Comment	2	1.9

Conversation Incentives

Comment	Formatting	Relevance	Reward
This run seemed to have failed on the transmission step at the v...	23 code: count: 1 score: "1" words: 0	1	23
@rndquu would you mind handling this? Getting payouts to work is...	27 code: count: 1 score: "1" words: 1	0.7	27
I'm setting up jest to mock the incoming event object...	2.2	0.57	2.2
It is somewhat stable (I discovered a bug!) so I'm working on th...	4.6	0.715	4.6
> Btw. I am not sure if I can see in the 'matrix' correctly toda...	23.6 a: count: 1 score: "1" words: 2 code: count: 1 score: "1" words: 0 hr: count: 1 score: "1" words: 0	0.765	23.6
This is a pretty solid co-op campaign...	1.6	0.76	1.6
Hey guys, I got it to [work](https://github.com/ubiquibot/produc...	18.4 a: count: 1 score: "1" words: 1 li: count: 3 score: "3" words: 24 code: count: 3 score: "3" words: 4	0.62	18.4
> ```diff > + Evaluating results. Please wait... > ``` Look...	5.6 a: count: 1 score: "1" words: 2 code: count: 1 score: "1" words: 0	0.695	5.6
Here is the full transmission for the broken comment: [170810...	18.6 a: count: 1 score: "1" words: 3 li: count: 3 score: "3" words: 47 code: count: 2 score: "2" words: 2 hr: count: 1 score: "1" words: 0	0.78	18.6
@rndquu would you mind handling this? Getting payouts to work is...	- code: count: 1 score: "0" words: 1	0.7	-
I'm setting up jest to mock the incoming event object...	-	0.57	-
It is somewhat stable (I discovered a bug!) so I'm working on th...	-	0.715	-
> Btw. I am not sure if I can see in the 'matrix' correctly toda...	- a: count: 1 score: "0" words: 2 code: count: 1 score: "0" words: 0 hr: count: 1 score: "0" words: 0	0.765	-
This is a pretty solid co-op campaign...	-	0.76	-
Hey guys, I got it to [work](https://github.com/ubiquibot/produc...	- a: count: 1 score: "0" words: 1 li: count: 3 score: "0" words: 24 code: count: 3 score: "0" words: 4	0.62	-
> ```diff > + Evaluating results. Please wait... > ``` Look...	- a: count: 1 score: "0" words: 2 code: count: 1 score: "0" words: 0	0.695	-
Here is the full transmission for the broken comment: [170810...	- a: count: 1 score: "0" words: 3 li: count: 3 score: "0" words: 47 code: count: 2 score: "0" words: 2 hr: count: 1 score: "0" words: 0	0.78	-
> code looks good, can you paste us some sort of debugging outpu...	2.4 a: count: 1 score: "2" words: 2	0.68	2.4
Let's give it a test @molecula451 ...	1.4	0.31	1.4
> code looks good, can you paste us some sort of debugging outpu...	1.2 a: count: 1 score: "1" words: 2	0.68	1.2
Let's give it a test @molecula451 ...	0.7	0.31	0.7

[ 10.7 WXDAI ]

@molecula451

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	5	9.1
Review	Comment	2	1.6

Conversation Incentives

Comment	Formatting	Relevance	Reward
you think you can have this one @gitcoindev ?...	0.8	0.625	0.8
we're all debugging on this one, so that's what im asking i thin...	2.4	0.65	2.4
30186? then is almost half of the acceptable character limit...	1	0.81	1
btw judging the gitcoindev message i think the character set it'...	4.6	0.8	4.6
indeed it happened ![image_2024-02-16_132738180](https://git...	0.3	0.755	0.3
code looks good, can you paste us some sort of debugging output/...	1.3	0.68	1.3
on my way!...	0.3	0.33	0.3

[ 31.95 WXDAI ]

@gitcoindev

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	5	31.95

Conversation Incentives

Comment	Formatting	Relevance	Reward
I may be able to fix this today, will try to set up and reproduc...	1.9	0.66	1.9
> you think you can have this one @gitcoindev ? Hi @molecula4...	7.4	0.655	7.4
Ah sure. I also see @pavlovcik has been experimenting in paralle...	2	0.68	2
Btw. I am not sure if I can see in the 'matrix' correctly today ...	11.35 code: count: 1 score: "0.25" words: 0	0.77	11.35
I also downloaded [json](https://api.github.com/repos/ubiquity/u...	9.3 a: count: 1 score: "0.25" words: 1 code: count: 1 score: "0.25" words: 0	0.665	9.3

ubiquibot[bot] commented 7 months ago

+ Evaluating results. Please wait...

ubiquibot[bot] commented 7 months ago

[ 330.3 WXDAI ]

@pavlovcik

Contributions Overview

View	Contribution	Count	Reward
Issue	Specification	1	23
Issue	Task	1.00	200
Issue	Comment	8	101.6
Issue	Comment	8	0
Review	Comment	2	3.8
Review	Comment	2	1.9

Conversation Incentives

Comment	Formatting	Relevance	Reward
This run seemed to have failed on the transmission step at the v...	23 code: count: 1 score: "1" words: 0	1	23
@rndquu would you mind handling this? Getting payouts to work is...	27 code: count: 1 score: "1" words: 1	0.67	27
I'm setting up jest to mock the incoming event object...	2.2	0.56	2.2
It is somewhat stable (I discovered a bug!) so I'm working on th...	4.6	0.63	4.6
> Btw. I am not sure if I can see in the 'matrix' correctly toda...	23.6 a: count: 1 score: "1" words: 2 code: count: 1 score: "1" words: 0 hr: count: 1 score: "1" words: 0	0.71	23.6
This is a pretty solid co-op campaign...	1.6	0.75	1.6
Hey guys, I got it to [work](https://github.com/ubiquibot/produc...	18.4 a: count: 1 score: "1" words: 1 li: count: 3 score: "3" words: 24 code: count: 3 score: "3" words: 4	0.74	18.4
> ```diff > + Evaluating results. Please wait... > ``` Look...	5.6 a: count: 1 score: "1" words: 2 code: count: 1 score: "1" words: 0	0.8	5.6
Here is the full transmission for the broken comment: [170810...	18.6 a: count: 1 score: "1" words: 3 li: count: 3 score: "3" words: 47 code: count: 2 score: "2" words: 2 hr: count: 1 score: "1" words: 0	0.79	18.6
@rndquu would you mind handling this? Getting payouts to work is...	- code: count: 1 score: "0" words: 1	0.67	-
I'm setting up jest to mock the incoming event object...	-	0.56	-
It is somewhat stable (I discovered a bug!) so I'm working on th...	-	0.63	-
> Btw. I am not sure if I can see in the 'matrix' correctly toda...	- a: count: 1 score: "0" words: 2 code: count: 1 score: "0" words: 0 hr: count: 1 score: "0" words: 0	0.71	-
This is a pretty solid co-op campaign...	-	0.75	-
Hey guys, I got it to [work](https://github.com/ubiquibot/produc...	- a: count: 1 score: "0" words: 1 li: count: 3 score: "0" words: 24 code: count: 3 score: "0" words: 4	0.74	-
> ```diff > + Evaluating results. Please wait... > ``` Look...	- a: count: 1 score: "0" words: 2 code: count: 1 score: "0" words: 0	0.8	-
Here is the full transmission for the broken comment: [170810...	- a: count: 1 score: "0" words: 3 li: count: 3 score: "0" words: 47 code: count: 2 score: "0" words: 2 hr: count: 1 score: "0" words: 0	0.79	-
> code looks good, can you paste us some sort of debugging outpu...	2.4 a: count: 1 score: "2" words: 2	0.67	2.4
Let's give it a test @molecula451 ...	1.4	0.36	1.4
> code looks good, can you paste us some sort of debugging outpu...	1.2 a: count: 1 score: "1" words: 2	0.67	1.2
Let's give it a test @molecula451 ...	0.7	0.36	0.7

[ 10.7 WXDAI ]

@molecula451

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	5	9.1
Review	Comment	2	1.6

Conversation Incentives

Comment	Formatting	Relevance	Reward
you think you can have this one @gitcoindev ?...	0.8	0.47	0.8
we're all debugging on this one, so that's what im asking i thin...	2.4	0.56	2.4
30186? then is almost half of the acceptable character limit...	1	0.83	1
btw judging the gitcoindev message i think the character set it'...	4.6	0.72	4.6
indeed it happened ![image_2024-02-16_132738180](https://git...	0.3	0.69	0.3
code looks good, can you paste us some sort of debugging output/...	1.3	0.77	1.3
on my way!...	0.3	0.34	0.3

[ 31.95 WXDAI ]

@gitcoindev

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	5	31.95

Conversation Incentives

Comment	Formatting	Relevance	Reward
I may be able to fix this today, will try to set up and reproduc...	1.9	0.56	1.9
> you think you can have this one @gitcoindev ? Hi @molecula4...	7.4	0.53	7.4
Ah sure. I also see @pavlovcik has been experimenting in paralle...	2	0.58	2
Btw. I am not sure if I can see in the 'matrix' correctly today ...	11.35 code: count: 1 score: "0.25" words: 0	0.78	11.35
I also downloaded [json](https://api.github.com/repos/ubiquity/u...	9.3 a: count: 1 score: "0.25" words: 1 code: count: 1 score: "0.25" words: 0	0.78	9.3

ubiquibot / comment-incentives