Qualitative Analysis: Filter Method Crash

0x4007 commented 8 months ago

https://github.com/ubiquibot/comment-incentives/actions/runs/7548284630/job/20550022286#step:5:1

Looks like the qualitative analysis of comments is shaky.

The error states it's crashing on the filter method. I know I implemented that part a bit sloppy, where a single function handles two scenarios, one of which I nest arrays too deeply. Please handle the filter method crashing.

Additional Context

This implementation will eventually need a brand new approach, but this approach is quick and dirty.
Currently I am asking GPT 3.5 to evaluate an array of the comment bodies, prompt below
Even with a low temperature, it will return results that are not of the expected length. There is a hard check for this, and if the length does not exactly match the conversation length, then it will try the whole procedure again (a brute force method that seems to work and is cheap enough for now.)

Prompt

I need to evaluate the relevance of GitHub contributors' comments to a specific issue specification. Specifically, I'm interested in how much each comment helps to further define the issue specification or contributes new information or research relevant to the issue. Please provide a float between 0 and 1 to represent the degree of relevance. A score of 1 indicates that the comment is entirely relevant and adds significant value to the issue, whereas a score of 0 indicates no relevance or added value. Each contributor's comment is on a new line.

[ Array of comment bodies ]

To what degree are each of the comments in the conversation relevant and valuable to further defining the issue specification? Please reply with an array of float numbers between 0 and 1, corresponding to each comment in the order they appear. Each float should represent the degree of relevance and added value of the comment to the issue. The total length of the array in your response should equal exactly ${array.length} elements.

Source

https://github.com/ubiquity/ubiquibot/issues/909#issuecomment-1894626568

0x4007 commented 8 months ago

@whilefoo are you able to look into this?

ubiquibot[bot] commented 8 months ago

@wannacfuture the deadline is at 2024-01-19T05:07:16.519Z

ubiquibot[bot] commented 7 months ago

# No linked pull requests to close

0x4007 commented 7 months ago

Now with tests this should be quick to fix.

0x4007 commented 7 months ago

/start

ubiquibot[bot] commented 7 months ago



Warning!  This task was created over 33 days ago. Please confirm that this issue specification is accurate before starting.
Deadline Mon, Feb 19, 11:20 AM UTC

Registered Wallet
0x4007CE2083c7F3E18097aeB3A39bb8eC149a341d

Tips:

Use /wallet 0x0000...0000 if you want to update your registered payment wallet address.
Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the task.

0x4007 commented 7 months ago

https://github.com/ubiquibot/comment-incentives/issues/8#issuecomment-1952131238

ubiquibot[bot] commented 6 months ago

# Issue was not closed as completed. Skipping.

ubiquibot / comment-incentives