opentensor / validators

Repository for bittensor validators
https://www.bittensor.com/
MIT License
14 stars 9 forks source link

Task validator filter #112

Closed p-ferreira closed 1 year ago

p-ferreira commented 1 year ago

Adds task validator filter to check if the proposed task is being accomplished.

This PR seeks to bring more consistency to the CoT developed by the validators and the miners, preventing tasks to be performed out of their prompts.

This is an initial simple keyword check verification model that can be evolved to more sophisticated use cases with real model implementation.

mrseeker commented 1 year ago

This can easily be bypassed by avoiding to mention "Question:". You might need to alter it to lowercase only and rethink the way questions are asked. The AI could also use "reason" or "thought" or something in those lines, or even avoid the words Question or Answer by giving an alternative word with a similar meaning.

p-ferreira commented 1 year ago

This can easily be bypassed by avoiding to mention "Question:". You might need to alter it to lowercase only and rethink the way questions are asked. The AI could also use "reason" or "thought" or something in those lines, or even avoid the words Question or Answer by giving an alternative word with a similar meaning.

Well noted, I implemented the verification to be case insensitive. I totally agree with you, as the network evolves the attacks will also evolve and we will need more sophisticated ways to validate if the completion is related to the task being proposed.

As of now, this simple verification should take care of good part of the current "topic drifts" that we see popping up in the network, we will follow this issue closely to what's being generated in the network