search

alan-turing-institute / ARC-MTQE

Critical Error Detection for Machine Translation
MIT License
1 stars 0 forks source link

Investigate use of Azure for LLMs #6

Closed radka-j closed 6 months ago

radka-j commented 7 months ago

Look at: https://learn.microsoft.com/en-us/azure/ai-services/openai/overview

Pricing information: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/

joannacknight commented 6 months ago

I created an OpenAI service on Azure and was able to use this in the code to compare the source and target sentences. However, the service has some content filtering in place (https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython-new). This is preventing a response from being returned when there is toxic language used in the source / target text. We discussed some options:

  1. Use the 'regular' OpenAI API instead of the Azure service
  2. Apply to be able to modify the content filtering - though I am not sure what level this would be applied to, e.g., just the ARC subscription? Or all Turing accounts? Also, don't know the timescales involved or the likelihood of it being accepted
  3. Look into using a different LLM
  4. Only test the LLM approach on sentences with non-toxic content

We've decided to go with option 1, although we don't know if there is a risk of the account getting blocked or moderated if we are sending through lots of toxic content.

joannacknight commented 6 months ago

Created issue #18 to follow on from this.