🤖 Identify an LLM Quick Win

[julialawrence]

User Story

As an Analytical Platform customer I would like a sandbox environment in which I can experiment with large language models and other generative ai models in a secure way so that I can access the value of this technology.

Value / Purpose

Quite a few teams around MOJ are running generative ai experiments at the moment without a clear idea of where those can be hosted. As such, AP could act as a platform for this exploration.

Useful Contacts

@julialawrence

User Types

AP Users

Hypothesis

If we provide a sandbox environment, we will enable the development of this technology to the benefit of the whole of MOJ. If we provide a sandbox environment, this will serve as a basis for research into a permanent productionised solution.

Proposal

Identify a quick win.

~We are about to launch a vscode tool as part of the AP offering, a long-requested feature. As proposed by @Gary-H9, we could set up a restricted release with ollama or similar tool rolled into the vscode environment. That said, as this is a sandbox, it would be highly undesirable to allow access to actual AP data. Therefore, the proposal is as following:~

• ~A restricted vscode release that deploys either in the AP or the DPAT EKS cluster with~
• ~Custom role that allows access to only one bucket~
• ~All users of the restricted release have access to the same bucket and everyone is able to see its contents (to prevent people loading up restricted data in it.~
• ~With minimal support~

The above is only one proposal. The goal of this story is to agree a quick win even if it isn't the above.

Additional Information

~https://ollama.com/~ https://mojdt.slack.com/archives/C0567L0FHAN/p1708962053031969

Definition of Done

• [x] Quick win identified
• [x] Implementation agreed
• [x] Implementation stories raised
• [x] ADR captured

[Gary-H9]

Identified: Bedrock for the quick win. Implementation:

• @michaeljcollinsuk control panel to add bedrock policy - please link story
• enable the above for a specific user

Stories to be raised:

• Create new ticket for .NET8 addition to vscode / release / deployment.
• Document the release process.
• ADR Capture

[julialawrence]

3473 <-- Linked feature request.

[julialawrence]

@carlosMOJ This is the story to track the work.

[michaeljcollinsuk]

Reopening as the ticket was closed early when the Control Panel changes were merged

[michaeljcollinsuk]

12/3/24 Deployed to dev for testing, where we have enabled a number of users so that they can use it in dev as part of the data engineering hackathon. Also tested enabling for myself, and I was able to query Bedrock endpoints using boto3 calls in Visual Studio Code.

The control panel changes have been deployed to production, however the policy needs to be created in the data-prod account before we can enable a user.

[julialawrence]

Raised the PR for Bedrock policy. Once deployed and @carlosMOJ appraised, story can be closed.

[carlosMoJ]

looking forward to this!!

[michaeljcollinsuk]

@carlosMoJ to get started, can you follow step 4 of the AP user guidance to get access to the Analytical Platform and logged in to the control panel. I'll need to enable Bedrock access for you after you have logged in for the first time, so let me know when you have done that and then I'll provide next steps. Any questions feel free to message on Slack.

[michaeljcollinsuk]

Closing as completed as Bedrock has been enabled for Carlos to use via the existing VS code tool.

Link to proposed ADR. This will need to be updated when we are ready to make a decision on offering Bedrock more widely in the AP.