Establish Granite Code endpoint in AWS for testing and development of recipes

ibm-granite-community / pm

Granite Community Project Management

0 stars 0 forks source link

Establish Granite Code endpoint in AWS for testing and development of recipes #6

Open adampingel opened 3 months ago

adampingel commented 3 months ago

Make these model available:

https://huggingface.co/ibm-granite/granite-34b-code-instruct

https://huggingface.co/ibm-granite/granite-20b-code-instruct

[ ] Enable and the above models in BedRock
[ ] 24/7 should be up
[ ] Provide cost estimate
[ ] initialize aws-account ai-alliance
[ ] iam: make Adam/Srikanth owners for now
[ ] Basic run book
[ ] Provide a log of how you connected that model using lang-chain (python code)

kpsrikanthibm commented 3 months ago

:: Meeting with AP: Jul 16th 2024

--> Investigate BedRock to stand these up .. (AA did this very easily) --> Can we use hugging face directly

What is the host name ? Do we need cloudfront for protection ?

https://python.langchain.com/v0.1/docs/integrations/llms/bedrock/

conversation.predict(input="Hi there!")

kpsrikanthibm commented 3 months ago

observed. it looks like we can only import model_type = llama into bedrock The model types we have are "model_type": "gpt_bigcode",

Also Requested to increase quota

https://support.console.aws.amazon.com/support/home#/case/?displayId=172202349600681

Limit name: Custom models per account
New limit value: 250.0

kpsrikanthibm commented 3 months ago

https://medium.com/@chetan.panchal270999/amazon-bedrock-custom-model-import-a-game-changer-for-foundation-models-41f6f1e9fda3#:~:text=Compatibility%3A%20Custom%20Model%20Import%20supports,integrated%20into%20the%20Bedrock%20ecosystem.

 Custom Model Import supports a range of model architectures, including Mistral, Flan, and Llama 2 and 3. 
This flexibility ensures that a wide variety of models can be integrated into the Bedrock ecosystem.

adampingel commented 2 months ago

Yesterday, @kpsrikanthibm mentioned that AWS is "going to provide more examples". I'm going to moved this to "blocked" until we get those examples.

kpsrikanthibm commented 2 months ago

After talking with Vincent.. I understood a big team is working on enabling granite models in AWS BedRock Current Progress is shared on Slack https://ibm-research.slack.com/archives/C072HAGFTE0/p1724796077343029 Currently progress is blocked as necessary TockenizerClass and Model Architecture are not SUpported in AWS BedRock