opensearch-project / opensearch-py-ml

Apache License 2.0

34 stars 64 forks source link

[FEATURE] enhance model_uploader workflow to support MIT-licensed models from huggingface #388

Closed zhichao-aws closed 4 months ago

zhichao-aws commented 7 months ago

Description

Code changes to enhance model upload workflow to support MIT licensed models from huggingface

Issues Resolved

387

Check List

[ ] New functionality includes testing.
- [ ] All tests pass
[ ] New functionality has been documented.
- [ ] New functionality has javadoc added
[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

zhichao-aws commented 7 months ago

The workflow works well in my repo GH actions (I commented out the manual approve part) They traced model and upload to S3 and create PR in my repo link: https://github.com/zhichao-aws/opensearch-py-ml/actions/runs/8843390731 https://github.com/zhichao-aws/opensearch-py-ml/actions/runs/8843323304

autocut update PR: https://github.com/zhichao-aws/opensearch-py-ml/pull/7 https://github.com/zhichao-aws/opensearch-py-ml/pull/8

zhichao-aws commented 7 months ago

With this workflow, to upload BGE-small models, we trigger it with these settings, and the last input MIT license url should be : https://github.com/FlagOpen/FlagEmbedding/raw/master/LICENSE

zhichao-aws commented 7 months ago

With this workflow, to upload BGE-small models, we trigger it with these settings, and the last input MIT license url should be : https://github.com/FlagOpen/FlagEmbedding/raw/master/LICENSE

To upload BGE-base, change the model id to BAAI/bge-base-en-v1.5

zhichao-aws commented 7 months ago

One screenshot for the upload workflow at my GH repo

zhichao-aws commented 6 months ago

This PR is ready for review.

zhichao-aws commented 6 months ago

@dhrubo-os Any clue why the IT fails? It seems not related to the change, is it a flaky test?

dhrubo-os commented 6 months ago

@dhrubo-os Any clue why the IT fails? It seems not related to the change, is it a flaky test?

Yeah, I think that was a flaky test.

zhichao-aws commented 4 months ago

close it for the issue of BGE training data. Feel free to re-open it when we need to upload other MIT models