search

CLARIN-PL / embeddings

Embeddings: State-of-the-art Text Representations for Natural Language Processing tasks, an initial version of library focus on the Polish Language
https://clarin-pl.github.io/embeddings/
MIT License
36 stars 3 forks source link

Add huggingface based models exporter #289

Closed ktagowski closed 1 year ago

ktagowski commented 1 year ago

TODO:

Example usage (POC)


from embeddings.config.lightning_config import LightningBasicConfig
from embeddings.pipeline.lightning_classification import LightningClassificationPipeline
from embeddings.utils.hf_model_exporter import HuggingFaceModelExporter

pipeline = LightningClassificationPipeline(
    embedding_name_or_path="hf-internal-testing/tiny-albert",
    devices="auto",
    accelerator="cpu",
    output_path=".",
    dataset_name_or_path="clarin-pl/polemo2-official",
    input_column_name=["text"],
    target_column_name="target",
    config=LightningBasicConfig(max_epochs=1)
)
result = pipeline.run()
HuggingFaceModelExporter("hf_model_export_test").export_model_from_pipeline(pipeline)
netlify[bot] commented 1 year ago

Deploy Preview for embeddingsclarinpl canceled.

Name Link
Latest commit 6a85b46b15147a1f6b09f4d329e31a8c00571b35
Latest deploy log https://app.netlify.com/sites/embeddingsclarinpl/deploys/64397af58b811c0008181aa5