search

567-labs / fastllm

A collection of LLM services you can self host via docker or modal labs to support your applications development
MIT License
181 stars 23 forks source link

Creating a modal script that will download the entire wikipedia dataset and embed it #20

Closed ivanleomk closed 9 months ago

jxnl commented 9 months ago

can you add a title so i know the goal of this pr?

jxnl commented 9 months ago

i updated batch size. can you make sure to use ruff on save?

also best to move the saving to a function, lets use a dataclass to organize the response objects too

jxnl commented 9 months ago

https://docs.wandb.ai/guides/track/log/distributed-training#use-wb-service-to-avoid-common-distributed-training-issues

ivanleomk commented 9 months ago

Closing this PR because we're deprioritizing wandb support for now