Welcome to the Synthia subnet, a bleeding-edge initiative to accelerate the open-source AI space. Our mission is to harness the power of Commune's decentralized incentive markets to produce a continuous stream of synthetic training data with verified quality at scale.
Synthia is utilizing the state-of-the-art Anthropic Claude3 API to generate open-ended high-quality and diverse synthetic in-depth explanations of subjects picked from the Opus latent space based on varying esotericity, with varying target-audience, level of detail and abstraction at scale.
While any model or API can theoretically mine in the subnet, the validation is designed to target Claude3-level quality, due to its substantially superior ability to generate the desired synthetic data. Hence, we advise mining with the Claude3 API, although support for OpenAI's API is available.
In the rapidly evolving world of artificial intelligence, synthetic data has emerged as a crucial component in the training of advanced models. By utilizing the state-of-the-art Anthropic Claude3 API, we can generate open-ended subject-unconstrained high-quality and diverse synthetic in-depth explanations.
Major AI labs have already recognized the potential of synthetic data and are actively utilizing it to enhance their models. However, access to such data remains limited for the broader open-source community. The Synthia subnet aims to change that.
By harnessing the power of Commune's decentralized crypto-economic incentives, we aim to create the largest reliably high-quality synthetic intelligence dataset in the world that will serve as a catalyst for innovation in the Open-Source AI space.
Join us on this important journey as we distill the Closed-Source intelligence right into the hands of the Open-Source Community!
Make sure you are on the latest CommuneX version.
pip install communex --upgrade
docker pull ghcr.io/agicommies/synthia:9d23f1f
docker run -v ~/.commune:/root/.commune -it [-p <port>:<port>] ghcr.io/agicommies/synthia:9d23f1f
poetry shell
to enter the enviromentdocker attach <id>
docker ps
-p <port>:<port>
) to bind them to your host machine.-e <VARIABLE>=<value>
.
e.g docker run -e ANTHROPIC_API_KEY=<your-anthropic-api-key> -v ~/.commune:/root/.commune -it ghcr.io/agicommies/synthia:9d23f1f
sudo apt install python3
poetry install
poetry shell
nix develop
or setup
direnv to automatically load the environment when you
enter the directory.poetry install
direnv
, just re-entering the directory will do the trick.direnv reload
poetry shell
to enter the Python environment.Get an API key from Anthropic.
Create a file named config.env
in the env/
folder with the following
contents (you can also see the env/config.env.sample
as an example):
ANTHROPIC_API_KEY="<your-anthropic-api-key>"
OPENROUTER_API_KEY="<your-openrouter-api-key>"
ANTHROPIC_MODEL=claude-3-opus-20240229
ANTHROPIC_MAX_TOKENS=1000
ANTHROPIC_TEMPERATURE=0.5
Alternatively, you can set up those values as enviroment variables. Note that you just need to provide the key to the provider that you're going to use
Serve the miner:
Make sure to be located in the root of synthia repository
cd synthia
Proceed with running the miner:
python3 -m synthia.miner.cli <your_commune_key> --ip 0.0.0.0 [--port]
Alternatively, if you want to run a openrouter miner:
python3 -m synthia.miner.cli <your_commune_key> --ip 0.0.0.0 [--port] --provider openrouter
The ip is passed as 0.0.0.0 to accept outside connections, since the default, 127.0.0.1 accepts only local connections. Synthia has the netuid 3. Key is a name of your commune wallet/key. If you don't have a wallet, generate one by running
comx key create <name>
Note: you need to keep this process alive, running in the background. Some options are tmux, pm2 or nohup.
Example using pm2
pm2 start "comx module serve synthia.miner.anthropic.AnthropicModule <key> --subnets-whitelist <synthia netuid> --ip 0.0.0.0" --name <name>
Finally register the module on the Synthia subnet:
comx module register <name> <your_commune_key> --ip <your-ip-address> --port <port> --netuid <synthia netuid>
Make sure to serve and register the module using the same key.
If you are not sure about your public ip
address:
curl -4 https://ipinfo.io/ip
Current <synthia netuid>
is 3. If you want to check for yourself, you can run:
comx subnet list
And look for the name synthia
Get an API key from Anthropic.
Gen an API key for embeddings from OpenAi
Create a file named config.env
in the env/
folder with the following contents (you can also see the env/config.env.sample
as an example):
ANTHROPIC_API_KEY="<your-anthropic-claude-api-key>"
OPENROUTER_API_KEY="<your-openrouter-api-key>"
ANTHROPIC_MODEL=claude-3-opus-20240229
ANTHROPIC_MAX_TOKENS=1000
ANTHROPIC_TEMPERATURE=0.5
OPENAI_API_KEY="<your-openai-api-key>"
Alternatively, you can set up those values as enviroment variables.
Register the validator
Note that you are required to register the validator first, this is because the validator has to be on the network in order to set weights. You can do this by running the following command:
comx module register <name> <your_commune_key> --netuid <synthia netuid>
The current synthia netuid is 3.
Serve the validator
python3 -m synthia.cli <your_commune_key> [--call-timeout <seconds>] [--provider <provider_name>]
The default value of the --call-timeout
parameter is 65 seconds.
You can pass --provider openrouter to run using openrouter provider
Note: you need to keep this process alive, running in the background. Some options are tmux, pm2 or nohup.
If you want to run up to ~10+ miners / validators
Eden has provided a new bash script that will walk you through the process of launching a validator or miner for a simpler and streamlined process.
Allow commands to be executed by the script:
chmod +x scripts/launch.sh
Run the launcher:
bash scripts/launch.sh
Just follow the prompts after that.
The launch script will prompt you step by step through the process of launching a validator or miner or both and execute the required commands without having to know details about the CLI.
Be aware that the launcher does execute commands that make changes on the block chain including balance transfers and module registration. Be sure you know what you'd like to do before using this tool as some actions cannot be undone. This tool is provided free of charge as is and with no warranty or guarantee. Use at your own risk.