For a quick overview, please see below the example I extracted from the readme.
A few notes:
I tried to keep the surface of the API as tight as possible
The main user facing classes are GptTagger, TaskConfig and LabelledNode. It's probably a good
idea to spend the most attention there
I added a lot of tests, which should help with evolving the code
I made a lot of assumptions. Adding actual use cases will help to iron them out
The code has a couple of integration tests, which are currently excluded from
the build (have to be run manually locally, and require a working API Key)
This PR is pretty large, appologies.
Not sure how you guys usually deal with large PRs:
I can break the PR down if you have suggestions
In previous companies, we'd do a live PR walkthrough (on Zoom) to lower the overhead. It's definitely an option if you folks are interested.
Thank you
Example
Note: An API key needs to be set. See below.
from pprint import pprint
from nxontology_ml.efo import get_efo_otar_slim
from nxontology_ml.gpt_tagger import TaskConfig, GptTagger
from nxontology_ml.utils import ROOT_DIR
# Create a config for EFO nodes labelling
config = TaskConfig(
name="precision",
prompt_path=ROOT_DIR / "prompts/precision_v1.txt",
openai_model_name="gpt-4",
node_attributes=["efo_id", "efo_label", "efo_definition"],
)
# Get a few EFO nodes
nxo = get_efo_otar_slim()
nodes = (nxo.node_info(node) for node in sorted(nxo.graph)[:20])
# Get their labels
tagger = GptTagger.from_config(config)
for ln in tagger.fetch_labels(nodes):
print(f"{ln.node_efo_id}: {ln.label}")
# Inspect metrics
print("\nTagger metrics:")
pprint(tagger.get_metrics())
You should get an output similar to:
DOID:0050890: medium
DOID:10113: low
DOID:10718: low
DOID:13406: medium
DOID:1947: low
DOID:7551: low
EFO:0000094: high
EFO:0000095: high
EFO:0000096: medium
EFO:0000174: medium
EFO:0000178: medium
EFO:0000180: medium
EFO:0000181: medium
EFO:0000182: medium
EFO:0000183: medium
EFO:0000186: medium
EFO:0000191: medium
EFO:0000195: low
EFO:0000196: medium
EFO:0000197: medium
Tagger metrics:
Counter({'ChatCompletion/total_tokens': 3187,
'ChatCompletion/prompt_tokens': 3009,
'ChatCompletion/completion_tokens': 178,
'Cache/get': 20,
'Cache/misses': 20,
'ChatCompletion/records_processed': 20,
'Cache/set': 20,
'ChatCompletion/create_requests': 1})
Note that the requests are cached in .cache/precision_v1.ldb, so if you run the same code again you will
see the same labels, but different metrics:
This PR adds a GptTagger class.
For a quick overview, please see below the example I extracted from the readme.
A few notes:
GptTagger
,TaskConfig
andLabelledNode
. It's probably a good idea to spend the most attention thereThis PR is pretty large, appologies. Not sure how you guys usually deal with large PRs:
Thank you
Example
Note: An API key needs to be set. See below.
You should get an output similar to:
Note that the requests are cached in
.cache/precision_v1.ldb
, so if you run the same code again you will see the same labels, but different metrics: