helicalAI / helical

This repository contains the python package for Helical
https://www.helical-ai.com/
GNU Affero General Public License v3.0
53 stars 5 forks source link

Include benchmarking #36

Closed bputzeys closed 4 months ago

bputzeys commented 4 months ago
bputzeys commented 4 months ago

In this commit, I fixed the seed of the random number generator: https://github.com/helicalAI/helical/pull/36/commits/b04735b724af4e3be82904146e10934cc43d4763 I ran a few checks with and without the fixed seed:

# frozen head model & no fixed seed
{'UCE with saved NN': {'accuracy': 0.9, 'precision': 0.6507936507936508, 'f1': 0.648018648018648, 'recall': 0.6481481481481483}}
{'UCE with saved NN': {'accuracy': 0.9, 'precision': 0.7321428571428572, 'f1': 0.7243107439755867, 'recall': 0.7198067632850241}}
{'UCE with saved NN': {'accuracy': 0.9, 'precision': 0.6507936507936508, 'f1': 0.648018648018648, 'recall': 0.6481481481481483}}
{'UCE with saved NN': {'accuracy': 0.92, 'precision': 0.6666666666666666, 'f1': 0.6589147286821705, 'recall': 0.6521739130434783}}

# frozen head model & fixed seed
{'UCE with saved NN': {'accuracy': 0.92, 'precision': 0.6666666666666666, 'f1': 0.6589147286821705, 'recall': 0.6521739130434783}}
{'UCE with saved NN': {'accuracy': 0.92, 'precision': 0.6666666666666666, 'f1': 0.6589147286821705, 'recall': 0.6521739130434783}}
{'UCE with saved NN': {'accuracy': 0.92, 'precision': 0.6666666666666666, 'f1': 0.6589147286821705, 'recall': 0.6521739130434783}}
{'UCE with saved NN': {'accuracy': 0.92, 'precision': 0.6666666666666666, 'f1': 0.6589147286821705, 'recall': 0.6521739130434783}}

It seems to work as expected.