Adds the formatters module and the JSONFormatter module, which in turn uses JSONFormer to better constrain the output of huggingface based models.
Performance delta on a 2023 MacBook Pro with Apple M3 Max:
With formatter: 1.48 s ± 23.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Without formatter: 738 ms ± 130 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
These deltas may be a result of...
Increased prompt size when using the JSONFormatter.
'Without Formatter' breaking early because of an invalid JSON return. (I.e., rather than iterate multiple times to complete generation, it breaks as soon as invalid JSON is encountered. This is consistent with observations of the returned data.)
Further investigation is planned. Additionally, tests currently are using gpt-2 as a sample model, but we shouldn't be doing any HF imports in the tests. When core-schema-impl is merged we will have access to the new testing harnesses, including the mock models.
notion-workspace[bot]
commented
1 week ago
Adds the formatters module and the JSONFormatter module, which in turn uses JSONFormer to better constrain the output of huggingface based models.
Performance delta on a 2023 MacBook Pro with Apple M3 Max:
These deltas may be a result of...
'Without Formatter' breaking early because of an invalid JSON return. (I.e., rather than iterate multiple times to complete generation, it breaks as soon as invalid JSON is encountered. This is consistent with observations of the returned data.)
Further investigation is planned. Additionally, tests currently are using gpt-2 as a sample model, but we shouldn't be doing any HF imports in the tests. When core-schema-impl is merged we will have access to the new testing harnesses, including the mock models.