We have this new /workflows/code/llm-assisted-codebase-to-pn-amr endpoint. This endpoint takes in a zip just like the codebase-to-pn-amr, but we use the LLM linespan functionality to automatically slice the code and send the dynamics as a snippet to the snippets-to-pn-amr endpoint. It would be cool if this endpoint and the regular codebase one could both be tested. We are working on our own model to do the linespan functionality well, and once we have an initial version of that, it will be incorporated into the standard codebase endpoint, so comparing the two will be nice to see how the coverage of our own model compares to using GPT in this new llm-assisted one.
Current Limitations (things we are working to fix/improve):
It is kind of slow and we have been working on making it's response time more consistent, so I would not test the full penn-chime repo on it for now until we optimize it some more. It will likely timeout.
Hidden files / artifacts that come from zipping files on a mac make it mess it up currently. but we are working on fixing that.
All these endpoints are demo'd and explained here. Including a sample calculation for coverage in the AMR-enrichment case. (edited)
From @Free-Quarks:
We have this new /workflows/code/llm-assisted-codebase-to-pn-amr endpoint. This endpoint takes in a zip just like the codebase-to-pn-amr, but we use the LLM linespan functionality to automatically slice the code and send the dynamics as a snippet to the snippets-to-pn-amr endpoint. It would be cool if this endpoint and the regular codebase one could both be tested. We are working on our own model to do the linespan functionality well, and once we have an initial version of that, it will be incorporated into the standard codebase endpoint, so comparing the two will be nice to see how the coverage of our own model compares to using GPT in this new llm-assisted one.
Current Limitations (things we are working to fix/improve):
All these endpoints are demo'd and explained here. Including a sample calculation for coverage in the AMR-enrichment case. (edited)