DARPA-ASKEM / knowledge-middleware

TA1 extraction pipelines
Apache License 2.0
3 stars 0 forks source link

Add support for LLM assisted code to AMR extraction #174

Closed brandomr closed 10 months ago

brandomr commented 10 months ago

Addresses #167

@Free-Quarks note that the LLM assisted code to AMR extraction isn't explicitly tested separately but we could do so down the road. Right now the way the test harness works:

  1. checks if dynamics exist for the code in the scenario and if so uses those
  2. otherwise, it just falls back to zipfile extraction

Since #2 defaults to using the LLM assistant in our knowledge-middleware service it will basically just do that. We could set it up so that for a given test scenario you can "turn off LLM assistance" but then we'd need to duplicate the scenario (e.g. have 2 SIDARTHE Code scenarios). We'd do this by adding llm_assisted.txt which would just be a bool flag for the scenario in the scenario directory and read that into the test harness runner. Let me know what you think.

@YohannParis tagging you as reviewer just for your visibility--no changes are needed on the HMI end; the default code to AMR (if no dynamics are specified) is to use LLM assisted mode now :)

codecov[bot] commented 10 months ago

Codecov Report

Attention: 111 lines in your changes are missing coverage. Please review.

Comparison is base (2c32c25) 81.78% compared to head (9fb50e2) 84.90%. Report is 85 commits behind head on main.

Files Patch % Lines
worker/operations.py 80.73% 84 Missing :warning:
worker/utils.py 88.44% 23 Missing :warning:
api/utils.py 75.00% 3 Missing :warning:
lib/settings.py 96.42% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #174 +/- ## ========================================== + Coverage 81.78% 84.90% +3.12% ========================================== Files 5 6 +1 Lines 505 828 +323 ========================================== + Hits 413 703 +290 - Misses 92 125 +33 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.