sotopia-lab / sotopia

Sotopia: an Open-ended Social Learning Environment (ICLR 2024 spotlight)
https://docs.sotopia.world
MIT License
127 stars 16 forks source link

Support vision input for generation utils. #119

Open ProKil opened 1 week ago

ProKil commented 1 week ago

This PR supports vision/image input for generation utils, which would allow interaction to be grounded in vision inputs. This PR relies on #116

📑 Description

Uses Langchain's ImagePromptTemplate to add image url to chat messages.

✅ Checks

ℹ Additional Information

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 41.66667% with 7 lines in your changes missing coverage. Please review.

Project coverage is 59.96%. Comparing base (8d9b9be) to head (f8e59c0).

@@            Coverage Diff             @@
##             main     #119      +/-   ##
==========================================
- Coverage   60.03%   59.96%   -0.07%     
==========================================
  Files          47       47              
  Lines        2402     2413      +11     
==========================================
+ Hits         1442     1447       +5     
- Misses        960      966       +6     
Files Coverage Δ
tests/generation_utils/test_generation.py 68.42% <ø> (ø)
sotopia/generation_utils/generate.py 51.53% <41.66%> (-0.27%) :arrow_down: