Plotreader

Use LLMs to generate and read plots. Combine them into a teacher-student pair to improve performance.

Problem

Build a tool that can aggregate information from scientific figures into structured data. For example, extract the functional properities of different variants of opsins - both natual and engineered.

Approach

Reader

Design an agent that can take in a paper and extract the quantitative results into structured data.

Generator

Design an agent that takes in a data scenario and paper/figure examples and then outputs structured data, the data plotted into a figure, and several quantitative questions about the figure with difficulty rankings.

Curriculum Design

For now, I'm prototyping independent plot readers and generators, but the original idea was to combine the reader and generator into an RL training loop with the reader as student and the generator as teacher.

Results

Plot Reading

Input Figure (from Machine learning-guided channelrhodopsin engineering enables minimally invasive optogenetics ):

Step 1: Extract experiments from figure.

Prompt:

prompt = "For each plot in each panel determine the experiment in terms of independent variables (IVs) and dependent variable (DV)."

Structured Output:

Panel: a
    Plot: Current traces
        independent_variables=['Time', 'ChR variant'] dependent_variables=['Current']
Panel: b
    Plot: Photocurrent strength with different wavelength excitation
        independent_variables=['ChR variant', 'Wavelength', 'Current type (Peak/Steady state)'] dependent_variables=['Photocurrent']
Panel: c
    Plot: Off-kinetics decay time
        independent_variables=['ChR variant'] dependent_variables=['Off-kinetics decay time (τoff)']
    Plot: Current traces for select variants
        independent_variables=['Time', 'ChR variant'] dependent_variables=['Current']
Panel: d
    Plot: Wavelength sensitivity
        independent_variables=['Wavelength', 'ChR variant'] dependent_variables=['Normalized photocurrent']
Panel: e
    Plot: Peak photocurrent vs Intensity
        independent_variables=['Light intensity', 'ChR variant'] dependent_variables=['Peak photocurrent']
    Plot: Steady-state photocurrent vs Intensity
        independent_variables=['Light intensity', 'ChR variant'] dependent_variables=['Steady-state photocurrent']

Step 2: Extract values for independent and dependent variables from a single plot

Prompts:

prompt = """In Panel {panel_name}, plot number {plot_ind}, what values are taken by the independent variable {ind_var_name}?
If the variable is not quantitative (like an image), only set the name field of IndependentVariable.
Return your answer as structured data.
""".format(
    panel_name = figure.panels[panel_ind].name, 
    plot_ind=plot_ind+1, 
    ind_var_name = iv
)

prompt = """In Panel {panel_name}, plot number {plot_ind}, what statistics used to quantify the deependant variable {dep_var_name} like mean or SEM?
If the variable is not quantitative (like an image), only set the name field of DependentVariable.
Return your answer as structured data.
""".format(
    panel_name = figure.panels[panel_ind].name, 
    plot_ind=plot_ind+1, 
    dep_var_name = dv
)

Structured Output:

The values for Wavelength are not correct. Hoping to resolve this by allowing RAG over full paper.

Panel d, Independent Vars:
name='Wavelength' values=[400, 450, 500, 550, 600, 650] unit='nm'
name='ChR variant' values=['ChRger1', 'ChRger2', 'ChRger3', 'ChR_25_9', 'ChR_15_10', 'CsChrimR', 'ChRiff'] unit='None'

Panel d, Dependent Vars:
name='Normalized photocurrent' statistics=['mean', 's.e.m.'] unit='None'

Step 3: Extract dependent variable statistics for each condition

Prompt:

columns = {iv.name: pd.Series() for iv in ind_vars}
columns.update({dv.name: pd.Series()})
df = pd.DataFrame(columns)

prompt = """In Panel {panel_name}, what are the values of the {dep_var_stat} for the dependent variable {dep_var_name}?
If the variable is not quantitative (like an image), only set the name field of IndependentVariable.
Get the value for each condition. To help, here are the values the independent variable takes:
{ind_vars}
IMPORTANT:
    THE EXPECTED VALUES FOR THE INDEPENDENT VARIABLES MAY BE INCORRECT. DO YOUR BEST TO ESTIMATE THE VALUE FOR THE INPUTS FROM THE PLOT.
The column schema is the following: {schema}.
""".format(
    panel_name = figure.panels[panel_ind].name, 
    dep_var_stat = dv.statistics[0],
    dep_var_name = dv.name,
    ind_vars = ind_vars,
    schema = ", ".join(df.columns),
)

Plotting extracted data:

Note: colors were not extracted and so not matched. Not a perfect result - some of this may come from incorrectly extracting the Wavelength values.

Extracted	Source

Plot Generation

Input

Prompt with both text and example figures.

generator = PlotGenerator(
        storage_dir = "~/dev/plotreader/storage", 
    )

generator.generate(
    data_scenario = (
        "Scientists are desigining new opsins by mutating existing ones. " +
        "They then measure the currents produced when exciting with different wavelengths of light. " + 
        "They plot comparisons of the different mutants and wild-type as function of these wavelengths."
    ),
    examples_dir='~/dev/plotreader/real_figures/opsins'
)

Two of the example figures it was given (from Machine learning-guided channelrhodopsin engineering enables minimally invasive optogenetics).

Example 1	Example 2

Output

Generated Figures

Example 1	Example 2

Generated Questions (with Output Figure Example 1)

The questions were saved into a .json file

[
  {
    "difficulty": "Easy",
    "question": "Which opsin variant shows the highest peak photocurrent at the higher light intensity (0.8 mW mm^-2)?",
    "answer": "ChRger1 shows the highest peak photocurrent at the higher light intensity, with a value of approximately 1500 pA."
  },
  {
    "difficulty": "Medium",
    "question": "Comparing ChR2 and ChRger3, what is the approximate percentage increase in peak photocurrent at the lower light intensity (8 \u00d7 10^-3 mW mm^-2)?",
    "answer": "At the lower light intensity, ChR2 has a peak photocurrent of about 100 pA, while ChRger3 has a peak photocurrent of about 600 pA. The percentage increase is approximately (600 - 100) / 100 * 100 = 500%. ChRger3 shows a 500% increase in peak photocurrent compared to ChR2 at the lower light intensity."
  },
  {
    "difficulty": "Hard",
    "question": "Based on the spectral sensitivity plot, at which wavelength does the difference in normalized photocurrent between ChRger3 and ChR2 appear to be the greatest, and what is the approximate magnitude of this difference?",
    "answer": "The difference in normalized photocurrent between ChRger3 and ChR2 appears to be greatest at around 480-490 nm. At this point, ChRger3 has a normalized photocurrent of about 0.75, while ChR2 has a normalized photocurrent of about 0.55. The magnitude of the difference is approximately 0.75 - 0.55 = 0.2, or a 20% difference in normalized photocurrent."
  }
]

Generated Data (with Output Figure Example 1)

Three .csv files were generated that contain the data used to create the figure.