18.12.-08.01.

[bowl-of-rais]

Miscellaneous

• [x] #23 (@d-isasterhub)
• [x] #19 (@bowl-of-rais)
• [x] #26 (@d-isasterhub)
• [x] #35 (@d-isasterhub)
• [x] make different output files for different parameters (@bowl-of-rais) (see #35 !)
  • [x] profiling levels
  • [x] prompt variations

Change profiling order 🔀

finding: profiling too early could cause hallucinations - let LLM describe first, then constrain behavior by profiling approach: ask to describe the heatmap first (once for all fifty users), then give human features

• [x] #24 (@d-isasterhub)
• [x] #32 (@bowl-of-rais)

Simulation Experiments 🏃

• [x] #27 (@d-isasterhub)
• [ ] baseline without any profiling (@d-isasterhub)
• [ ] easier questions (@d-isasterhub)

Agreement questions 👍

profiling so far could include bird descriptions, accuracy

• [x] #25 (@d-isasterhub)
• [x] #36
• [x] #33 (@bowl-of-rais)
• [x] #41 (@d-isasterhub)

[d-isasterhub]

Different profiling order notes

@bowl-of-rais

This is what the prompting currently looks like:

The idea is the following:

First, get the heatmap description that will be reused for every (user, question) pair. This should only be done once, so please save the answer in a file. When running the script, there should be a check if we already have a heatmap description:

SYSTEM ...

USER Attached is an image that consists of two sub-images. The top of the given image displays the original image of a bird and the bottom displays the same image combined with a heatmap that was generated by an explainable artificial intelligence model to explain the species predicted for the image. Describe what areas of the bird are highlighted in red by the heatmap. Keep the number of tokens low.

response = openai.ChatCompletion.create(...)

Second, use the description to get a bird species classification prediction by chat completion:

SYSTEM ...

USER Attached is an image that consists of two sub-images. The top of the given image displays the original image of a bird and the bottom displays the same image combined with a heatmap that was generated by an explainable artificial intelligence model to explain the species predicted for the image. Describe what areas of the bird are highlighted in red by the heatmap.

ASSISTANT [description]

USER You believe that the classification model distinguishes between four possible species classes based on the following features:

• Rhinoceros Auklets: HEATMAP_FEATURES_RA
• Least Auklets: HEATMAP_FEATURES_LA
• Parakeet Auklets: HEATMAP_FEATURES_PA
• Crested Auklets: HEATMAP_FEATURES_CA Based on the descriptions and the areas highlighted by the heatmap, which bird species do you think was predicted for the given image? Choose one of the following options for your answer:
• Crested Auklet
• Least Auklet
• Parakeet Auklet
• Rhinoceros Auklet For each bird description, explain why it might or might not be that species of bird based on your heatmap description. Conclude your answer by stating only the selected option in the last line of your answer. Keep the number of tokens low.

response = openai.ChatCompletion.create(
            model = "gpt-4-vision-preview",
            max_tokens = 400,
            messages = 
                (get_msg(role="system", prompt=...) if profiling_level == 'full' else []) +\
                get_msg_with_image(role="user", prompt=..., image=...) +\
                get_msg(role="assistant", prompt=...) +\
                get_msg(role="user", prompt=...) 
        )

Mind that "Keep the number of tokens low." appears in different modalities!

[d-isasterhub]

Maybe, we can somehow save more tokens since we already have the description. Do you have any ideas?

[d-isasterhub]

The "Keep the number of tokens low" is still fixed for the other prompting order (see prompts).