UI RefExp with Donut

Fine tuning Donut transformers for UI Referring Expressions task

This repo contains fine tuning code and a playground for the UI Referring Expression task - using natural language to reference UI components in a screenshot.

• Fine tuning notebook: Fine_tune_Donut_on_UI_RefExp.ipynb.
• Inference playground: Inference_Playground_Donut_UI_RefExp_Gradio.ipynb