does gist token support Question Answering use-case with static context, varying question, with 24K+ context length?

[IamExperimenting]

Hi @jayelm @eltociear ,

I just read your paper "Learning to Compress Prompts with Gist Tokens" . It is an impressive idea.

I thought this approach would help me to solve my problem.

where I'm working on a use-case Question Answering, and I will be passing the static prompt to the LLM.

My question is,

1. Can this gist token method be used for the static prompt?
2. my current context length varies between 23,000 and 25,000, is it possible to perform gist token for the larger prompt?

Basically, I have a pdf file which is about 45 pages, and it remains static. I would like ask questions about the pdf, here instruction, context will remain as it is(static) only question will be changing.

please find the example prompt for your reference

Prompt = """ You're a helpful chatbot. Please read the context between and and answer the question accordingly. If the context does not contain any information about the question, please just say I don't know.

iPhone 7 - Technical Specifications Finish Rose Gold Gold Silver Black Jet Black1 (PRODUCT)RED™ Capacity2 32GB 128GB 256GB (PRODUCT)RED™ is exclusively available in 128GB and 256GB models. Size and Weight3 Height: 5.44 inches (138.3 mm) Width: 2.64 inches (67.1 mm) Depth: 0.28 inch (7.1 mm) Weight: 4.87 ounces (138 grams) Display Retina HD display 4.7-inch (diagonal) widescreen LCD Multi-Touch display with IPS technology 1334-by-750-pixel resolution at 326 ppi 1400:1 contrast ratio (typical) Wide color display (P3) 625 cd/m2 max brightness (typical) Dual-domain pixels for wide viewing angles Fingerprint-resistant oleophobic coating Support for display of multiple languages and characters simultaneously Display Zoom Reachability Splash, Water, and Dust Resistant4 Rated IP67 (maximum depth of 1 meter up to 30 minutes) under IEC standard 60529 Chip A10 Fusion chip Camera 12MP camera ƒ/1.8 aperture Digital zoom up to 5x Optical image stabilization Six‑element lens Quad-LED True Tone flash Panorama (up to 63MP) Sapphire crystal lens cover Backside illumination sensor Hybrid IR filter Autofocus with Focus Pixels Tap to focus with Focus Pixels Live Photos with stabilization Wide color capture for photos and Live Photos Improved local tone mapping Exposure control HDR for photos Auto image stabilization Burst mode Timer mode Photo geotagging Image formats captured: HEIF and JPEG Video Recording 4K video recording at 30 fps 1080p HD video recording at 30 fps or 60 fps 720p HD video recording at 30 fps Optical image stabilization for video 2x optical zoom; digital zoom up to 6x (iPhone 7 Plus only) Quad-LED True Tone flash Slo‑mo video support for 1080p at 120 fps and 720p at 240 fps Time‑lapse video with stabilization Cinematic video stabilization (1080p and 720p) Continuous autofocus video Take 8MP still photos while recording 4K video Playback zoom Video geotagging Video formats recorded: HEVC and H.264 FaceTime HD Camera 7MP camera 1080p HD video recording Retina Flash ƒ/2.2 aperture Wide color capture for photos and Live Photos HDR for photos Backside illumination sensor Auto image stabilization Burst mode Exposure control Timer mode Touch ID Fingerprint sensor built into the Home button Apple Pay Pay with your iPhone using Touch ID in stores, within apps, and on the web Send and receive money in Messages Complete purchases made with Apple Pay on your Mac Learn more about Apple Pay Cellular and Wireless Model A1660* FDD-LTE (Bands 1, 2, 3, 4, 5, 7, 8, 12, 13, 17, 18, 19, 20, 25, 26, 27, 28, 29, 30) TD-LTE (Bands 38, 39, 40, 41) TD-SCDMA 1900 (F), 2000 (A) CDMA EV-DO Rev. A  (800, 1900, 2100 MHz) UMTS/HSPA+/DC-HSDPA (850, 900, 1700/2100, 1900, 2100 MHz) GSM/EDGE (850, 900, 1800, 1900 MHz) Model A1778* FDD-LTE (Bands 1, 2, 3, 4, 5, 7, 8, 12, 13, 17, 18, 19, 20, 25, 26, 27, 28, 29, 30) TD-LTE (Bands 38, 39, 40, 41) UMTS/HSPA+/DC-HSDPA (850, 900, 1700/2100, 1900, 2100 MHz) GSM/EDGE (850, 900, 1800, 1900 MHz) All models 802.11ac Wi‑Fi with MIMO Bluetooth 4.2 wireless technology NFC with reader mode Express Cards Location Assisted GPS, GLONASS, Galileo, and QZSS Digital compass Wi-Fi Cellular iBeacon microlocation Video Calling5 FaceTime video calling over Wi‑Fi or cellular Audio Calling5 FaceTime audio Voice over LTE (VoLTE)6 Wi‑Fi calling6 Audio Playback Audio formats supported: AAC-LC, HE-AAC, HE-AAC v2, Protected AAC, MP3, Linear PCM, Apple Lossless, FLAC, Dolby Digital (AC-3), Dolby Digital Plus (E-AC-3), and Audible (formats 2, 3, 4, Audible Enhanced Audio, AAX, and AAX+) User-configurable maximum volume limit Video Playback Video formats supported: HEVC, H.264, MPEG-4 Part 2, and Motion JPEG AirPlay Mirroring, photos, and video out to Apple TV (2nd generation or later)7 Video mirroring and video out support: Up to 1080p through Lightning Digital AV Adapter and Lightning to VGA Adapter (adapters sold separately)7 Siri8 Use your voice to send messages, set reminders, and more Get intelligent suggestions on your Lock screen and in Messages, Mail, QuickType, and more Activate hands-free with only your voice using “Hey Siri” Use your voice to run shortcuts from your favorite apps Learn more about Siri External Buttons and Connectors Home/Touch ID sensor Volume up/down Ring/Silent switch Side button Built-in stereo speaker Built-in microphone Lightning connector Power and Battery9 Lasts up to 2 hours longer than iPhone 6s Talk time (wireless): Up to 14 hours on 3G Standby: Up to 10 days Internet use: Up to 12 hours on 3G, Up to 12 hours on LTE, Up to 14 hours on Wi-Fi Wireless video playback: Up to 13 hours Wireless audio playback: Up to 40 hours Built-in rechargeable lithium-ion battery Charging via USB to computer system or power adapter Sensors Touch ID fingerprint sensor Barometer Three-axis gyro Accelerometer Proximity sensor Ambient light sensor Operating System iOS 12 iOS is the world’s most personal and secure mobile operating system, packed with powerful features that help you get the most out of every day. See what’s new in iOS Accessibility Accessibility features help people with disabilities get the most out of their new iPhone 7. With built-in support for vision, hearing, physical and motor skills, and learning and literacy, you can fully enjoy the world’s most personal device. Learn more about Accessibility Features include: VoiceOver Zoom Magnifier RTT and TTY support Siri and Dictation Type to Siri Switch Control Closed Captions AssistiveTouch Speak Screen Built-in Apps Camera Photos Health Messages Phone FaceTime Mail Music Wallet Safari Maps Siri Calendar iTunes Store App Store Notes News Contacts Books Home Weather Reminders Clock TV Stocks Calculator Voice Memos Compass Podcasts Watch Tips Find My iPhone Find My Friends Settings Files Measure Free Apps from Apple Pages, Numbers, Keynote, iMovie, GarageBand, iTunes U, Clips, and Apple Store app are preinstalled. iMovie Pages Keynote Numbers iTunes U GarageBand Apple Store Trailers Apple TV Remote iTunes Remote Music Memos Clips Shortcuts Headphones EarPods with Lightning Connector SIM Card Nano-SIM iPhone 7 and iPhone 7 Plus are not compatible with existing micro-SIM cards. Rating for Hearing Aids iPhone 7 (Model A1660, A1778): M3, T4 Mail Attachment Support Viewable document types .jpg, .tiff, .gif (images); .doc and .docx (Microsoft Word); .htm and .html (web pages); .key (Keynote); .numbers (Numbers); .pages (Pages); .pdf (Preview and Adobe Acrobat); .ppt and .pptx (Microsoft PowerPoint); .txt (text); .rtf (rich text format); .vcf (contact information); .xls and .xlsx (Microsoft Excel); .zip; .ics; .usdz (USDZ Universal) System Requirements Apple ID (required for some features) Internet access10 Syncing with iTunes on a Mac or PC requires: Mac: OS X 10.11.6 or later PC: Windows 7 or later iTunes 12.8 or later on Mac and iTunes 12.9 or later on PC (free download from www.itunes.com/download) Environmental Requirements Operating ambient temperature: 32° to 95° F (0° to 35° C) Nonoperating temperature: ‑4° to 113° F (‑20° to 45° C) Relative humidity: 5% to 95% noncondensing Operating altitude: tested up to 10,000 feet (3000 m) Languages Language support English (Australia, UK, U.S.), Chinese (Simplified, Traditional, Traditional Hong Kong), French (Canada, France), German, Italian, Japanese, Korean, Spanish (Latin America, Mexico, Spain), Arabic, Catalan, Croatian, Czech, Danish, Dutch, Finnish, Greek, Hebrew, Hindi, Hungarian, Indonesian, Malay, Norwegian, Polish, Portuguese (Brazil, Portugal), Romanian, Russian, Slovak, Swedish, Thai, Turkish, Ukrainian, Vietnamese QuickType keyboard support English (Australia, Canada, India, Singapore, UK, U.S.), Chinese - Simplified (Handwriting, Pinyin, Stroke), Chinese - Traditional (Cangjie, Handwriting, Pinyin, Stroke, Sucheng, Zhuyin), French (Belgium, Canada, France, Switzerland), German (Austria, Germany, Switzerland), Italian, Japanese (Kana, Romaji), Korean, Spanish (Latin America, Mexico, Spain), Arabic (Modern Standard, Najdi), Armenian, Azerbaijani, Belarusian, Bengali, Bulgarian, Catalan, Cherokee, Croatian, Czech, Danish, Dutch, Emoji, Estonian, Filipino, Finnish, Flemish, Georgian, Greek, Gujarati, Hawaiian, Hebrew, Hindi (Devanagari, Transliteration), Hinglish, Hungarian, Icelandic, Indonesian, Irish, Kannada, Latvian, Lithuanian, Macedonian, Malay, Malayalam, Maori, Marathi, Norwegian, Odia, Persian, Polish, Portuguese (Brazil, Portugal), Punjabi, Romanian, Russian, Serbian (Cyrillic, Latin), Slovak, Slovenian, Swahili, Swedish, Tamil (Script, Transliteration), Telugu, Thai, Tibetan, Turkish, Ukrainian, Urdu, Vietnamese, Welsh QuickType keyboard support with predictive input English (Australia, Canada, India, Singapore, UK, U.S.), Chinese (Simplified, Traditional), French (Belgium, Canada, France, Switzerland), German (Austria, Germany, Switzerland), Italian, Japanese, Korean, Russian, Spanish (Latin America, Mexico, Spain), Portuguese (Brazil, Portugal), Thai, Turkish Siri languages English (Australia, Canada, India, Ireland, New Zealand, Singapore, South Africa, UK, U.S.), Spanish (Chile, Mexico, Spain, U.S.), French (Belgium, Canada, France, Switzerland), German (Austria, Germany, Switzerland), Italian (Italy, Switzerland), Japanese, Korean, Mandarin (China mainland, Taiwan), Cantonese (China mainland, Hong Kong), Arabic (Saudi Arabia, United Arab Emirates), Danish (Denmark), Dutch (Belgium, Netherlands), Finnish (Finland), Hebrew (Israel), Malay (Malaysia), Norwegian (Norway), Portuguese (Brazil), Russian (Russia), Swedish (Sweden), Thai (Thailand), Turkish (Türkiye) Dictation languages English (Australia, Canada, India, Indonesia, Ireland, Malaysia, New Zealand, Philippines, Saudi Arabia, Singapore, South Africa, United Arab Emirates, UK, U.S.), Spanish (Argentina, Chile, Colombia, Costa Rica, Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Mexico, Panama, Paraguay, Peru, Spain, Uruguay, U.S.), French (Belgium, Canada, France, Luxembourg, Switzerland), German (Austria, Germany, Luxembourg, Switzerland), Italian (Italy, Switzerland), Japanese, Korean, Mandarin (China mainland, Taiwan), Cantonese (China mainland, Hong Kong, Macao), Arabic (Kuwait, Qatar, Saudi Arabia, United Arab Emirates), Catalan, Croatian, Czech, Danish, Dutch (Belgium, Netherlands), Finnish, Greek, Hebrew, Hindi (India), Hungarian, Indonesian, Malaysian, Norwegian, Polish, Portuguese (Brazil, Portugal), Romanian, Russian, Shanghainese (China mainland), Slovakian, Swedish, Thai, Turkish, Ukrainian, Vietnamese Definition dictionary support English, Chinese (Simplified, Traditional), Danish, Dutch, French, German, Hebrew, Hindi, Italian, Japanese, Korean, Norwegian, Portuguese, Russian, Spanish, Swedish, Thai, Turkish Thesaurus English (UK, U.S.) Bilingual dictionary support with English Arabic, Chinese (Simplified, Traditional), Dutch, French, German, Hindi, Italian, Japanese, Korean, Portuguese, Russian, Spanish Spell check English, French, German, Italian, Spanish, Danish, Dutch, Finnish, Korean, Norwegian, Polish, Portuguese, Russian, Swedish, Turkish In the Box iPhone with iOS 12 EarPods with Lightning Connector Lightning to USB Cable USB Power Adapter Documentation

Question : what is the display size of iphone 7? Answer: """ Here, only the question will change (basically, user will be asking the question about the document which is context). Here instruction, context remains the same. Question will come from the user and model has to predict/respond with an answer. is it doable with gist token approach?

[jayelm]

Hi, thanks for your interest!

My question is,

1. Can this gist token method be used for the static prompt?

Yes, though note gist tokens are specialized for cases where you'd like to compress multiple prompts, but you do not know what they are ahead of time and would like to arbitrary compress multiple prompts without finetuning. If you instead know exactly the prompt you'd like to compress ahead of time and don't mind fine-tuning, you might consider simply fine-tuning to distill away a single prompt using a method like prefix/prompt-tuning or LoRA.

2. my current context length varies between 23,000 and 25,000, is it possible to perform gist token for the larger prompt? Basically, I have a pdf file which is about 45 pages, and it remains static. I would like ask questions about the pdf, here instruction, context will remain as it is(static) only question will be changing.

Again, if you have a single very long prompt, you'll probably get the best performance simply distilling away the prompt directly, rather than using a meta-learning method like gist tokens. Gist tokens might perform ok on this task, but I imagine your ability to compress extremely long documents is an empirical question and depends on having a rich enough dataset of long documents + QA pairs so your model can learn general prompt compression.