'select' values sometimes include preceding characters

[andysalerno]

The bug In some prompts, I have a select like so:

I will use: {{select 'thought_action' options=valid_actions}} because {{gen temperature=0.1 'action_reason' stop='</tho'}}</thought>

invoked like:

program(valid_actions=['WEB_SEARCH', 'OTHER'])

And in certain cases - especially with very long contexts - the value chosen by select is not 1:1 with the options I provided. Instead it will be one of the options, plus some duplicated suffix from the preceding characters. A bad output would be like this:

I will use: se: WEB_SEARCH because blah blah blah
            ^--- unexpected characters

Digging through the _select.py code, it looks like it's building a string that has each option, prepended with the full context. And then later it is slicing away the context, to get back to the option. And I have a feeling the index calculation isn't always correct, but not sure why.

To Reproduce It's hard to make a short prompt that triggers this. Here's a colab that reproduces it reliably: https://colab.research.google.com/drive/1JRZN2Ib3QMHN0T4OHiAxjpMHTD8llKma?usp=sharing

NOTE: I am using a community-provided model (wizard-vicuna), and gptq, and a hacky wrapper to load a gptq model as a Transformer in guidance. So the root cause could be in any of those places, and not in guidance. However, I've only seen this problem in guidance.

[slundberg]

Thanks for sharing a specific example to walk through! I narrowed it down to a change in the way the sentence piece tokenizer for LLaMA handles spacing for initial tokens when you have the EOS token (</s>) in your prompt explicitly (like you do). Not sure exactly how to address this yet, but that is the cause...

[slundberg]

Fixed now in 0.0.57, just needed to embrace the fact that round trip encoding and then decoding can change the string.

[zacharyblank]

I am using 0.0.63 and am having the same issue with a fine-tuned MPT 7B Storywriter model. It is only happening on long inputs (>4k tokens). Any thoughts?

My code looks like"

input = """
Skip to main content .us Hello Select your address Toys & Games All Departments Alexa Skills Amazon Devices Amazon Pharmacy Amazon Warehouse Appliances Apps & Games Arts, Crafts & Sewing Audible Books & Originals Automotive Parts & Accessories Baby Beauty & Personal Care Books CDs & Vinyl Cell Phones & Accessories Clothing, Shoes & Jewelry Women Men Girls Boys Baby Collectibles & Fine Art Computers Credit and Payment Cards Digital Music Electronics Garden & Outdoor Gift Cards Grocery & Gourmet Food Handmade Health, Household & Baby Care Home & Business Services Home & Kitchen Industrial & Scientific Just for Prime Kindle Store Luggage & Travel Gear Luxury Stores Magazine Subscriptions Movies & TV Musical Instruments Office Products Pet Supplies Premium Beauty Prime Video Smart Home Software Sports & Outdoors Subscribe & Save Subscription Boxes Tools & Home Improvement Toys & Games Under $10 Video Games Whole Foods Market EN Hello, sign in Account & Lists Returns & Orders 0 Cart Sign in New customer? Start here. Your Lists Create a List Find a List or Registry Your Account Account Orders Recommendations Browsing History Watchlist Video Purchases & Rentals Kindle Unlimited Content & Devices Subscribe & Save Items Memberships & Subscriptions Prime Membership Amazon Credit Cards Music Library Start a Selling Account Register for a free Business Account Customer Service Sign in New customer? Start here. All Disability Customer Support Clinic Best Sellers Customer Service Amazon Basics New Releases Prime Music Today's Deals Books Registry Fashion Amazon Home Pharmacy Gift Cards One Medical Toys & Games Sell Coupons Luxury Stores Automotive Find a Gift Beauty & Personal Care Computers Home Improvement Health & Household Products Video Games Pet Supplies Smart Home Audible Shop Father's Day gifts Toys & Games Kids Gift Guide Shop Toys by Character Shop Best Selling Toys Shop Newly Released Toys Shop Amazon Exclusive Toys Shop Toy Deals Create a Gift List Toys & Games › Building Toys › Building Sets Enjoy fast, FREE delivery, exclusive deals and award-winning movies & TV shows with Prime Try Prime and start saving today with Fast, FREE Delivery Delivery Pickup $39.95$39.95 FREE Returns Return this item for free Free returns are available for the shipping address you chose. You can return the item for any reason in new and unused condition: no shipping charges Learn more about free returns. How to return the item? FREE delivery Sunday, June 11 Or fastest delivery Tomorrow, June 6. Order within 5 hrs 48 mins Select delivery location In Stock Qty: 1 2 3 4 5 Qty:1  $$39.9539.95 () Includes selected options. Includes initial monthly payment and selected options. Details Price ($39.95x) $39.95 Subtotal $$39.9539.95 Subtotal Initial payment breakdown Shipping cost, delivery date, and order total (including tax) shown at checkout. Add to Cart Buy Now Enhancements you chose aren't available for this seller. Details To add the following enhancements to your purchase, choose a different seller.  Payment Secure transaction Your transaction is secure We work hard to protect your security and privacy. Our payment security system encrypts your information during transmission. We don’t share your credit card details with third-party sellers, and we don’t sell your information to others. Learn more Ships from Amazon.com Sold by Amazon.com Returns Eligible for Return, Refund or Replacement within 30 days of receipt Eligible for Return, Refund or Replacement within 30 days of receipt This item can be returned in its original condition for a full refund or replacement within 30 days of receipt. Read full return policy Details Payment Secure transaction We work hard to protect your security and privacy. Our payment security system encrypts your information during transmission. We don’t share your credit card details with third-party sellers, and we don’t sell your information to others. Learn more Ships from Amazon.com Sold by Amazon.com Returns Eligible for Return, Refund or Replacement within 30 days of receipt This item can be returned in its original condition for a full refund or replacement within 30 days of receipt. Read full return policy Add a gift receipt for easy returns Add to List Added to Unable to add item to List. Please try again. Sorry, there was a problem. There was an error retrieving your Wish Lists. Please try again. Sorry, there was a problem. List unavailable. Sorry, there was a problem. There was an error retrieving your Registries. Please try again. +  Select a gift tag No tag Add a new gift tag Create gift tags Gift tags group items on your list. Make it easy for guests to find a gift, create tags to group items however you choose - name, room, item type, color, etc. Enter a gift tag name Create a gift tag You have reached the maximum of 15 gift tags. Please remove one and try again. The input you entered is invalid, please try again. Back Gift Tags can be seen by anyone viewing your gift list. Are you sure you want to delete the gift tag ? Any items on your gift list with this gift tag will become untagged. No, nevermind Yes, delete Sorry, there was a problem. There was an error in retrieving your gitf tags. Please try again. $39.95$39.95 FREE pickup Saturday, June 10 Or fastest pickup Tomorrow, June 6. Order within 13 hrs 18 mins Amazon Locker - Capsule 1.87 mi | Vancouver 98682 How pickup works Pick up from nearby pickup location Step 1: Place Your Order Select the “Pickup” option on the product page or during checkout. Step 2: Receive Notification Once your package is ready for pickup, you'll receive an email and app notification. Step 3: Pick up Bring your order ID or pickup code (if applicable) to your chosen pickup location to pick up your package. In Stock. Qty: 1 2 3 4 5 Qty:1 Add to Cart Buy Now Payment Secure transaction Your transaction is secure We work hard to protect your security and privacy. Our payment security system encrypts your information during transmission. We don’t share your credit card details with third-party sellers, and we don’t sell your information to others. Learn more Sold by Amazon.com Returns Eligible for Return, Refund or Replacement within 30 days of receipt Eligible for Return, Refund or Replacement within 30 days of receipt This item can be returned in its original condition for a full refund or replacement within 30 days of receipt. Read full return policy Details Payment Secure transaction We work hard to protect your security and privacy. Our payment security system encrypts your information during transmission. We don’t share your credit card details with third-party sellers, and we don’t sell your information to others. Learn more Sold by Amazon.com Returns Eligible for Return, Refund or Replacement within 30 days of receipt This item can be returned in its original condition for a full refund or replacement within 30 days of receipt. Read full return policy New & Used (67) from $36.99$36.99 & FREE Shipping Other Sellers on Amazon Added Not added Add to Cart View Cart $43.90 & FREE Shipping. Details Sold by: Kayfabe LLC Sold by: Kayfabe LLC (1625 ratings) 95% positive over last 12 months In Stock. Shipping rates and Return policy Added Not added Add to Cart View Cart $43.95 & FREE Shipping. Details Sold by: MM Bricks LLC Sold by: MM Bricks LLC (1295 ratings) 100% positive over lifetime In Stock. Shipping rates and Return policy Added Not added Add to Cart View Cart $44.95 & FREE Shipping. Details Sold by: Cool Dad's Toys Sold by: Cool Dad's Toys (40600 ratings) 98% positive over last 12 months Only 2 left in stock - order soon. Shipping rates and Return policy Have one to sell? Sell on Amazon 6 VIDEOS LEGO Architecture London 21034 Building Kit Merchant Video To view this video download Flash Player VIDEOS 360° VIEW IMAGES LEGO Architecture London Skyline 21034 Collectible Model Building Kit with London Eye, Big Ben, and Tower Bridge, Office Home Décor, Skyline Collection, Gift Idea for Father's Day from Son or Daughter Visit the LEGO Store 4.8 4.8 out of 5 stars 11,635 ratings | 26 answered questions Amazon's Choice highlights highly rated, well-priced products available to ship immediately. Amazon's Choice for "lego architecture" $39.95$39.95 FREE Returns Return this item for free Free returns are available for the shipping address you chose. You can return the item for any reason in new and unused condition: no shipping charges Learn more about free returns. How to return the item? Get $60 off instantly: Pay $0.00 upon approval for Amazon Visa. No annual fee. Available at a lower price from other sellers that may not offer free Prime shipping. Extra Savings 90 days FREE. Terms apply. 1 Applicable Promotion 90 days FREE. Terms apply. 90 days FREE of Amazon Music Unlimited. Included with purchase of an eligible product. You will receive an email with signup instructions. Renews automatically. New subscribers only. Terms apply. Offered by Amazon.com. Here's how (restrictions apply) Color: Multicolor Build a detailed LEGO Architecture model interpretation of the London skyline featuring famous landmarks! The model building kit for adults and kids aged 12+ features the National Gallery, Nelson’s Column, London Eye, Big Ben and Tower Bridge London’s Thames River is represented by transparent baseplate tiles This set has been designed to deliver a rewarding and relaxing building experience and makes an eye-catching decor piece for the home or office Includes collectible booklet containing information about the design, architecture and history of the building LEGO Architecture models are compatible with all LEGO construction sets for creative building and make a great birthday, Christmas or anytime gift Looking to give something special to Dad, this makes the perfect Father's Day gift from a thoughtful son or daughter. Build together with Dad or give as a DIY gift for him to complete on his own Consider a similar item MEGA BLOKS Fisher Price Toddler Building Blocks, Green Town Sort & Recycle Squad with 51 Pieces, 3 Figures, Toy Gift Ideas for Kids (491) $16.23 Climate Pledge Friendly Special offers and product promotions 90 days FREE of Amazon Music Unlimited. Included with purchase of an eligible product. You will receive an email with signup instructions. Renews automatically. New subscribers only. Terms apply. Offered by Amazon.com. Here's how (restrictions apply) Frequently bought together This item: LEGO Architecture London Skyline 21034 Collectible Model Building Kit with London Eye, Big Ben, and Tower Bridge, Office Home Décor, Skyline Collection, Gift Idea for Father's Day from Son or Daughter $39.95$39.95 + LEGO Architecture Paris Skyline 21044 Collectible Model Building Kit with Eiffel Tower and The Louvre, Skyline Collection, Office Home Décor, Unleash Dad's Creativity This Father's Day $47.99$47.99 + LEGO Architecture New York City Skyline 21028, Collectible Model Kit for Adults to Build, Creative Activity, Home Decor Gift Idea, Perfect Father's Day Build for Dads Who Love to Create $52.99$52.99 Total price:$140.93 Try again! Details Added to Cart Add all three to Cart Some of these items ship sooner than the others. Show details Products related to this item Sponsored Page 1 of 1Start overPage 1 of 1 Previous page of related Sponsored Products LEGO Ideas A-Frame Cabin 21338 Collectible Display Set, Buildable Model Kit for Adu... 3 $179.99 City London Bus Station Building Blocks Set, Double Deckers London Travel Bus and B... 30 $46.99 Save 5% with coupon Givenni Architecture The Roman Colosseum Building Set; Collectible Model for Adults... $149.89 Save 15% with coupon YEECHAO Chinese Architecture of Suzhou Garden Micro Building Blocks Set, Creative B... 55 $49.99 Save $5.00 with coupon Givenni Architecture Landmark Collection The Acropolis in Athens Building Set; Coll... 17 $94.89 GEYANG Chinese Architecture of Turret Micro Building Blocks Set, Creative Building ... 2 $49.90 NEWABWN Peach Blossom Pond Micro Mini Building Blocks Set for Adults and… 42 $49.99 Save 10% with coupon dOvOb Real Big Ben Micro Building Blocks Set (3600PCS) - World Famous Architectural... 1,172 $46.99 Geniteen Building Blocks Set Roman Colosseum Model Micro Building Blocks 5594… 37 $49.99 Stranger Thinng Architecture Skylines Building Blocks Toy for TV Fans Compatible… 3 $26.98 Save $3.00 with coupon Next page of related Sponsored Products Product information Product Dimensions 10.32 x 7.52 x 2.4 inches Item Weight 1.14 pounds Domestic Shipping Item can be shipped within U.S. International Shipping This item is not eligible for international shipping. Learn More Country of Origin Mexico ASIN B01KJENN0U Item model number 6174060 Manufacturer recommended age 12 years and up Best Sellers Rank #2,428 in Toys & Games (See Top 100 in Toys & Games) #120 in Toy Building Sets Customer Reviews 4.8 4.8 out of 5 stars 11,635 ratings 4.8 out of 5 stars Is Discontinued By Manufacturer No Release date January 1, 2017 Manufacturer LEGO Warranty & Support If you'd like a copy of the manufacturer's warranty for a product found on Amazon.com, you can contact the manufacturer directly or visit their website for more information. Manufacturer's warranties may not apply in all cases, depending on factors like the use of the product, where the product was purchased, or who you purchased the product from. Please review the warranty carefully, and contact the manufacturer if you have any questions. Feedback Would you like to tell us about a lower price? Please sign in to provide feedback. Product Description Celebrate the architectural diversity of London with this detailed LEGO brick model. The LEGO Architecture Skyline Collection offers models suitable for display and decor in the home and office, and has been developed for all with an interest in travel, architectural culture, history and design. Each set is scaled to give an accurate representation of the comparative size of each structure, with true-to-life color depiction. This set features the National Gallery, Nelson’s Column, London Eye, Big Ben (the Elizabeth Tower) and Tower Bridge, and is finished with a decorative “London” nameplate. LEGO Architecture sets make a great birthday, Christmas or any time gift, or a relaxing building experience for yourself! Contains 468 pieces. Measures over 5” (15cm) high, 11” (28cm) wide and 3” (8cm) deep. Important information Safety Information No From the brand Let your imagination soar! LEGO sets are one of the first toys a child plays with when growing up and this love of the building spans the test of time and age. Previous page LEGO NEW Releases Visit the Store LEGO Best Sellers Visit the Store Shop by Age Visit the Store Shop by Theme Visit the Store Shop by Interest Visit the Store Next page From the manufacturer London Calling Put London’s architectural diversity on display with this buildable set, featuring five of the city’s best-loved landmarks. A mindful way to unwind and a lasting keepsake of an unstoppable capital, LEGO Architecture London Skyline makes a stylish addition to any Anglophile’s home or office. 21026 Venice 21027 Berlin 21028 New York City 21032 Sydney 21033 Chicago 21034 London Model Size 5" (14cm) high, 9" (25cm) wide and 1" (3cm) deep 9" (23cm) high, 9" (25cm) wide and 1" (5cm) deep 10" (26cm) high, 9" (25cm) wide and 1" (4cm) deep 8” (21cm) high, 9” (25cm) wide and 2” (6cm) deep 7” (20cm) high, 8” (22cm) wide and 1” (5cm) deep 5” (15cm) high, 11” (28cm) wide and 3” (8cm) deep Buildings 6 5(6 incl. Berlin wall brick) 5 4 6 5 Number of Bricks 212 289 598 361 444 468 Release Date 1/1/2016 1/1/2016 1/1/2016 1/1/2017 1/1/2017 1/1/2017 What's in the box LEGO Building Elements Product guides and documents Installation Manual (PDF) Related products with free delivery on eligible orders Sponsored | Try Prime for unlimited fast, free shipping Page 1 of 1Start overPage 1 of 1 Previous page of related Sponsored Products LEGO Ideas A-Frame Cabin 21338 Collectible Display Set, Buildable Model Kit for Adu... 3 $179.99 YEECHAO Chinese Architecture of Suzhou Garden Micro Building Blocks Set, Creative B... 55 $49.99 Save $5.00 with coupon MACTANO Architecture Mini Building Block Set, Chinese Temple of… $49.99 Stranger Thinng Architecture Skylines Building Blocks Toy for TV Fans Compatible… 3 $26.98 Save $3.00 with coupon Johnny 5 Robot Building Block Set, Short Open Circuit Johnny Five Robot… 2 $19.99 Save 10% with coupon 3 in 1 City Great Vehicles Farm Tractor Building Kit, Compatible with Lego… 10 $47.99 Nifeliz DIVN Race Car MOC Building Kit and Engineering Toy, Adult Collectible Sport... 253 $149.95 Givenni Architecture The Roman Colosseum Building Set; Collectible Model for Adults... $149.89 Save 15% with coupon City London Bus Station Building Blocks Set, Double Deckers London Travel Bus and B... 30 $46.99 Save 5% with coupon BIRANCO. Remote Control Tracked 3in1 Stunt Racer Building Toy with Power… 1,724 $59.97 Save 8% with coupon Next page of related Sponsored Products Brief content visible, double tap to read full content. Full content visible, double tap to read brief content. Videos Page 1 of 1Start OverPage 1 of 1 Previous page Videos for this product 16:07 Customer Review: Not a Complete London Skyline but Still a Great LEGO Build Jeremy Herbert 0:22 Customer Review: Would wish to have some improvements ladygoodmood 0:45 Quick Close up look at the London Lego set on display J. Tonioli 4:56 LEGO 21034 - London Skyline - Great Display Piece Drone Valley 0:42 REVIEW LEGO Architecture London Skyline Collection FreshTake Videos for related products 1:19 LEGO Architecture Taj Mahal - Super detailed build! Jason Hesch 0:07 LEGO 360 Video Lego systems 0:46 LEGO Architecture Dubai 21052 Lego systems 0:41 LEGO Las Vegas Skyline Close Up look at the finished build J. Tonioli 0:36 LED Lighting Kit for LEGO Skyline Collection New York 21028 Brick Loot - Fast FREE Shipping from the USA Next page Video Player is loading. Play Video Play Mute Current Time 0:00 / Duration 0:00 Loaded: 0% Stream Type LIVE Seek to live, currently behind liveLIVE Remaining Time -0:00 1x Playback Rate Chapters Chapters Descriptions descriptions off, selected Captions Captions off, selected Audio Track Fullscreen This is a 
"""

# input = """For sale (changing the input), a cuddly FAO Swartz Golden Teddy Bear FAO Swartz $22.91"""

marker = "### Response"
program = guidance("""{{input}}

### Instructions
If there is no price, return "null" for both price and currency_code

{{marker}}
{
    "brand": {{gen stop='\n'}}
    "title": {{gen stop='\n'}}
    "price": {{gen stop='\n' temperature=0}}
    "currency_code": {{gen stop='\n'}}
}
""")
    # "animal": {{gen stop='\n'}}
    # "price": {{gen pattern='((\d+\.\d{2})|null)\n' stop='\n' max_tokens=4}}, We canot use pattern yet because it is too slow. see: https://github.com/microsoft/guidance/blob/main/guidance/llms/_transformers.py#L477
    # "currency_code": "{{gen pattern='(USD|EUR|GBP|null)' max_tokens=4}}"
    # "color": "{{gen stop='"'}}",

# isomeric.cache.clear()

# quantized inference time: 2+ minutes (I canceled it.)
# Unquantized inference time: 6 minutes

out = program(input=input,
            marker=marker,
            stream=False,
            silent=True,
        #   async_mode=False,
            llm=isomeric)

start_index = out.text.find(marker)

print(out.text[start_index + len(marker)+1:])

With the following output:

{
    "brand": ": "LEGO",
    "title": ": "LEGO Architecture London Skyline 21034 Collectible Model Building Kit with London Eye, Big Ben, and Tower Bridge, Office Home Décor, Skyline Collection, Gift Idea for Father's Day from Son or Daughter",
    "price": ": "$39.95",
    "currency_code": ": "USD",
}

Notice the extra ": in each line.

With a shorter input it seems to work just fine.