Extraction often only gets some of the data

[simonw]

I'm testing with data from this page: https://ogs.ny.gov/procurement/ogs-centralized-contracts-list

I pasted in this:

Award # Group   Award Title Type    Keywords
23295   20915   Furniture, All Types (Except Hospital Room and Patient Handling) (Statewide)    Commodity   Conference Furniture, Dormitory Furniture, General Purpose Tables, High Density Filing, Household Furniture, Library Furniture, Office Furniture, School Furniture, Specialty Seating, Tall Seating, Bariatric, Gang Seating, Laboratory Stools, Systems Furniture
23287   05500   Fuel Oil, Heating (Grades #2, #6 Kerosene and Bioheating Fuel) (Statewide)  Commodity   Fuel oil, Heating, Kerosene, Bioheating Fuel, Heating
23321   05900   Natural Gas (Firm Supply - Specific Locations Within National Grid Territories) Commodity   Natural Gas
23283   05800   Liquefied Petroleum Gas (LPG) - Propane (Statewide) Commodity   Cylinders, Gallons, Tanks, Installation, Testing, Inspections, LP, Liquid Propane, Butane, Isobutene
23315   01800   Road Salt, Treated Salt, & Emergency Standby Road Salt (Statewide)  Commodity   Ice, Snow, Sodium, Chloride
23272   50030   Wove & Kraft Envelopes  Commodity   Printed Envelopes, Non-Printed Envelopes
23254   40524   School Buses (Statewide)    Commodity   Bus, Conventional Bus
23241   10201   Pharmaceuticals (Individual Prescriptions) Statewide & Regional Commodity   Drugs, Pharmacists Services, Prescription Delivery, Over the Counter, OTC, Pharmaceutical Products, Medicine, Medication
23200   20600   Floor Coverings and Related Services (Statewide Piggyback)  Commodity   Carpet, Tile, Broadloom, Vinyl, LVT, Rubber Tile, Hardwood, Linoleum, Floormat, Ceramic, Installation, Padding
23238   79006   Air Travel Services (Statewide) Commodity   Plane Travel
23222   10150   Personal Protective Equipment (PPE) and Related Items (Statewide)   Commodity   Respirators, Masks, Face Shields, Goggles, Gowns, Covers, Hand Sanitizer, Wipes, Fit Test Kits, N95, Disinfecting Wipes, Surgical Mask, Alcohol Wipes, PPE
23239   01600   Milk, Fluid (Statewide) Commodity   Low Fat Milk, Reduced Fat Milk, Skim Milk, Homogenized Milk
23073   30204   Athletic Equipment (Statewide)  Commodity   Gymnasium Equipment, Physical Education Equipment, Fitness, Exercise, Elliptical, Bike, Barbell, Dumbbell, Bench, Cardiovascular, Strength Training, Stairclimber, Treadmill, Weights, Mats
23123   30310   Vehicle and Equipment Parts and Related Product (Statewide) Commodity   Light Duty Vehicle Parts, Heavy Duty Vehicle Parts, Heavy Equipment Parts, Direct Order Parts, Commonly Stocked Parts, Vehicle Cleaning Supplies, Vehicle Paint, Vehicle Tools
23204   05700   Motor Oil, Hydraulic Oil, and Diesel Exhaust Fluid (Statewide)  (Replaces 23012-RA, SW) Commodity   Motor Crankcase Oil, Hydraulic Oil, Diesel Exhaust Fluid, Refined Oil, Re-Refined Oil, Lubricating Oil, High Detergent, 5W-30, 5W-20, 10W-30, 15W-40
23149   30600   Tires, Tubes, and Services (Statewide)  Commodity    
PGB-23243   35000   Vehicle Lifts and Associated Garage Equipment Sourcewell Piggyback (Statewide)  Commodity   Garage Associated Parts, Garage Associated Supplies, Garage Associated Accessories, Vehicle Lift Installation, Vehicle Lift Repair, Vehicle Lift Maintenance
23166   40440   Vehicles, Class 1 – 8 (Statewide)   Commodity   Single OEM Vehicles, Chassis, Complete Vehicles, Car, Truck, SUV, Van, Sedan, One-ton Truck, Cargo Van
23170   40523   Buses, Transit (Adult Passenger) (Statewide)    Commodity   FTA Adult Passenger Transit Buses, Associated Transit Bus Equipment
23260   20070   Books, Serials, Databases, and Library Resource Management Products Commodity   Serials, Databases, Library Resource Management Products and Services, Printed Publications, Non-Print Library Materials, Electronic Publications, Research Support Products, Printed Periodicals, Electronic Periodicals, eBooks, Streaming Audio, Video Content, Magazines, Newspapers, Journals, Legal Research, Books, Textbooks
23185   23106   STEM / STEAM, Science Laboratory Educational Supplies And Equipment (Statewide) Commodity    
23268   32100   Snow and Ice Control Agents (Statewide) Commodity   Liquid, Calcium, Chloride, Organic, Based Performance Enhancer, OBPE, Magnesium, Corrosion, Inhibited, Treated, Salt, Flake, Pellet, Solar
23054   40061   Protective Outerwear    Commodity   Safety Shoes, Specialty Boots, Firefighting Turnout Gear, Firefighting Proximity Clothing, Wildland Clothing, Helmets, Gloves, Firefighting Boots, EMS/Search and Rescue Clothing, Bunker Gear
PGB-23197   10200   Distribution of Vaccines Including Influenza Vaccines (Statewide) (Replaces Award PGB-22797)    Commodity   General Vaccines, Seasonal Influenza Vaccines, Pharmaceuticals, Drugs, Flu Shot, COVID-19 Vaccine
PGB-23290   10200   MMCAP Infuse Influenza Vaccines (Statewide) Commodity   General Vaccines, Seasonal Influenza Vaccines, Pharmaceuticals, Drugs, Flu Shot

Configured like this:

But it only extracted the first seven:

[
  {
    "award_number": 23295,
    "group": 20915,
    "award_title": "Furniture, All Types (Except Hospital Room and Patient Handling) (Statewide)",
    "type": "Commodity",
    "keywords": "Conference Furniture, Dormitory Furniture, General Purpose Tables, High Density Filing, Household Furniture, Library Furniture, Office Furniture, School Furniture, Specialty Seating, Tall Seating, Bariatric, Gang Seating, Laboratory Stools, Systems Furniture"
  },
  {
    "award_number": 23287,
    "group": 5500,
    "award_title": "Fuel Oil, Heating (Grades #2, #6 Kerosene and Bioheating Fuel) (Statewide)",
    "type": "Commodity",
    "keywords": "Fuel oil, Heating, Kerosene, Bioheating Fuel, Heating"
  },
  {
    "award_number": 23321,
    "group": 5900,
    "award_title": "Natural Gas (Firm Supply - Specific Locations Within National Grid Territories)",
    "type": "Commodity",
    "keywords": "Natural Gas"
  },
  {
    "award_number": 23283,
    "group": 5800,
    "award_title": "Liquefied Petroleum Gas (LPG) - Propane (Statewide)",
    "type": "Commodity",
    "keywords": "Cylinders, Gallons, Tanks, Installation, Testing, Inspections, LP, Liquid Propane, Butane, Isobutene"
  },
  {
    "award_number": 23315,
    "group": 1800,
    "award_title": "Road Salt, Treated Salt, & Emergency Standby Road Salt (Statewide)",
    "type": "Commodity",
    "keywords": "Ice, Snow, Sodium, Chloride"
  },
  {
    "award_number": 23272,
    "group": 50030,
    "award_title": "Wove & Kraft Envelopes",
    "type": "Commodity",
    "keywords": "Printed Envelopes, Non-Printed Envelopes"
  },
  {
    "award_number": 23254,
    "group...,": 23239,
    "award_title": "Milk, Fluid (Statewide)",
    "type": "Commodity",
    "keywords": "Low Fat Milk, Reduced Fat Milk, Skim Milk, Homogenized Milk"
  }
]

[simonw]

First suspicion: there's some default number of tokens in the output that this is falling victim to.

So I added max_tokens=4096 since a few random searches seemed to hint that was the maximum.

And I got 9 instead of 7. When I pasted the output JSON into a token counter it was only 888 tokens, so nowhere near the limit.

[simonw]

This may need to be solved by documentation: a note on the page that warns you that it will not necessarily get everything.

This tool is going to need quite a bit of inline documentation to help people deal with its limitations.