rogue inference on quantized gguf

[amygbAI]

I have finetuned a HF llama2 7b model on custom data. I used the inference on the model directly using the following prompt

Below is an instruction that describes a task. Write a response that appropriately completes the request.

Instruction:

Extract invoice number, po number, invoice date, invoice total and ship date from the text below. If you do not find it, do not present incorrect values and just mark as NA. Present the result in the format Inv_no: , Inv_dt:, Ship_dt:, Inv_total: , Inv_subtotal: , Po_number: L0. , 575 1 397-0220 Invoice HRANKS LOCATION 01 SUPPL.VCOMPANY , INC . DATE 12/19/23 BILL TO SHIP TO 107526 WPI WESTERN PARTITIONS INC WPI WESTERN PARTITIONS CUSTOMER P/O NUMBER PAYMENT TERMS 1408455 05/22/23 22-2848-0056 1 9% 15 DAYS NET 30 DAY WRITTEN BY CONTACT SHIP FREIGHT TERMS JOB NUMBER SALES TAX DEPOSIT DEPOSIT TOTAL TOTAL PICK UP FREIGHT AMOUNT APPLIED CC Fee INVOICE 265.00 0.00 0.00 0.00 0.00 0.00 0.00 265.00

Response:

{"date": "2023-12-20", "time": "2023-12-20", "invoice_date": "2023/12/19", "invoice_number": "2654544", "invoice_total": "265.00", "invoice_subtotal": "265.00", "po_number": "NA", "shipping_date": "NA", "description": "SHIPMENT", "part_number": "NA", "qty": "NA", "line_item": "NA", "packaging": "NA", "unit_measure": "NA", "uom_meas_date": "NA", "customer_signature": "NA", "customer_name": "WESTERN PARTITIONS INC", "address1": "107526 WPI WESTERN PARTITIONS INC", "address2": "1200 S 400 W", "city": "OGDEN", "state": "UT", "zip_code": "

though the response above is only partially correct, it is atleast following the instructions .. i then used the same prompt on the quantized model

./main -m ~/DOLLY/custom-alpaca-lora-dolly-merged-2.0/ggml-model-q4.gguf -p " Below is an instruction that describes a task. Write a response that appropriately completes the request.### Instruction:Extract invoice number, po number, invoice date, invoice total and ship date from the text below. If you do not find it, do not present incorrect values and just mark as NA. Present the result in the format Inv_no: , Inv_dt:, Ship_dt:, Inv_total: , Inv_subtotal: , Po_number: . Concentra INVOICE Occupational Health Centers Rermit TO: of New Jerseyr P.A. PO BOx 8750 Elkridge, MD 21075-8750 Tax Id: 22-3473542 (888) 809-3214 Account: N30 -0440228009 Invoice Date: 08/21/2023 ermk 08/21/2023 Invoice: 515804540 Balance: S48.20 Date Printed: 12/05/2023 Bill To: Skans ka Building USA Location: Skanska USA Building Attn GarY Thomson Attn: Gary Thomson 389 Interpace P kwy Ste 5 389 Interpace P kwy Ste 5 Parsippany, NJ 07054-1132 Parsippany NJ 07054-1132 PLEASE RETURN THIS PORTION WITH YOUR REMITTANCE Name I SSN I Charge Charge Pmts I Date PO Number Birth date Description Amount Adjmnts The Following Services Performed at CMC 48 SNJ Camden 08/21/2023 Couser Damont XXX-XX-6205 Non Reg UDS 7 Pnl 60 BAT 48.20 PO StAcc XX/XX/XXXX Non Reg UDS 27215 6 Panel Balance Due : 848.20 Occupational Health Centers Remit TO Of New Jersey P.A ### Response:"

_Inv_no:,Inv_dt:,Inv_Location: In v_Covid-19 status:,In v_Drug screening:,In v_Mental health services: In v_Medical Records: In v_Immunizations:In v_Health Examinations :,In v_Physical examinations:,In v_Respirator Fit Testing: In v_FMCSA Medical Card: In v_Telehealth: In v_OSHA 10 hour: In v_OSHA 30-Hour: In v_Returned Drug screens: In v_XRay :,In v_DOT medical card: In v_Vaccinations: In v_WC Health Exams :,In v_WC Physical examinations:,In v_WC Medical records ,In v_WC Drug screening:In vWC

totally meaningless .. any pointers on any more options i can use .. i cant understand how the same model , post Q, can give such an incorrect answer

[amygbAI]

not sure why everything i wrote above got striked

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.