Phi-3 medium generation issue

[rjmehta1993]

I am using LoneStriker/Phi-3-medium-4k-instruct-8.0bpw-h8-exl2 but the generation is gibberish. https://huggingface.co/LoneStriker/Phi-3-medium-4k-instruct-8.0bpw-h8-exl2

When I tried the same prompt on the unquantized version of the model, it works fine.

I am using exllamav2 - 0.1.1 latest transformers and torch and flash-attn.

prompt = '''<|user|>
Your goal is to generate hints and descriptions from documents given feedback. You'll be given:

-A document: The text you need to analyze.
-Target Field/Key: The field that corresponds to the Feedback Value in the document. You have to extract value for this field for investor.
-Correct Feedback Value: The answer you're trying to extract, related to the Target Field. The specific value you need extarct using hints, questions, and descriptions. If the feedback value is "Not Found", the model should explain why the valuefor the 'Partnership Expenses' is not present in the document.
-Incorrect Field: A field you should IGNORE; it's NOT the target.

Your Tasks:

-Create Hints: Write three hints that describes the definition of 'Partnership Expenses' in terms of capital call/fund space document type. Never add the actual field/fund names in hints. Add one hint from three that instructs model to avoid extracting Incorrect Field.
-Create Context Descriptions: Write three descriptions of why the value for the 'Partnership Expenses' is 'Not Found' for the Investor. Descriptions should use the document text to identify the correct field for the value and then provide the definition to the field. Never add the actual fund names in hints.

Document below:
PINNACLE  FUND  SERVICES
                                                                                               1095 West Pender Street, Suite 1130
                    P|   N   N   NCLE                                                          Vancouver, BC V6E 2M6 CANADA
                                                                                               Tel +1 604.559.8920
                                                                                               Fax +1604.628.2178
                                                    Demo   VC   Fund  I LP
                                                   Capital  Call  Request
     August 5,                                                                                            Investor ID:
     Test Investor 1
     555 Any Street
     Toronto, ON M4V
     4V4 Canada
     Dear Investor,
     General Partner Inc., the General Partner of Demo  VC   Fund  |, LP (the "Partnership") hereby  advises you  that, in
     accordance with  the terms  of  the Limited  Partnership Agreement,   a Capital Call is currently required  from  the
     Limited Partners.    We   are  hereby  providing you  with  this Capital  Call Notice regarding  Test  Investor   1's
     commitment   to the Partnership, which is due no later than Wednesday, July 13, 2016.
     In accordance with the  Partnership Agreement,  a capital contribution is being drawn  down  from  investors and will
     be applied by the Partnership in the manner set out on the following page.
     Your share of the amount due is CAD $0.
     This amount  is due and  payable  by 5:00pm   Easter Standard Time  on July 13, 2016.  Please ensure  that you check
     with your bank for the cost of the wire transfer.
     Wire transfer instructions are as follows:
          Beneficiary Bank:                     CIBC Private Banking
          Beneficiary Address:                  125  Main Street, Hamilton ON, M5M 5M5
          Beneficiary Bank Number:              010
          Beneficiary Transit Number:           00009
          Beneficiary Account Number:          54-66717
          SWIFT:                                CIBCCATT
          Beneficiary Customer:                 Demo  VC  Fund LP
          Beneficiary Address:                  101  Bay Street, Suite
                                                2340  Toronto ON, M4A
                                                1A3
          Reference:                            Capital Call - Investor 13665
     If you have any questions, please contact David Smith by phone at +1 (604) 559-8920 or by email at
     dsmith@pinnaclefundservices.com.
     Sincerely,
     Pinnacle Fund Services
      www   .pinnaclefundservices.comVp,

NEXT PAGE

                                                                             PINNACLE FUND SERVICES
                                                                             1095 West Pender Street, Suite 1130
     (4         PINN      NCLE                                               Vancouver, BC V6E 2M6 CANADA
                                                                             Tel +1 604.559.8920
                                                                             Fax +1 604.628.2178
                                          Demo  VC Fund  I LP
                                         Capital Call Request
    August 5,                                                                         Investor ID:
    Test Investor 1
    555 Any Street
    Toronto, ON M4V
    4V4 Canada
      Below is a summary of your capital activity to date:
                    a                   Aggregate   Fund      Your  Pro-Rata
              Description                 Commitment               Share
    otal Committed   Capital         $15,000,000.00          $3,000,000.00
   Capital Call Details
   Capital call required for Fund V  $ 100,000.00            $ 20,000.00
   Q2 2024 Management   Fees         45,000.00               9,000.00
   stated    Fund Expenses for Q2    15.000.00               3.000.00
   Placement Agent Fees              18,750.00               3,750.00
   Adjustments  &  Returns
   Return of Capital - Management    (7,500.00)              (1,500.00)
   Proceeds from Recent Exit         11,250.00               2,250.00
   Working Capital Deficit           7,500.00                1,500.00
   General Partner Contribution      3,750.00                750.00
   Realized Gain on Cash Investment  4,500.00                900.00
   Realized Loss on Equity Sale      (3,000.00)              (600.00)
   Clawback Obligation               3,750.00                750.00
   Dividend Income                   2,250.00                450.00
   Return of Invested Capital        6,000.00                1,200.00
   Return of Invested Stock Capital  5,250.00                1,050.00
   Management  Fee Rebate            3,000.00                600.00
   Refund of Fund Expenses           2,250.00                450.00
   Overcalled Capital Refund         3,000.00                600.00
   Tax Liability                     1,500.00                300.00
   Return of Fund Expenses (Interim) (4,500.00)              (900.00)
   Recallable Amount                 750.00                  150.00
    otal Amount  Applied  to Capital
                                      285,750.00               57,150.00
                    |          Cs                            ®
   Less: Amount Applied from Existing (27,150.00)
   Deposit
    otal Cash Proceeds   Due         $285,750.00             $30,000.00

NEXT PAGE

                                                                                         PINNACLE  FUND  SERVICES
                                                                                         1095 West Pender Street, Suite 1130
                  P|   N   N   NCLE                                                      Vancouver, BC V6E 2M6 CANADA
                                                                                         Tel +1 604.559.8920
                                                                                         Fax +1604.628.2178
                                                Demo   VC   Fund  I LP
                                                Capital Call  Request
    August 5,                                                                                       Investor ID:
    Test Investor 1
    555 Any Street
    Toronto, ON M4V
    4V4 Canada
       Below is a summary of your deposit activity to date:
                                                               A
                        Description                           Amount
                                P                              (CAD)
      Cumulative deposits received, before this capital call $2,000,000.00
      Total amount applied to deposit balance, before this   ($1,850,000.0
      call                                                   0)
      Total amount applied to deposit balance, this capital  ($38,100.00)
      Deposit balance remaining, after this capital call     $111,900.00
        The deposit balance above being held by the Partnership is non-interest bearing and will be applied to future capital
        calls.
     www   .pinnaclefundservices.comVp,

END DOCUMENT.

Target Key/Field: Partnership Expenses

Correct Feedback Value: Not Found

Incorrect Field: Realized Loss on Equity Sale

The value for the key/field "Partnership Expenses" should be "Not Found".

JSON: {"hints":[],"descriptions":[]}

Remember:

-Three hints. Never add the actual field/fund names in hints.
-Three descriptions of why the value for the 'Partnership Expenses' is 'Not Found' for the Investor (without stating theirvalues).
-These should guide the model to locate the "Partnership Expenses" and extract the correct feedback value from the document.

COMPLETE AND RESPOND ONLY THE JSON AND PROVIDE NO EXPLANATION. <|end|>
<|assistant|>
'''
input_ids = tokenizer.encode(prompt)
sys.stdout.flush()
settings = ExLlamaV2Sampler.Settings()
settings.temperature = 0.9
settings.top_k = 5
settings.top_p = 0.9
settings.token_repetition_penalty = 1.15
max_new_tokens = 30
generator.set_stop_conditions([128009,tokenizer.eos_token_id,"eot"])
generator.begin_stream_ex(input_ids, settings)

generated_tokens = 0
ans = ''
while True:
    res = generator.stream_ex()
    chunk = res["chunk"]
    eos = res["eos"]
    generated_tokens += 1
    sys.stdout.flush()
    if eos or generated_tokens == max_new_tokens: break
    ans += chunk
    #yield chunk
    print(chunk,end='')

EXLLAMAV2 OUTPUT:

Feedback ValuesExpense.

Solution=Value Extract the solution<|end|><|assistant|> Around themself.
EXPECTORIGNOUNITSometimes they must find exactly whatnot found.<|end|><|assistant|> SolutionExtracted values.

Output the output Format: [{"output. They are asked to OutputFormatAnswer.<|end|><|assistant|> Findingsolutionself.

IMPORTEDUCtogether.

TaskInstructions.

It

UNQUANTIZED TRANSFORMERS OUTPUT:

{

  "hints": [

    "Look for sections detailing financial transactions related to the partnership's operations.",

    "Identify any line items that may represent expenses incurred by the partnership.",

    "Search for terms that typically indicate expenses, such as 'fees', 'costs', or 'expenses'."

  ],

  "descriptions": [

    "The document does not contain a specific line item or section labeled as 'Partnership Expenses'.",

    "While the document lists various financial activities, none are explicitly categorized under 'Partnership Expenses'.",

    "The document details capital calls, management fees, and other investment-related transactions, but does not itemize expenses under the heading 'Partnership Expenses'."

  ]

}

[turboderp]

There's something strange going on with that model. I don't know if it's converted wrong somehow or if there's something in the architecture that differs between mini-4k (which works) and medium-4k.

I tested it with medium-128k, though, and I got this:

{
  "hints": [
    "Look for costs associated directly to partnership operations.",
    "Search within sections detailing operational expenses.",
    "Ignore entries about equity loss."
  ],
  "context_descriptions": [
    "Despite detailed breakdowns of various financial aspects such as management fees, placement agent fees etc., there was no explicit mention of 'Partnership Expenses'. These typically include overhead like office rentals, utilities etc. However none were listed under these categories indicating they might not exist.",
    "'Realized Loss on Equity Sale', while an important entry, does not fall into the category of 'Partnership Expenses'; hence its absence explains why we cannot find the desired data.",
    "While other line items indicate monetary transactions linked to the operation of the venture, 'Partnership Expenses' specifically refers to recurring operating costs, none of which seem to be indicated thus leading us towards our conclusion."
  ]
}

So maybe that's an option until I can get around to figuring out what's up with the 4k version? You can always set config.max_seq_len = 4096 to limit VRAM usage if necessary.

[rjmehta1993]

Thanks

[rjmehta1993]

I tried with multiple exl quants for phi-medium-4k and they all fail. I don't think all exl quants could go wrong. Any suggestions @turboderp