Inference example

[tomhoper]

Would it be possible to provide a self-contained example of how to use the model for one task and one input text? Including the full input format? How exactly is the instruction template file supposed to be used? For example, for the AnatEM task?

[tomhoper]

hi @Mihir3009 and @Mirzyaaliii checking whether you saw this? thanks

[Mihir3009]

Thanks for reminding us. We have added an inference example in Readme. I hope that helps. Please let us know any queries.

[tomhoper]

Thank you @Mihir3009 . Below I tried doing this for the chemical-protein relation example. As a sanity check I simply used as Instance the exact same example given in Positive Example, including entity_1 and entity_2 from the example, however when I try it, it doesn't predict the correct relation (even though it is given explicitly in the example, and thus should in theory just be copied). The correct relation should be CPR:4, but I get CPR:6. Is the format I used below correct?

"Instruction: Definition: The goal of this task is to generate a relationship between chemical and protein entities. In this task, you are given a combination of Pubmed title and abstract as context, two entities as entity_1 and entity_2. For the given entities, your task is to generate one of these five relations: CPR:3 (activation), CPR:4 (inhibition), CPR:5 (agonist), CPR:6 (antagonist) and CPR:9 (substrate)., Positive Examples: [[input: Context: Use of angiotensin (Ang) II AT1 receptor antagonists for treatment of hypertension is rapidly increasing, yet direct comparisons of the relative efficacy of antagonists to block the renin-angiotensin system in humans are lacking. In this study, the Ang II receptor blockade induced by the recommended starting dose of 3 antagonists was evaluated in normotensive subjects in a double-blind, placebo-controlled, randomized, 4-way crossover study. At 1-week intervals, 12 subjects received a single dose of losartan (50 mg), valsartan (80 mg), irbesartan (150 mg), or placebo. Blockade of the renin-angiotensin system was assessed before and 4, 24, and 30 hours after drug intake by 3 independent methods: inhibition of the blood pressure response to exogenous Ang II, in vitro Ang II receptor assay, and reactive changes in plasma Ang II levels. At 4 hours, losartan blocked 43% of the Ang II-induced systolic blood pressure increase; valsartan, 51%; and irbesartan, 88% (P<0.01 between drugs). The effect of each drug declined with time. At 24 hours, a residual effect was found with all 3 drugs, but at 30 hours, only irbesartan induced a marked, significant blockade versus placebo. Similar results were obtained when Ang II receptor blockade was assessed with an in vitro receptor assay and by the reactive rise in plasma Ang II levels. This study thus demonstrates that the first administration of the recommended starting dose of irbesartan induces a greater and longer lasting Ang II receptor blockade than that of valsartan and losartan in normotensive subjects. entity_1: irbesartan entity_2: Ang II.,  output: CPR:4, explanation: From the context, we can see that the dose of irbesartan causes inhibition for Ang II protein, hence generated relation inhibition (CPR:4) is right.]]; Instance:  input: Context: Use of angiotensin (Ang) II AT1 receptor antagonists for treatment of hypertension is rapidly increasing, yet direct comparisons of the relative efficacy of antagonists to block the renin-angiotensin system in humans are lacking. In this study, the Ang II receptor blockade induced by the recommended starting dose of 3 antagonists was evaluated in normotensive subjects in a double-blind, placebo-controlled, randomized, 4-way crossover study. At 1-week intervals, 12 subjects received a single dose of losartan (50 mg), valsartan (80 mg), irbesartan (150 mg), or placebo. Blockade of the renin-angiotensin system was assessed before and 4, 24, and 30 hours after drug intake by 3 independent methods: inhibition of the blood pressure response to exogenous Ang II, in vitro Ang II receptor assay, and reactive changes in plasma Ang II levels. At 4 hours, losartan blocked 43% of the Ang II-induced systolic blood pressure increase; valsartan, 51%; and irbesartan, 88% (P<0.01 between drugs). The effect of each drug declined with time. At 24 hours, a residual effect was found with all 3 drugs, but at 30 hours, only irbesartan induced a marked, significant blockade versus placebo. Similar results were obtained when Ang II receptor blockade was assessed with an in vitro receptor assay and by the reactive rise in plasma Ang II levels. This study thus demonstrates that the first administration of the recommended starting dose of irbesartan induces a greater and longer lasting Ang II receptor blockade than that of valsartan and losartan in normotensive subjects. entity_1: irbesartan entity_2: Ang II.,  output: ?"

`

[tomhoper]

Wondering if you have a chance to look at this in the past 19 days. thanks.

[Mihir3009]

Sorry for the late response. We missed the previous email about this issue.

Your input format is correct. However, we chose to use positive examples for only generation tasks since we observed that they are not helpful in classification tasks. We mentioned this in the paper as well (see in Section 5.1 - Example selection).

I tried one sample from the same task with the below input format:

Instruction: Definition: The goal of this task is to generate a relationship between chemical and protein entities. In this task, you are given a combination of Pubmed title and abstract as context, two entities as entity_1 and entity_2. For the given entities, your task is to generate one of these five relations: CPR:3 (activation), CPR:4 (inhibition), CPR:5 (agonist), CPR:6 (antagonist) and CPR:9 (substrate).; Instance: input: Context: BACKGROUND & AIMS: The critical role of cyclooxygenase (COX) products in maintenance of renal function in cirrhosis with ascites discourages the use of nonsteroidal anti-inflammatory drugs in this disease. The recent development of selective COX-2 inhibitors opens new avenues for the use of these compounds in decompensated cirrhosis. The current study evaluates the effects of a selective COX-2 inhibitor (SC-236) on renal function in cirrhotic rats with ascites. METHODS: In protocol 1, urine volume, urinary excretion of sodium and prostaglandins, glomerular filtration rate, and renal plasma flow were measured before and after administration of SC-236 (n = 12) or ketorolac (n = 10) to rats with cirrhosis. Protocol 2 was aimed at assessing the effects of COX inhibitors on renal water metabolism in 28 cirrhotic rats. RESULTS: Administration of SC-236 to cirrhotic animals did not produce significant renal effects, whereas administration of the nonselective COX-1/COX-2 inhibitor, ketorolac, resulted in a marked reduction in urine volume, urinary excretion of prostaglandins, and glomerular filtration rate and in a significant impairment in renal water metabolism. CONCLUSIONS: These findings indicate that SC-236 does not significantly impair renal function in rats with cirrhosis. entity_1: SC-236 entity_2: COX-2, output: ?

The gold output is CPR:4 and it's giving me CPR:4 as a prediction.

Another point I want to make is that the F1-score for this task we get with In-BoXBART is ~64%, hence there might be instances in the test set for which you may get wrong predictions as well.

Thank you!