xenova
commented
1 year ago Sure! I can add this 👍
In the meantime, could you please provide some example python code (and output) using the pipeline function?
Closed trenta3 closed 1 year ago
xenova
commented
1 year ago Sure! I can add this 👍
In the meantime, could you please provide some example python code (and output) using the pipeline function?
trenta3
commented
1 year ago Yeah sure, I wrote some phrases in different languages
import json
from transformers import pipeline
classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
sequences = [
"Devo ancora pagare la bolletta del gas",
"I still have to pay the gas bill",
"Ich werde morgen klettern. Kommst du mit?",
"I will go tomorrow to hike. Do you want to come?",
"Je suis cusinier dans un restaurant",
"I am a cook in a restaurant",
"El Papa entrena cada dos días",
"The pope works out every two days",
]
candidate_labels = ["travel", "hiking", "sport", "dancing", "cooking", "authority", "utilities", "work"]
for sequence in sequences:
for multi_label in [True, False]:
result = classifier(sequence, candidate_labels, multi_label=multi_label)
result["multi_label"] = multi_label
for idx, key in enumerate(["sequence", "labels", "scores", "multi_label"]):
prefix = "{ " if idx == 0 else " "
suffix = " }" if idx + 1 == len(result) else ","
print(prefix + '"' + key + '": ' + json.dumps(result[key]) + suffix)which when run prints
{ "sequence": "Devo ancora pagare la bolletta del gas",
"labels": ["cooking", "utilities", "work", "authority", "travel", "sport", "hiking", "dancing"],
"scores": [0.991447389125824, 0.9612486362457275, 0.8709714412689209, 0.2068490833044052, 0.010441462509334087, 0.0028406695928424597, 0.001590077648870647, 0.001008122693747282],
"multi_label": true }
{ "sequence": "Devo ancora pagare la bolletta del gas",
"labels": ["cooking", "utilities", "work", "authority", "travel", "sport", "dancing", "hiking"],
"scores": [0.5983912348747253, 0.25206613540649414, 0.10138948261737823, 0.02908896468579769, 0.007862254977226257, 0.004890661220997572, 0.003495732555165887, 0.00281552504748106],
"multi_label": false }
{ "sequence": "I still have to pay the gas bill",
"labels": ["utilities", "work", "authority", "cooking", "hiking", "travel", "sport", "dancing"],
"scores": [0.9984584450721741, 0.8743591904640198, 0.07910361140966415, 0.01081591285765171, 0.005842195358127356, 0.004699218086898327, 0.002067639259621501, 0.0006328970775939524],
"multi_label": true }
{ "sequence": "I still have to pay the gas bill",
"labels": ["utilities", "work", "authority", "hiking", "travel", "sport", "cooking", "dancing"],
"scores": [0.8816797733306885, 0.08556484431028366, 0.014206331223249435, 0.005271111615002155, 0.004757406190037727, 0.0037542979698628187, 0.0027197289746254683, 0.002046391600742936],
"multi_label": false }
{ "sequence": "Ich werde morgen klettern. Kommst du mit?",
"labels": ["travel", "work", "cooking", "authority", "utilities", "hiking", "sport", "dancing"],
"scores": [0.9717028141021729, 0.9534307718276978, 0.8326925039291382, 0.5644204020500183, 0.4671698808670044, 0.42344552278518677, 0.2895689010620117, 0.1618945449590683],
"multi_label": true }
{ "sequence": "Ich werde morgen klettern. Kommst du mit?",
"labels": ["travel", "work", "cooking", "hiking", "authority", "dancing", "utilities", "sport"],
"scores": [0.3776012659072876, 0.3074505031108856, 0.1677756905555725, 0.04651238024234772, 0.0353471115231514, 0.02822057157754898, 0.02182605117559433, 0.015266447328031063],
"multi_label": false }
{ "sequence": "I will go tomorrow to hike. Do you want to come?",
"labels": ["hiking", "travel", "sport", "authority", "work", "utilities", "cooking", "dancing"],
"scores": [0.9942196607589722, 0.9879510998725891, 0.8137379884719849, 0.8078529834747314, 0.008462291210889816, 0.0036291773431003094, 0.0003602092037908733, 0.00025504903169348836],
"multi_label": true }
{ "sequence": "I will go tomorrow to hike. Do you want to come?",
"labels": ["hiking", "travel", "sport", "authority", "work", "utilities", "cooking", "dancing"],
"scores": [0.5533843636512756, 0.3380189538002014, 0.058553021401166916, 0.038313984870910645, 0.005027722101658583, 0.003661608323454857, 0.0016780057922005653, 0.001362350769340992],
"multi_label": false }
{ "sequence": "Je suis cusinier dans un restaurant",
"labels": ["cooking", "work", "authority", "dancing", "travel", "sport", "utilities", "hiking"],
"scores": [0.9927762150764465, 0.5199383497238159, 0.3442917764186859, 0.04534728452563286, 0.03189418837428093, 0.010471467860043049, 0.004911400377750397, 8.186535706045106e-05],
"multi_label": true }
{ "sequence": "Je suis cusinier dans un restaurant",
"labels": ["cooking", "work", "authority", "travel", "dancing", "utilities", "sport", "hiking"],
"scores": [0.8023020029067993, 0.08228448033332825, 0.06041828915476799, 0.01709311082959175, 0.012606504373252392, 0.012165196239948273, 0.011867685243487358, 0.0012627566466107965],
"multi_label": false }
{ "sequence": "I am a cook in a restaurant",
"labels": ["cooking", "work", "authority", "utilities", "sport", "dancing", "travel", "hiking"],
"scores": [0.996135950088501, 0.9956350922584534, 0.0006340791587717831, 0.0001712672965368256, 0.00014821744116488844, 6.90977685735561e-05, 6.830081110820174e-05, 3.603912773542106e-05],
"multi_label": true }
{ "sequence": "I am a cook in a restaurant",
"labels": ["cooking", "work", "authority", "utilities", "sport", "travel", "dancing", "hiking"],
"scores": [0.6078384518623352, 0.38483986258506775, 0.0018544865306466818, 0.001734126009978354, 0.0013757537817582488, 0.0009676438057795167, 0.0007441536290571094, 0.0006455256370827556],
"multi_label": false }
{ "sequence": "El Papa entrena cada dos d\u00edas",
"labels": ["authority", "work", "sport", "utilities", "dancing", "cooking", "travel", "hiking"],
"scores": [0.9936944246292114, 0.9889096021652222, 0.8996925354003906, 0.01591523177921772, 0.007262231782078743, 0.0031147277913987637, 0.002472982043400407, 0.001052334439009428],
"multi_label": true }
{ "sequence": "El Papa entrena cada dos d\u00edas",
"labels": ["work", "authority", "sport", "utilities", "travel", "dancing", "cooking", "hiking"],
"scores": [0.4954349398612976, 0.3925631046295166, 0.08369459211826324, 0.012066002935171127, 0.0060667796060442924, 0.004071310628205538, 0.0032307838555425406, 0.002872476354241371],
"multi_label": false }
{ "sequence": "The pope works out every two days",
"labels": ["authority", "work", "sport", "hiking", "travel", "utilities", "dancing", "cooking"],
"scores": [0.9307130575180054, 0.9188652038574219, 0.7416408658027649, 0.033093079924583435, 0.0051829395815730095, 0.0014411408919841051, 0.00011244666529819369, 7.980306691024452e-05],
"multi_label": true }
{ "sequence": "The pope works out every two days",
"labels": ["work", "authority", "sport", "travel", "utilities", "hiking", "cooking", "dancing"],
"scores": [0.46334323287010193, 0.2907411456108093, 0.19170965254306793, 0.01879042014479637, 0.018452158197760582, 0.008880219422280788, 0.004273386672139168, 0.003809793619439006],
"multi_label": false }
Hi! Just an update: So, I got it working, but the bart model you provided seems to give very poor performance when quantized. The results are identical when using the unquantized version, but the model is quite large, so, it wouldn't work in a browser environment (it will probably work in Node, if that is your intended environment).
I've verified the quantized outputs (JS library vs python library) and although they are the same, they differ quite a lot from the unquantized version.
Although a drop in performance is expected for quantization (due to the ~4x reduction in model size), the provided bart model seems to produce very poor results. Do you perhaps have any other (possibly smaller) models you think it would work with?
xenova
commented
1 year ago I've pushed those changes (https://github.com/xenova/transformers.js/commit/230984e59c3d2aaa87aa353aaf274a5ffb01f610) in case you want to get started early.
I will add example usage and documented support tomorrow. I'll close the issue when that is done :)
trenta3
commented
1 year ago Thank you a lot for all the work. I will look into alternative and smaller models that hopefully won't have such a performance problem.
xenova
commented
1 year ago I ran different (quantized) models on your example inputs (using this code)
let classifier = await pipeline('zero-shot-classification', 'typeform/distilbert-base-uncased-mnli');
let sequences = [
"Devo ancora pagare la bolletta del gas",
"I still have to pay the gas bill",
"Ich werde morgen klettern. Kommst du mit?",
"I will go tomorrow to hike. Do you want to come?",
"Je suis cusinier dans un restaurant",
"I am a cook in a restaurant",
"El Papa entrena cada dos días",
"The pope works out every two days",
];
let candidate_labels = ["travel", "hiking", "sport", "dancing", "cooking", "authority", "utilities", "work"];
for (let sequence of sequences) {
for (let multi_label of [true, false]) {
let result = await classifier(sequence, candidate_labels, { multi_label });
result["multi_label"] = multi_label;
console.log(JSON.stringify(result, null, 2));
}
}and here is what I got:
As mentioned before, there is quite a drop in performance when using the quantized models. Not 100% sure why this is, but there are some warning messages when exporting, so, that might have to do with it.
typeform/distilbert-base-uncased-mnli seems to perform well on your examples (particularly for the English ones). The others are much less confident with their outputs.
If possible, I would recommend that you use unquantized versions of the models (you would have to export these yourself, see README for instructions).
xenova
commented
1 year ago I'm just finishing up the demo, then will publish the update :)
Edit: GitHub actions/pages are down right now (https://www.githubstatus.com/incidents/z3c6q056q332)... will publish when resolved :')
xenova
commented
1 year ago Here it is! 🥳 v1.3.5 is out (https://www.npmjs.com/package/@xenova/transformers) and zero-shot classification is now supported.
Example code:
// npm i @xenova/transformers
import { pipeline } from '@xenova/transformers';
// Allocate pipeline
// (uses 'typeform/distilbert-base-uncased-mnli' by default)
let classifier = await pipeline('zero-shot-classification');
// Specify text and classes
let text = 'I have a problem with my iphone that needs to be resolved asap!';
let classes = ['urgent', 'not urgent', 'phone', 'tablet', 'microwave'];
let output = await classifier(text, classes, { multi_label: true });
// {
// sequence: "I have a problem with my iphone that needs to be resolved asap!",
// labels: ["phone", "urgent", "not urgent", "tablet", "microwave"],
// scores: [0.783870881, 0.451184637, 0.009843352, 0.001410044, 0.000039902]
// }Let me know if you have any other questions 👍
trenta3
commented
1 year ago Thank you, this was really fast! I haven't had the time to search for another model, but will report back on it.
I'm requesting to add the pipeline "zero-shot-classification" for label extraction from a text based on a pretrained model and hypotheses. To view usage examples: https://huggingface.co/facebook/bart-large-mnli
This is a very requested feature as can be measured by the number of downloads of the linked model in the last month (> 2.5M) And thank you for the extremely useful work you are doing!