neulab / prompt2model

prompt2model - Generate Deployable Models from Natural Language Instructions
Apache License 2.0
1.95k stars 173 forks source link

IsADirectoryError in cli_demo.py #260

Closed neubig closed 1 year ago

neubig commented 1 year ago

I'm getting an IsADirectory error when I run the CLI demo.

 python cli_demo.py 
                                               ____                            _     ____    __  __           _      _                                              
                                              |  _ \ _ __ ___  _ __ ___  _ __ | |_  |___ \  |  \/  | ___   __| | ___| |                                             
                                              | |_) | '__/ _ \| '_ ` _ \| '_ \| __|   __) | | |\/| |/ _ \ / _` |/ _ \ |                                             
                                              |  __/| | | (_) | | | | | | |_) | |_   / __/  | |  | | (_) | (_| |  __/ |                                             
                                              |_|   |_|  \___/|_| |_| |_| .__/ \__| |_____| |_|  |_|\___/ \__,_|\___|_|                                             
                                                                        |_|                                                                                         

Do you want to start from scratch? (y/n)
y
Enter your task description and few-shot examples (or 'done' to finish):
Predict whether a beer review is positive or negative. For example:

Review: It is very smooth, it drinks like butter.
Answer: Positive

Review: It's so bitter that I could barely drink it.
Answer: Negative.
done
Parsing prompt...
Prompt parsed.
Retrieving dataset...
Downloading and preparing dataset json/default to /Users/gneubig/.cache/huggingface/datasets/json/default-5fbd1c2fe3f27327/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e...
Dataset json downloaded and prepared to /Users/gneubig/.cache/huggingface/datasets/json/default-5fbd1c2fe3f27327/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e. Subsequent calls will reuse this data.

-------------------------------------------------

Here are the datasets I've retrieved for you:
#       Name    Description
1):     amazon_polarity The Amazon reviews dataset consists of reviews from amazon. The data span a period of 18 years, including ~35 million reviews up to March 2013. Reviews include product and user information, ratings, and a plaintext review.
2):     turkish_product_reviews Turkish Product Reviews. This repository contains 235.165 product reviews collected online. There are 220.284 positive, 14881 negative reviews.
3):     sst2    The Stanford Sentiment Treebank consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence. We use the two-way (positive/negative) class split, and use only sentence-level labels.
4):     maritaca-ai/sst2_pt     The Stanford Sentiment Treebank consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence. We use the two-way (positive/negative) class split, and use only sentence-level labels.
5):     muchocine       The Muchocine reviews dataset contains 3,872 longform movie reviews in Spanish language, each with a shorter summary review, and a rating on a 1-5 scale.
6):     ruanchaves/b2w-reviews01        B2W-Reviews01 is an open corpus of product reviews. It contains more than 130k e-commerce customer reviews, collected from the Americanas.com website between January and May, 2018. B2W-Reviews01 offers rich information about the reviewer profile, such as gender, age, and geographical location. The corpus also has two different review rates
7):     humicroedit     This new dataset is designed to assess the funniness of edited news headlines.
8):     ruanchaves/test_stanford        Manually Annotated Stanford Sentiment Analysis Dataset by Bansal et al..
9):     ruanchaves/stan_small   Manually Annotated Stanford Sentiment Analysis Dataset by Bansal et al..
10):    cdt     The Cyberbullying Detection task was part of 2019 edition of PolEval competition. The goal is to predict if a given Twitter message contains a cyberbullying (harmful) content.
11):    snli    The SNLI corpus (version 1.0) is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE).
12):    shibing624/snli-zh      The SNLI corpus (version 1.0) is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE).
13):    erjoy/setFit    The SNLI corpus (version 1.0) is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE).
14):    erjoy/sentBert-v2       The SNLI corpus (version 1.0) is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE).
15):    sst     The Stanford Sentiment Treebank, the first corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment in language.
16):    jeffnyman/rotten_tomatoes_reviews       Movie Review Dataset.  This is a dataset containing 4,265 positive and 4,265 negative processed sentences from Rotten Tomatoes movie reviews.
17):    polinaeterna/amazon_us_reviews  Amazon Customer Reviews (a.k.a. Product Reviews) is one of Amazons iconic products. In a period of over two decades since the first review in 1995, millions of Amazon customers have contributed over a hundred million reviews to express opinions and describe their experiences regarding products on the Amazon.com website. This makes Amazon Customer Reviews a rich source of information for academic researchers in the fields of Natural Language Processing (NLP), Information Retrieval (IR), and Machine Learning (ML), amongst others. Accordingly, we are releasing this data to further research in multiple disciplines related to understanding customer product experiences. Specifically, this dataset was constructed to represent a sample of customer evaluations and opinions, variation in the perception of a product across geographical regions, and promotional intent or bias in reviews.  Over 130+ million customer reviews are available to researchers as part of this release. The data is available in TSV files in the amazon-reviews-pds S3 bucket in AWS US East Region. Each line in the data files corresponds to an individual review (tab delimited, with no quote and escape characters).  Each Dataset contains the following columns:  - marketplace: 2 letter country code of the marketplace where the review was written. - customer_id: Random identifier that can be used to aggregate reviews written by a single author. - review_id: The unique ID of the review. - product_id: The unique Product ID the review pertains to. In the multilingual dataset the reviews for the same product in different countries can be grouped by the same product_id. - product_parent: Random identifier that can be used to aggregate reviews for the same product. - product_title: Title of the product. - product_category: Broad product category that can be used to group reviews (also used to group the dataset into coherent parts). - star_rating: The 1-5 star rating of the review. - helpful_votes: Number of helpful votes. - total_votes: Number of total votes the review received. - vine: Review was written as part of the Vine program. - verified_purchase: The review is on a verified purchase. - review_headline: The title of the review. - review_body: The review text. - review_date: The date the review was written.
18):    amazon_us_reviews       Amazon Customer Reviews (a.k.a. Product Reviews) is one of Amazons iconic products. In a period of over two decades since the first review in 1995, millions of Amazon customers have contributed over a hundred million reviews to express opinions and describe their experiences regarding products on the Amazon.com website. This makes Amazon Customer Reviews a rich source of information for academic researchers in the fields of Natural Language Processing (NLP), Information Retrieval (IR), and Machine Learning (ML), amongst others. Accordingly, we are releasing this data to further research in multiple disciplines related to understanding customer product experiences. Specifically, this dataset was constructed to represent a sample of customer evaluations and opinions, variation in the perception of a product across geographical regions, and promotional intent or bias in reviews.  Over 130+ million customer reviews are available to researchers as part of this release. The data is available in TSV files in the amazon-reviews-pds S3 bucket in AWS US East Region. Each line in the data files corresponds to an individual review (tab delimited, with no quote and escape characters).  Each Dataset contains the following columns:  - marketplace: 2 letter country code of the marketplace where the review was written. - customer_id: Random identifier that can be used to aggregate reviews written by a single author. - review_id: The unique ID of the review. - product_id: The unique Product ID the review pertains to. In the multilingual dataset the reviews for the same product in different countries can be grouped by the same product_id. - product_parent: Random identifier that can be used to aggregate reviews for the same product. - product_title: Title of the product. - product_category: Broad product category that can be used to group reviews (also used to group the dataset into coherent parts). - star_rating: The 1-5 star rating of the review. - helpful_votes: Number of helpful votes. - total_votes: Number of total votes the review received. - vine: Review was written as part of the Vine program. - verified_purchase: The review is on a verified purchase. - review_headline: The title of the review. - review_body: The review text. - review_date: The date the review was written.
19):    dbpedia_14      The DBpedia ontology classification dataset is constructed by picking 14 non-overlapping classes from DBpedia 2014. They are listed in classes.txt. From each of thse 14 ontology classes, we randomly choose 40,000 training samples and 5,000 testing samples. Therefore, the total size of the training dataset is 560,000 and testing dataset 70,000. There are 3 columns in the dataset (same for train and test splits), corresponding to class index (1 to 14), title and content. The title and content are escaped using double quotes ("), and any internal double quote is escaped by 2 double quotes (""). There are no new lines in title or content.
20):    TurkuNLP/Suomi24-toxicity-annotated     This dataset consists of Suomi24 comments which have been labeled by human raters for toxic behavior.
21):    vesteinn/swe-nerc       The corpus consists of ca. 150.000 words of text.
22):    wiki_qa Wiki Question Answering corpus from Microsoft
23):    allegro_reviews Allegro Reviews is a sentiment analysis dataset, consisting of 11,588 product reviews written in Polish and extracted from Allegro.pl - a popular e-commerce marketplace. Each review contains at least 50 words and has a rating on a scale from one (negative review) to five (positive review).  We recommend using the provided train/dev/test split. The ratings for the test set reviews are kept hidden. You can evaluate your model using the online evaluation tool available on klejbenchmark.com.
24):    bookcorpus      Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story.This work aims to align books to their movie releases in order to providerich descriptive explanations for visual content that go semantically farbeyond the captions available in current datasets. \
25):    matejklemen/falko_merlin        Falko-MERLIN is a grammatical error correction corpus consisting of essays and exams.

-------------------------------------------------

If none of these are relevant to your prompt, we'll only use generated data. Are any of these datasets relevant? (y/N)
3
Which dataset would you like to use? Give the number between 1 and 25.
3

-------------------------------------------------

Found cached dataset sst2 (/Users/gneubig/.cache/huggingface/datasets/sst2/default/2.0.0/9896208a8d85db057ac50c72282bcb8fe755accc671a57dd8059d4e130961ed5)

-------------------------------------------------

Loaded dataset. Example row:
{
    "idx": 0,
    "sentence": "hide new secretions from the parental units ",
    "label": 0
}

Which column(s) should we use as input? Provide a comma-separated list from: idx, sentence, label.
sentence
Will use the columns ["sentence"] as input.

Which column(s) should we use as the target? Choose a single value from: idx, sentence, label.
label
Will use the column "label" as our target.

-------------------------------------------------

Retrieving model...
  0%|                                                                                                                                                                    | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/Users/gneubig/work/prompt2model/cli_demo.py", line 435, in <module>
    main()
  File "/Users/gneubig/work/prompt2model/cli_demo.py", line 163, in main
    retriever = DescriptionModelRetriever(
  File "/Users/gneubig/work/prompt2model/prompt2model/model_retriever/description_based_retriever.py", line 105, in __init__
    self.load_model_info()
  File "/Users/gneubig/work/prompt2model/prompt2model/model_retriever/description_based_retriever.py", line 148, in load_model_info
    or len(open(os.path.join(self.model_descriptions_index_path, f)).read())
IsADirectoryError: [Errno 21] Is a directory: 'huggingface_data/huggingface_models/model_info/model_info'
zhaochenyang20 commented 1 year ago

Yeah. I encountered this. I just moved all the .json into huggingface_data/model_info, instead of huggingface_data/model_info/model_info. @viswavi

I think it's related to the dataset retriever.

neubig commented 1 year ago

I have time now, I'll fix it.