Test Data Gathering

[VeriHeppi]

Find out where to get test data?

[VeriHeppi]

Where to get possible training/testing data for the LLM of choice?

[VeriHeppi]

avoindata.fi

[Lauri-Niemela]

ChatGPT prompt:

I am creating an LLM-based text classifier. The purpose of this project is to input text-based data and have the program analyse and classify themes and other things from the data. The program will run in cloud and will be created using Python, and will also use Docker.

What kind of data would be best for testing such a program? What format should the data be in? How much data is enough for tests?

Answer:

For testing your LLM-based text classifier, selecting the right data, format, and quantity is crucial for evaluating its performance. Here are some guidelines:

1. Types of Data for Testing

The best testing data depends on the nature of your text classification tasks, but here are some commonly used types:

• Labeled Dataset: You'll need a dataset with clear labels that correspond to the themes or categories you are classifying. For example, if you're classifying news articles by topic (e.g., sports, politics, entertainment), each text sample should have a corresponding label.
• Diverse Topics: The data should cover a variety of themes to test the versatility of your classifier. Ensure it includes various domains such as articles, emails, social media posts, etc., depending on your project's focus.
• Balanced Data: Ensure that your dataset is balanced, meaning each category has a sufficient number of examples to prevent model bias toward any specific class.

• Real-World Data: Try using real-world, unstructured text data to see how well the model handles the noise, such as typos, slang, or abbreviations, common in user-generated content.

  Sources:

• Public datasets: Datasets like AG News, IMDB reviews, 20 Newsgroups, or Sentiment140 can be good starting points.
• Custom data: Collect domain-specific data relevant to your use case (e.g., customer reviews, product descriptions, research papers).

2. Format of the Data

The most common format for text classification tasks is plain text files with corresponding labels. Here are a few formats:

• CSV: Each row contains a text sample and a label. Example:

   text,label
   "The new smartphone model has excellent features.", "tech"
   "The stock market saw significant growth today.", "business"

• JSON: Useful for more complex structures where you may need to include additional metadata. Example:

   [
     {"text": "The new smartphone model has excellent features.", "label": "tech"},
     {"text": "The stock market saw significant growth today.", "label": "business"}
   ]

• TXT: Plain text files where each line contains the text followed by its label, often used for simple applications.

• Hugging Face Dataset Format: If you're using libraries like Transformers by Hugging Face, you can directly load their datasets in a structured format.

  Recommendation: Use CSV or JSON for initial testing, as they are human-readable and easy to manipulate in Python.

3. How Much Data Is Enough for Tests?

• Small-Scale Testing: For initial testing of your infrastructure and code, even a small dataset (500–1,000 samples) can help you evaluate whether the system works.
• Benchmark Testing: To properly evaluate your classifier’s performance, you'll need a moderate dataset (5,000–50,000 samples) to measure accuracy, precision, recall, and F1 scores.

• Stress Testing: For scaling tests or production-level deployment, larger datasets (50,000+ samples) are beneficial to assess how well the classifier handles large volumes of data.

  General Rule:

• Start small for testing functionality.
• Gradually increase the dataset size to assess performance, efficiency, and accuracy.

4. Balancing Data for Testing

Ensure you have an even distribution of samples across all categories to avoid skewing model performance toward any single class. For instance, if you're classifying text into five categories, aim for a roughly equal number of examples in each category. If your dataset is imbalanced, you might consider techniques like oversampling or using metrics like weighted accuracy during evaluation.

Conclusion

• Data Type: Labeled datasets with diverse themes relevant to your classification task.
• Format: CSV or JSON formats are preferred for ease of use.
• Data Size: Start with a small dataset for debugging, but aim for at least 5,000 samples for meaningful performance tests, and 50,000+ samples for stress testing.

[Lauri-Niemela]

Data that could be good for software testing:

https://www.avoindata.fi/data/fi/dataset/eduskuntavaalien-2015-ylen-vaalikoneen-vastaukset-ja-ehdokkaiden-taustatiedot (13MB)

https://www.avoindata.fi/data/fi/dataset/helsingin-vapaa-aikakyselyn-vastaukset (2,4MB)

https://www.avoindata.fi/data/fi/dataset/turun-kaupungin-turvallisuuskysely (1,3MB)

[VeriHeppi]

This for larger datasets: https://www.kaggle.com/datasets