search

guyfe / LongSumm

LongSumm - Scientific Document Summarization Task
74 stars 11 forks source link

readme

LongSumm

A Shared Task at COLINNG 2022 that focuses on generation of long summaries for scientific documents. LongSumm shared task conducted as part of: 3rd Workshop on Scholarly Document Processing.

This is the thrid year of this task, previous reports can be found at 2020 and 2021.

Important announcement (June 30, 2022) :

Evaluation period has started, see Submission Instructions.

Important announcements (May 4, 2022) :

LongSumm - Overview

Most of the work on scientific document summarization focuses on generating relatively short summaries. Such a length constraint might be appropriate when summarizing news articles but it is less adequate for scientific work. In fact, such a short summary resembles an abstract and cannot cover all the salient information conveyed in a given scientific text. Writing longer summaries requires expertise and a deep understanding in a scientific domain, as can be found in some researchers blogs.

To address this point, the LongSumm task opted to leverage blog posts created by researchers in the NLP and Machine learning communities that summarize scientific articles and use these posts as reference summaries.

The corpus for this task includes a training set that consists of 1705 extractive summaries, and 531 abstractive summaries of NLP and Machine Learning scientific papers. The extractive summaries are based on video talks from associated conferences (Lev et al. 2019 TalkSumm) while the abstractive summaries are blog posts created by NLP and ML researchers. In addition, we created a test set of abstractive summaries for testing submissions. Each submission is judged against one reference summary (gold summary) using ROUGE and should not exceed 600 words.

If you use this dataset in your work, please cite our paper:

@inproceedings{chandrasekaran-etal-2020-overview-insights,
    title = "Overview and Insights from the Shared Tasks at Scholarly Document Processing 2020: {CL}-{S}ci{S}umm, {L}ay{S}umm and {L}ong{S}umm",
    author = "Chandrasekaran, Muthu Kumar  and
      Feigenblat, Guy  and
      Hovy, Eduard  and
      Ravichander, Abhilasha  and
      Shmueli-Scheuer, Michal  and
      de Waard, Anita",
    booktitle = "Proceedings of the First Workshop on Scholarly Document Processing",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.sdp-1.24",
    doi = "10.18653/v1/2020.sdp-1.24",
    pages = "214--224",
}

LongSumm - Data and Instructions

You are invited to participate in the LongSumm Shared Task at SDP@COLINNG 2022. This repository contains the dataset and instructions on how to participate in the task.

Training Data

The training data is composed of abstractive and extractive summaries.

Abstractive Summaries:

The abstractive summaries are from different domains of CS including ML, NLP, AI, vision, storage, etc.

The training data contains around 700 abstractive summaries that can be found at data/abstractive/cluster. The folder contains clusters of summaries with length varying between 100-1500 words. Each sub-folder clusters into bins of size 100 words. (i.e., summary of 541 words will appear in the corresponding cluster of 500-600). We used the Python NLTK library to count the number of words and to segment summary text into sentences.

The format of a summary is a JSON file with the following entries: Entry Description
id Record id (unique)
blog_id The id of the blog
summary An array of the sentences of the summary
author_id The id of the author
pdf_url The link to the original paper
author_full_name The author full name
source_website the website in which the original blog appears

Example: 

{
  "id": "79792577",
  "blog_id": "4d803bc021f579d4aa3b24cec5b994",
  "summary": [
    "Task of translating natural language queries into regular expressions ...",
    "Proposes a methodology for collecting a large corpus of regular expressions ...",
    "Reports performance gain of 19.6% over state-of-the-art models.",
    "Architecture  LSTM based sequence to sequence neural network (with attention) Six layers ...",
    "Attention over encoder layer.",
    "...."
  ],
  "author_id": "shugan",
  "pdf_url": "http://arxiv.org/pdf/1608.03000v1",
  "author_full_name": "Shagun Sodhani",
  "source_website": "https://github.com/shagunsodhani/papers-I-read"
}

Each papers' summary should be linked the corresponding text of the original paper. Due to copyright restrictions will not publish the original papers, here are the suggested steps to fully construct the dataset:

Extractive Summaries

The extractive summaries are based on the TalkSumm (Lev et al. 2019) dataset. The dataset contains 1705 automatically-generated noisy extractive summaries of scientific papers from the NLP and Machine Learning domain based on video talks from associated conferences (e.g., ACL, NAACL, ICML)  Summaries can be found under data/extractive/. Each summary provides the top-30 sentences, which are on average around 990 words.  The format of each summary file is as follows:

If you wish to create extractive summaries of a paper that doesn't not exist in the dataset, you will need to follow the instructions from: https://github.com/levguy/talksumm

Test Data (Blind)

There are 22 papers for the test set, as listed below.

Paper id Paper title Paper link
1000 Unsupervised Part-of-Speech Tagging with Bilingual Graph-Based Projections https://www.aclweb.org/anthology/P11-1061.pdf
1001 RNN Fisher Vectors for Action Recognition and Image Annotation https://arxiv.org/pdf/1512.03958.pdf
1002 TALK SUMM: A Dataset and Scalable Annotation Method for Scientific Paper Summarization Based on Conference Talks https://arxiv.org/pdf/1906.01351.pdf
1003 Emotion Detection from Text via Ensemble Classification Using Word Embeddings https://dl.acm.org/doi/pdf/10.1145/3121050.3121093
1004 Classifying Emotions in Customer Support Dialogues in Social Media https://www.aclweb.org/anthology/W16-3609.pdf
1005 MetAdapt: Meta-Learned Task-Adaptive Architecture for Few-Shot Classification https://arxiv.org/pdf/1912.00412.pdf
1006 Detecting Egregious Conversations between Customers and Virtual Agents https://www.aclweb.org/anthology/N18-1163.pdf
1007 Understanding Convolutional Neural Networks for Text Classification https://www.aclweb.org/anthology/W18-5408.pdf
1008 An Editorial Network for Enhanced Document Summarization https://www.aclweb.org/anthology/D19-5407.pdf
1009 DIMSIM: An Accurate Chinese Phonetic Similarity Algorithm Based on Learned High Dimensional Encoding https://www.aclweb.org/anthology/K18-1043.pdf
1010 Improved Neural Relation Detection for Knowledge Base Question Answering https://www.aclweb.org/anthology/P17-1053.pdf
1011 Interactive Dictionary Expansion using Neural Language Models http://ceur-ws.org/Vol-2169/paper-02.pdf
1012 Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts https://papers.nips.cc/paper/6787-interpretable-and-globally-optimal-prediction-for-textual-grounding-using-image-concepts.pdf
1013 Learning Implicit Generative Models by Matching Perceptual Features https://arxiv.org/pdf/1904.02762.pdf
1014 Scalable Demand-Aware Recommendation http://papers.nips.cc/paper/6835-scalable-demand-aware-recommendation.pdf
1015 Neural Response Generation for Customer Service based on Personality Traits https://www.aclweb.org/anthology/W17-3541.pdf
1016 A Low Power, High Throughput, Fully Event-Based Stereo System https://openaccess.thecvf.com/content_cvpr_2018/CameraReady/3791.pdf
1017 Characterization and Learning of Causal Graphs with Latent Variables from Soft Interventions https://papers.nips.cc/paper/9581-characterization-and-learning-of-causal-graphs-with-latent-variables-from-soft-interventions.pdf
1018 Complex Program Induction for Querying Knowledge Bases in the Absence of Gold Programs https://www.aclweb.org/anthology/Q19-1012.pdf
1019 Unsupervised Dual-Cascade Learning with Pseudo-Feedback Distillation for Query-based Extractive Summarization https://arxiv.org/pdf/1811.00436
1020 High quality, lightweight and adaptable TTS using LPCNet https://arxiv.org/pdf/1905.00590
1021 Sobolev Independence Criterion https://arxiv.org/pdf/1910.14212.pdf

Evaluation

The intrinsic evaluation will be done by ROUGE, using ROUGE-1, -2, -L metrics. The average of the ROUGE-F scores obtained against the multiple summaries would be used for final ranking. In addition, a randomly selected subset of the summaries will undergo human evaluation.

Submission Instructions

We will use Codalab to evaluate submissions against the hidden test set.

Please follow the below instructions to evaluate and report your team results:

  1. Create a Codalab account
  2. In the "User Settings" pane, and under "Competition settings", set "Team name" to the name you are using for the shared task (this name will appear in the leaderboard)
  3. Create testing.json file with your system generated summaries on the test set. The submission should be a single json file containing all generated test set summaries. The testing.json file should have the following format: Submission Format
  4. Compress the testing.json file into testing.zip file
  5. Login to Codalab, select the competition: https://codalab.lisn.upsaclay.fr/competitions/5693
  6. Select the Participate tab--> Submit / View Results. Select the Submit button and choose your local testing.zip file (from step 4). The table below the Submit button will show the status of your submission.
  7. Once the submission is uploaded and evaluated against the hidden test set the status will change to Finished. You can choose to report your results to the leaderboard or to download the scores to a text file by selecting the Download output from scoring step option.

Make sure to report the highest obtained score to the leaderboard before the evaluation period ends

Submission Format

The submission should be a single json file containing all summaries, following the format:

{
"paper_id_1":"summary of paper 1",
"paper_id_2":"summary of the paper 2"
}

Evaluation Script

Evaluation script

Leaderboard

We will use Codalab leaderboard to evaluate the quality of the submissions. We will use the following evaluation script: https://github.com/guyfe/LongSumm/blob/master/scripts/evaluation_script.py

Submission instructions will be updated soon.

LongSumm Leaderboard from previous years

Previous years leaderboard

Previous Years system Reports

LongSumm 2021 - https://aclanthology.org/volumes/2021.sdp-1/

LongSumm 2020 - https://aclanthology.org/volumes/2020.sdp-1/

Rules

Timeline

Submission Disclaimer

You should only submit summaries that are part of the test data. Please do not submit any confidential or personal information. Please see the IBM Terms of use (https://www.ibm.com/legal)

Credits

We would like to thank the following blog authors and to ShortScience.org who genereously allowed us to share the content as part of this dataset.

License

Disclaimer

The data was copied from the above mentioned blogs as-is. IBM is not responsible for the content of the data, nor for any claim related to the data (including claims related to alleged intellectual property or privacy breach).

Contacts

For further information about this dataset please contact the organizers of the shared task:



IBMResearch       Piiano logo