[website] GluonNLP Website Rework

[eric-haibin-lin]

Goal: better feature/model discoverability for the GluonNLP website

Side Navigation Bar

• Installation
• Models
• Tutorials
• Demos (in the future)
• API Documentations
• Community

Tutorials

Table Of Content:

• Datasets and Vocabularies
  • Data Loading APIs (was http://gluon-nlp.mxnet.io/api/notes/data_api.html)
  • Vocabulary APIs (was https://gluon-nlp.mxnet.io/api/notes/vocab_emb.html)
• Representation Learning
  • Using Pre-trained Word Embedding
  • Word Embedding Training and Evaluation
  • Extracting Sentence Features with Pre-trained ELMo
  • Fine-tuning Pre-trained BERT Model
• Language Modeling
  • Using Pre-trained Language Model (2nd part of https://gluon-nlp.mxnet.io/examples/language_model/language_model.html)
  • Train your own LSTM based Language Model (1st part of https://gluon-nlp.mxnet.io/examples/language_model/language_model.html)
• Machine Translation
  • Training GNMT on IWSLT 2015 Dataset (was: https://gluon-nlp.mxnet.io/examples/machine_translation/gnmt.html)
  • Using Pre-trained Transformer (was: https://gluon-nlp.mxnet.io/examples/machine_translation/transformer.html)
• Sentiment Analysis
  • Fine-tuning LSTM-based Language Model (was https://gluon-nlp.mxnet.io/examples/sentiment_analysis/sentiment_analysis.html)
  • Training Structured Self-attentive Sentence Embedding (was: https://gluon-nlp.mxnet.io/examples/sentence_embedding/self_attentive_sentence_embedding.html)
• Question Answering (to be added later)
  • Inference with Pre-trained BERT-QA (from AMLC notebook (https://github.com/eric-haibin-lin/AMLC19-GluonNLP/), to be added later)
• Text Generation
  • Inference with Beam Search Sampler and Sequence Sampler
• Deployment (in the future)
  • Export Models to JSON for Inference (in the future)
  • Inference with Quantized Models (in the future)
  • Inference with TVM (in the future)

Main changes for tutorials

• Tutorials are grouped by topics.
• In general, in each topic we should cover both model training and inference with pre-trained models.
• Moved data loading and embedding APIs notes to tutorials

Model Zoo (involves splitting/moving scripts, to be worked on later)

Table Of Content:

• Representation Learning
  • Word Embedding
  • BERT (Pre-training)
• Language Modeling: AWD-RNN, Cache LM, GPT-2, Transformer-XL
• Machine Translation: GNMT, Transformer
• Sentiment Analysis and Text Classification
  • TextCNN from sentiment analysis (http://gluon-nlp.mxnet.io/model_zoo/sentiment_analysis/index.html)
  • FastText from text classification (http://gluon-nlp.mxnet.io/model_zoo/text_classification/index.html)
• Question Answering: BERT, QANet (in the future), BiDAF (in the future)
• Natural Language Inference: Decomposable Attention, BERT
• Text Generation: AWD-RNN, GPT-2
• Named Entity Recognition: BERT
• Intent Classification and Slot Labeling: BERT
• Dependency Parsing: Biaffine
• Model Gallery: Links to all trained models

Main changes:

• We group models by tasks.
  • sentiment analysis (http://gluon-nlp.mxnet.io/model_zoo/sentiment_analysis/index.html) and text classification (http://gluon-nlp.mxnet.io/model_zoo/text_classification/index.html) are merged to one category
• If a task involves lots of model architectures and becomes too heavy, we break it down into two
  • For example, if we train both GNMT and Transformer on 20+ languages, we’ll create 2 separate pages under the machine translation task, to avoid over-loaded contents.
  • BERT is split into several tasks: Representational learning, NER, IC/SL, NLI and QA
• Apart from listing pre-trained models in the task-specific page (e.g. showing GPT-2 in Language Model), we add a model gallery which indexes all available pre-trained models. In general, if a model takes more than 30 minutes to train, including the model in model gallery is encouraged to save users’ time.
• Each page should include the link for the training script, command for training and reported accuracy. Reference format: https://gluon-cv.mxnet.io/model_zoo/classification.html#vgg

Community

• Contribution Guideline (the first half of https://gluon-nlp.mxnet.io/community/contribute.html)
• Git HowTos (was: https://gluon-nlp.mxnet.io/community/contribute.html#git-workflow-howtos)
• Release Checklist
• User Gallery (see https://spacy.io/universe and https://allennlp.org/gallery, we should add icons for github and paper links)
  • Dynamic Key-Value Memory Networks for Knowledge Tracing (http://papers.www2017.com.au.s3-website-ap-southeast-2.amazonaws.com/proceedings/p765.pdf). https://github.com/jennyzhang0215/DKVMN
  • STAR-GCN: Stacked and Reconstructed Graph Convolutional Networks for Recommender Systems https://github.com/jennyzhang0215/STAR-GCN (https://github.com/jennyzhang0215/STAR-GCN)
  • https://arxiv.org/pdf/1906.04287.pdf Chinese Embedding via Stroke and Glyph Information:A Dual-channel View
  • https://arxiv.org/pdf/1905.09899.pdf Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning
  • Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual https://arxiv.org/pdf/1908.10763.pdf
  • Non-Parallel Text Style Transfer via Iterative Matching and Translation (http://zhijing-jin.com/files/papers/Iterative.pdf) http://zhijing-jin.com/files/papers/Iterative.pdf
  • SELF-ATTENTION NETWORKS FOR CONNECTIONIST TEMPORALCLASSIFICATION IN SPEECH RECOGNITION https://arxiv.org/pdf/1901.10055.pdf (https://arxiv.org/pdf/1901.10055.pdf)
  • Contextual Phonetic Pretraining for End-to-endUtterance-level Language and Speaker Recognition https://arxiv.org/pdf/1907.00457.pdf
  • https://github.com/imgarylai/bert-embedding Token level embeddings from BERT model on mxnet and gluonnlp
  • https://github.com/elitcloud/elit Emory Language Information Toolkit or Evolution of Language and Information Technology
  • Adversarial Representation Learning for Text-to-Image Matching https://arxiv.org/pdf/1908.10534.pdf

installation

• Better installation experience with selection boxes: https://gluon-cv.mxnet.io/install.html

[eric-haibin-lin]

Other websites for reference:

• https://spacy.io/
• https://allennlp.org
• https://github.com/pytorch/fairseq
• https://huggingface.co/transformers/

[eric-haibin-lin]

@StrayBird-ATSH

[chenw23]

I will follow up with this issue soon.

[sxjscience]

I think we can revise our documentation page and utilize colab, e.g.,

http://d2l.ai/chapter_recurrent-modern/lstm.html and https://colab.research.google.com/github/d2l-ai/d2l-en-colab/blob/master/chapter_recurrent-modern/lstm.ipynb

[eric-haibin-lin]

I had discussion with @astonzhang about colab. Ideally we should have notebooks on both colab and SageMaker.

[sxjscience]

Yes, we will create runnable notebooks on either colab or sagemaker.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Haibin Lin notifications@github.com Sent: Thursday, December 26, 2019 11:26:56 PM To: dmlc/gluon-nlp gluon-nlp@noreply.github.com Cc: Xingjian SHI xshiab@connect.ust.hk; Comment comment@noreply.github.com Subject: Re: [dmlc/gluon-nlp] [website] GluonNLP Website Rework (#1081)

I had discussion with @astonzhanghttps://github.com/astonzhang about colab. Ideally we should have notebooks on both colab and SageMaker.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/dmlc/gluon-nlp/issues/1081?email_source=notifications&email_token=ABHQH3WBURD3SV2F2VOZAW3Q2WU4BA5CNFSM4J7D6GW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHWXMJA#issuecomment-569210404, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHQH3XK2OP5TZV2QDABJNDQ2WU4BANCNFSM4J7D6GWQ.

[eric-haibin-lin]

I opened a separate github issue for colab/sagemaker integration. https://github.com/dmlc/gluon-nlp/issues/1089

[eric-haibin-lin]

Considering that we will re-implement a few models using numpy, let's focus on re-organizing the following pages first:

Tutorial:

1. create hierarchies/categories for the tutorials. Specifically, first level is always a category. Inside each category, we have one or more related tutorials.
2. reorganize existing tutorials/api notes on the website into categories. For the ones from other repositories, we skip them in the first pass.

Community:

1. add the user gallery mentioned above

For model zoo, it involves restructure the script folders. I'll take a look at the complexity of the scripts and comment later.

[sxjscience]

@eric-haibin-lin I also think that we can use markdowns for basic documentations under scripts: https://github.com/dmlc/gluon-nlp/tree/master/scripts, which is similar to the other packages like fairseq and allennlp. In that case the user can preview the content in the folder without clicking index.rst.

[sxjscience]

@eric-haibin-lin In the numpy version, we will improve the tutorial about tokenization in the Prepare Datasets and Vocabularies section.

[chenw23]

Since this rework involves a huge amount of work, I will decompose this issue into several steps to achieve its goal. Currently the first pull request is merged with partial goal of this rework completed. In pull request #1108 , I modified the toc structure of the examples part of the website. This refactor enables this part of the website to display multiple level of menus, which is essential for future enhancements. In this pull request, I did not change the contents. In the future pull requests, I will continue to work on the detailed contents update and enhancement that are mentioned in this issue.