Annif: train the model

[hortongn]

• [ ] Download ETDs with metadata from the ETD Center and see how well they're described.
• [ ] Document the training process including how to pre-process.
• [ ] Download papers and metadata from our vendor databases. Determine feasibility of using as training sets.

Create a "machine" that will suggest basic metadata from a document.

annif train tfidf-en /path/to/Annif-corpora/training/yso-finna-en.tsv.gz

[scherztc]

https://github.com/NatLibFi/Annif-corpora https://github.com/NatLibFi/Annif-tutorial/blob/master/exercises/OPT_custom_corpus.md https://github.com/NatLibFi/Annif-tutorial/tree/master/data-sets/yso-nlf/docs

[scherztc]

https://github.com/NatLibFi/Annif-tutorial/tree/master/data-sets

This directory holds two example data sets. The tutorial exercises may be completed using either data set.

yso-nlf is a data set consisting of the trilingual General Finnish Ontology YSO, a training data set constructed from metadata records from the Finna.fi discovery service, and some 2000 English language Master's and doctoral theses from the University of Jyväskylä.

stw-zbw contains the STW thesaurus for economics, metadata used in the ZBW retrieval system EconBiz and full texts of working papers in economics uploaded to EconStor.

[crowesn]

Download source files here: annif-docker.zip

Annif in Docker Test

• Unzip file to /tmp on your local machine (or wherever)
• Start annif in docker, bind to local project directory, in this case /tmp. docker run -v /tmp/annif-docker:/annif-projects -u $(id -u):$(id -g) -it quay.io/natlibfi/annif bash
• load the vocabulary tsv $ annif load-vocab uc-vocab ./uc-vocab.tsv --language en
• train on dataset $ annif train uc-en training-data/ -- no wildcard, just point to directory

Notes

• Needed to manually remove newlines from all of the .txt files
• Vocab needed 'http' protocol prepended to all urls in uc-vocab.tsv and .tsv for each work.

[crowesn]

In Annif repo, you can start the web ui with docker compose and point to your project: ANNIF_PROJECTS=/tmp/annif-docker MY_UID=$(id -u) MY_GID=$(id -g) docker-compose up