Release v0.3.18 (penelope)

[roger-mahler]

Release that includes a number of new features as bug fixes.

Noteworthy changes are:

• Support for merging consecutive words into phrases (#127, humlab/penelope#57).
• Added PoS-padding/markers that enables true word distances in co-occurrence computation (#133)
• Added optional append of PoS-tag ahead of co-occurrence / word trends computation (humlab/penelope#73)
• The goodness of fit tab is now disabled (improved performance)
• Replaced/simplified Data tab in co-occurrence/word trends explorer
• Improved performance of PoS statistics via caching of extended document index
• Added tagged data caching using Apache Arrow feather protocol (performance improvements)
• Stop words filter is now applied after co-occurrence computation.
• SUC PoS grouping revised (#134)
• Many bug fixes (see https://github.com/humlab/penelope/issues)
• Improved CLI for co-occurrence and word trends
• Merged PR https://github.com/humlab/penelope/pull/61
• Added optional multiprocessing of PoS-data (improved performance)
• Refactorings (adaptations) related to forthcoming Parla-CLARIN data

[roger-mahler]

The release is live on https://westac.humlab.umu.se

[roger-mahler]

@fredrik1984 @Stubbendorff @Gunnarnordahl

[roger-mahler]

Command Line Interface

┌──[16:57:32]─[roger@p2]─[~/.../welfare-state-analytics/workbench_sandbox]
└──>  (poetry) λ co_occurrence --help

Usage: co_occurrence [OPTIONS] CORPUS_CONFIG INPUT_FILENAME OUTPUT_FILENAME

Options:
  -c, --concept TEXT              Concept
  --no-concept                    Filter out concept word
  --count-threshold INTEGER       Filter out co_occurrences below threshold
  -w, --context-width INTEGER     Width of context on either side of concept.
                                  Window size = 2 * context_width + 1

  -m, --phrase TEXT               Phrase
  -n, --phrase-file TEXT          Phrase filename
  -p, --partition-key TEXT        Partition key(s)
  -i, --pos-includes TEXT         List of POS tags to include e.g. "|NN|JJ|".
  -m, --pos-paddings TEXT         List of POS tags to replace with a padding
                                  marker.

  -x, --pos-excludes TEXT         List of POS tags to exclude e.g.
                                  "|MAD|MID|PAD|".

  -b, --lemmatize / --no-lemmatize
                                  Use word baseforms
  -l, --to-lowercase / --no-to-lowercase
                                  Lowercase words
  -r, --remove-stopwords [swedish|english]
                                  Remove stopwords using given language
  --min-word-length INTEGER RANGE
                                  Min length of words to keep
  --max-word-length INTEGER RANGE
                                  Max length of words to keep
  --keep-symbols / --no-keep-symbols
                                  Keep symbols
  --keep-numerals / --no-keep-numerals
                                  Keep numerals
  --only-alphabetic               Keep only tokens having only alphabetic
                                  characters

  --only-any-alphanumeric         Keep tokens with at least one alphanumeric
                                  char

  -f, --force / --no-force        Ignore checkpoints
  --help                          Show this message and exit.

Example call using the CLI

┌─[16:55:58]─[roger@p2]─[~/.../welfare-state-analytics/workbench_sandbox]
└──>  (poetry) λ nohup co_occurrence --pos-includes "NN|PM" --pos-paddings "UO|JJ|AB|HA|IE|IN|PL|PP|KN|SN|VB|PC" --context-width 5 --lemmatize --to-lowercase --partition-key year --remove-stopwords swedish --concept information ./doit.yml  /data/westac/riksdagens-protokoll.1920-2019.sparv4.csv.zip ./output/information_w5_NNPM_UOJJABHAIEINPLPPKNSNVBPC_lemma_no_stops_NEW/information_w5_NNPM_UOJJABHAIEINPLPPKNSNVBPC_lemma_no_stops_NEW &

[roger-mahler]

Välj "riksdagens-protokoll" i bölädderlista för PoS-statistik. Övriga val är inte konfigurerade.