Computing rouge is significantly slow

vikigenius commented 2 years ago

Checklist

[x] I have verified that the issue exists against the main branch of AllenNLP.
[x] I have read the relevant section in the contribution guide on reporting bugs.
[x] I have checked the issues list for similar or identical bug reports.
[x] I have checked the pull requests list for existing proposed fixes.
[x] I have checked the CHANGELOG and the commit log to find out if the bug was already fixed in the main branch.
[x] I have included in the "Description" section below a traceback from any exceptions related to this bug.
[x] I have included in the "Related issues or possible duplicates" section beloew all related issues and possible duplicate issues (If there are none, check this box anyway).
[x] I have included in the "Environment" section below the name of the operating system and Python version that I was using when I discovered this bug.
[x] I have included in the "Environment" section below the output of pip freeze.
[x] I have included in the "Steps to reproduce" section below a minimally reproducible example.

Description

The rouge metric computation used in the metric is extremely slow. I am testing this using the Bart model.

Particularly the call here: https://github.com/allenai/allennlp-models/blob/3e3b3ecf8531d8c4d900fdf616926426b401b9ee/allennlp_models/generation/models/bart.py#L260

Related issues or possible duplicates

None

Environment

OS: Linux

Python version: 3.9

Output of pip freeze:

``` absl-py==1.2.0 aiohttp @ file:///home/void/.cache/pypoetry/artifacts/c3/d1/97/85daa5493c9bf3eb25c509bc3eb629985f82b89336bfcb5adcb76b78fe/aiohttp-3.8.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl aiosignal @ file:///home/void/.cache/pypoetry/artifacts/d3/d7/92/f7aa28b137b4b00e62270f58330cedde299917a71a3c00ee64913aa3c2/aiosignal-1.2.0-py3-none-any.whl alabaster @ file:///home/void/.cache/pypoetry/artifacts/48/57/b7/0e44080d4c6f80ece83e427f90368b376fdec42759b60c2c37a04da23b/alabaster-0.7.12-py2.py3-none-any.whl allennlp @ file:///home/void/.cache/pypoetry/artifacts/14/4f/13/65c796d48a7ad702aa08a9b49c5535ad1b20033bb2c1b6928322c552dd/allennlp-2.9.2-py3-none-any.whl allennlp-models @ file:///home/void/.cache/pypoetry/artifacts/47/88/8b/4a3be6913f179f5ff20fa64ea5ec9878fe3b0689b1c63b1e63896f3eee/allennlp_models-2.9.0-py3-none-any.whl arger @ file:///home/void/.cache/pypoetry/artifacts/be/a9/97/92eedbef45f7661c903a6a82517d2d19b7b808717bee4ca85c941531b0/arger-1.4.2-py3-none-any.whl astor @ file:///home/void/.cache/pypoetry/artifacts/1f/03/36/982d1222edac5e8cb8ac6e0464249747fa800d4fb04728a99153ecfe4d/astor-0.8.1-py2.py3-none-any.whl async-timeout @ file:///home/void/.cache/pypoetry/artifacts/62/ff/30/bd0ec891880d93bf40cf6e7c53ab08dd9eb6af45fa532c030731e6eed0/async_timeout-4.0.2-py3-none-any.whl attrs @ file:///home/void/.cache/pypoetry/artifacts/5b/25/94/5d165cd3190cb7dfdc98de86c28e8131fc6e19d83a504412664abffec0/attrs-21.4.0-py2.py3-none-any.whl autorepr @ file:///home/void/.cache/pypoetry/artifacts/79/b0/2c/559adfad5f45b74ef658fdab7bff52f955339a62d1aeae798f390f88d9/autorepr-0.3.0-py2.py3-none-any.whl Babel @ file:///home/void/.cache/pypoetry/artifacts/1e/28/b6/110e8fc8ccc0dafe21bbc521139d0d0674574f7daa6f6f90137bdb4759/Babel-2.9.1-py2.py3-none-any.whl bandit @ file:///home/void/.cache/pypoetry/artifacts/fa/13/c2/a232fa22fd0f429ae8209f7cbe0ad0e0a1a2f97bb3f99d384b58645316/bandit-1.7.2-py3-none-any.whl base58 @ file:///home/void/.cache/pypoetry/artifacts/2e/53/33/a15f42f485704caf86395e9701bb3440206ec9bffc062d7a597f43518e/base58-2.1.1-py3-none-any.whl blis @ file:///home/void/.cache/pypoetry/artifacts/12/c4/39/2501d7749523f2dc3dd58b05c65b7a798a500443434e86cda77c140018/blis-0.7.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl boto3 @ file:///home/void/.cache/pypoetry/artifacts/d9/a2/a4/da57f98666cc512046665f06c99f53f8da4601ec7c2ce17beccf77b7f7/boto3-1.21.32-py3-none-any.whl botocore @ file:///home/void/.cache/pypoetry/artifacts/34/6a/fa/87cdb7201212caae3ac92915f97d2325964115043d1294f43b1596ba49/botocore-1.24.32-py3-none-any.whl cached-path @ file:///home/void/.cache/pypoetry/artifacts/5d/51/03/21b1b3459165f5c7b4d2f598288cc08342351bf505742c829444521529/cached_path-1.1.1-py3-none-any.whl cachetools @ file:///home/void/.cache/pypoetry/artifacts/8a/c3/ee/4c03d5398d7858010aa0573d8e7ef7d1ff2a39ee05b48344ca931bb3a4/cachetools-5.0.0-py3-none-any.whl cachy @ file:///home/void/.cache/pypoetry/artifacts/c0/ce/2b/be65c61ed593659749cff39804b346369e358f14f09151abe591f0c5ab/cachy-0.3.0-py2.py3-none-any.whl catalogue @ file:///home/void/.cache/pypoetry/artifacts/d0/e5/e8/780e087ffacede01d2f7e1980f3186ca0bfc77132a3ad172b84461810c/catalogue-2.0.7-py3-none-any.whl certifi @ file:///home/void/.cache/pypoetry/artifacts/71/9a/ba/a51b34ce9aacf9ac5dbb90d7c7335877522ee188189d9a521ee1a9c411/certifi-2021.10.8-py2.py3-none-any.whl charset-normalizer @ file:///home/void/.cache/pypoetry/artifacts/2b/25/ee/fa64b49cb44c9c392acf26e69ee8314d3b386932de86a9945f4ad2e633/charset_normalizer-2.0.12-py3-none-any.whl click @ file:///home/void/.cache/pypoetry/artifacts/c1/84/b3/e7705da6adc67a18c1140d92bf8b4477ee2308a09e5eabd7fedba9bec6/click-8.1.2-py3-none-any.whl ConfigUpdater @ file:///home/void/.cache/pypoetry/artifacts/40/87/c8/f412301c7171b20fe38c61640da4e7ace132fe8ae24cf0850f49992eb5/ConfigUpdater-3.1-py2.py3-none-any.whl conllu @ file:///home/void/.cache/pypoetry/artifacts/db/ca/f6/bbbd41f4db6714a57bd002c4edfa9d92819e40c52171de18acb5eb4b8f/conllu-4.4.1-py2.py3-none-any.whl coverage @ file:///home/void/.cache/pypoetry/artifacts/67/c6/b5/6278ee4c01ddec492ba84935b44db1898a08d44567af67faf8787802ca/coverage-6.3.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl cymem @ file:///home/void/.cache/pypoetry/artifacts/f1/e5/0d/418ca076950a2a7841dd4199fe239ce71934f6eed6448b117560c8b594/cymem-2.0.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl darglint @ file:///home/void/.cache/pypoetry/artifacts/2d/47/f9/e4df12fe8185cdb64f2f77c23d349693941e09d8d5374c68454fb12e4f/darglint-1.8.1-py3-none-any.whl datasets @ file:///home/void/.cache/pypoetry/artifacts/a5/d8/a3/8f9ef48effda4e8665e0f8e31b35fd8e5c55084ea188a49d3002240354/datasets-2.0.0-py3-none-any.whl dictdiffer @ file:///home/void/.cache/pypoetry/artifacts/bc/85/a4/4043bd24ee0498bfa674ee88c3b151fb29f0946a2b17b0f3e5f02c906b/dictdiffer-0.9.0-py2.py3-none-any.whl dill @ file:///home/void/.cache/pypoetry/artifacts/26/56/9e/73963d2285e6c700801f185e8c1d28f1f971c09aaa411cec9b799a5fca/dill-0.3.4-py2.py3-none-any.whl doc8 @ file:///home/void/.cache/pypoetry/artifacts/02/8c/ea/ee72e46c18bb92524f64422cb67e575cca213c43a4c48411b1bac48dd3/doc8-0.10.1-py3-none-any.whl docker-pycreds @ file:///home/void/.cache/pypoetry/artifacts/9b/0b/be/891931da9caf5e55102337a635d3a7eeeb92c93b4bd39c24d0810f1f25/docker_pycreds-0.4.0-py2.py3-none-any.whl docutils @ file:///home/void/.cache/pypoetry/artifacts/43/28/92/79000933ad30371dc938d9b368a9000e20ac0bb467a716c19ef1fbd3c7/docutils-0.17.1-py2.py3-none-any.whl dparse @ file:///home/void/.cache/pypoetry/artifacts/0e/d3/3f/27b08502c8c8da888ca1b2701c4f1d91b6777be03197cc2566b65eb3dd/dparse-0.5.1-py3-none-any.whl dpath @ file:///home/void/.cache/pypoetry/artifacts/7c/ba/dd/0b2e0c72068f1f80e3050cbc8ccb9c336a39faea2a4b0659be32e74e21/dpath-2.0.6-py3-none-any.whl enum-compat @ file:///home/void/.cache/pypoetry/artifacts/67/87/53/4895e23520ad7f668ab9a1dacbaf6aabe5716c52f8bde75a5c4219ce20/enum_compat-0.0.3-py3-none-any.whl eradicate @ file:///home/void/.cache/pypoetry/artifacts/12/ce/ac/197035fe6d51568abb7ea160f5ad416d2164a2010005e8356b8229e550/eradicate-2.0.0.tar.gz fairscale @ file:///home/void/.cache/pypoetry/artifacts/e8/28/f6/20d1265c725b0b2b366bd8997d1875bbe144b263a8dd3269f7338f8bc4/fairscale-0.4.6.tar.gz filelock @ file:///home/void/.cache/pypoetry/artifacts/72/44/bd/cade636c68ae8857ae195402d49def82f65a669e0fcf4e2dbd51ee6e7d/filelock-3.6.0-py3-none-any.whl flake8 @ file:///home/void/.cache/pypoetry/artifacts/99/5c/01/a49c075bc6dead5c60588c1cd1701f63d614740792c9a09fb133b41d58/flake8-4.0.1-py2.py3-none-any.whl flake8-bandit @ file:///home/void/.cache/pypoetry/artifacts/7e/e4/46/e15782d941f9cde39b64ca5b636180f47573f2b2c9315be56b55152f17/flake8_bandit-2.1.2.tar.gz flake8-broken-line @ file:///home/void/.cache/pypoetry/artifacts/a3/9f/36/0224d46b3050800707535f9c03fee2b60fcc1b48f3c4b419d2cd1bcbd0/flake8_broken_line-0.4.0-py3-none-any.whl flake8-bugbear @ file:///home/void/.cache/pypoetry/artifacts/70/6f/8b/dba01c4ea9cacb2c31ffb4cb9f030f9d0f3ae5f864bfc703eed10f37b4/flake8_bugbear-22.3.23-py3-none-any.whl flake8-commas @ file:///home/void/.cache/pypoetry/artifacts/32/06/3d/62f07a797bd4a6d9f15ead0b6306a47bd0650ca9b3e20aa08d4ee2c23d/flake8_commas-2.1.0-py2.py3-none-any.whl flake8-comprehensions @ file:///home/void/.cache/pypoetry/artifacts/8a/f0/bd/c5b42fff0dce6be658153f234d81795c2344f5a2c4f69e1c7e6e9736d8/flake8_comprehensions-3.8.0-py3-none-any.whl flake8-debugger @ file:///home/void/.cache/pypoetry/artifacts/66/04/47/7bef98a8d237eb17cbfbcb803343be1c79e2c0674ceba163717b6c8e1b/flake8_debugger-4.0.0-py3-none-any.whl flake8-docstrings @ file:///home/void/.cache/pypoetry/artifacts/e0/85/e9/6b482a11d48cf26e1170d9f5bf0b044a5a6c9b816ffe70945e90fc3e56/flake8_docstrings-1.6.0-py2.py3-none-any.whl flake8-eradicate @ file:///home/void/.cache/pypoetry/artifacts/94/a4/1b/56cd39944d5d4d2639e80929837e7fc3b2ff432e095fd7a4d528c56090/flake8_eradicate-1.2.0-py3-none-any.whl flake8-isort @ file:///home/void/.cache/pypoetry/artifacts/73/53/0f/7e71fed3a8631dfabd048e16a84f9fb8e15c8c9a381bda2a444bf42961/flake8_isort-4.1.1-py3-none-any.whl flake8-plugin-utils @ file:///home/void/.cache/pypoetry/artifacts/d1/c5/cd/002f81a0e6fe87b5dc718a0f7381e43f0814e72acccd04d84899cb342b/flake8_plugin_utils-1.3.2-py3-none-any.whl flake8-polyfill @ file:///home/void/.cache/pypoetry/artifacts/28/17/cc/952c11cd5ffb2608137557f928dc4f9365b4dbe1e2a6015eeea78583ac/flake8_polyfill-1.0.2-py2.py3-none-any.whl flake8-pytest-style @ file:///home/void/.cache/pypoetry/artifacts/da/07/65/f01e20adbfd9eacf37bea3f2a189a12bd2ad42eb8edad18b487e3c4af1/flake8_pytest_style-1.6.0-py3-none-any.whl flake8-quotes @ file:///home/void/.cache/pypoetry/artifacts/76/04/0d/e3326c63986618bbd2e54c3211274f66a0bfae21c700e3e67ed14640cd/flake8-quotes-3.3.1.tar.gz flake8-rst-docstrings @ file:///home/void/.cache/pypoetry/artifacts/07/75/20/16ffb132e46114df2fbe48fe2f50254b3f8ea6016371f04addc6738f41/flake8_rst_docstrings-0.2.5-py3-none-any.whl flake8-string-format @ file:///home/void/.cache/pypoetry/artifacts/24/89/bb/7ce8e216f8c7289aa8a2ad4c44f30f87af6c7cdaf5d510110d566d66ec/flake8_string_format-0.3.0-py2.py3-none-any.whl flatten-dict @ file:///home/void/.cache/pypoetry/artifacts/26/bb/15/9b7ce0da158ec2a7d5ea6bad65b1411625b78bdc4d4fadfee34f74c5a1/flatten_dict-0.4.2-py2.py3-none-any.whl frozenlist @ file:///home/void/.cache/pypoetry/artifacts/ed/2c/87/e16164d5388ad4c9d0f19a5e24f8e5acf77ecf926aeb714dd7b8587f41/frozenlist-1.3.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl fsspec @ file:///home/void/.cache/pypoetry/artifacts/11/37/96/7d3584b00d04a83cdc1e638c7d71bda5569fdae7c5170930af2cc64b1a/fsspec-2022.3.0-py3-none-any.whl ftfy @ file:///home/void/.cache/pypoetry/artifacts/74/bd/82/f784771e34d48a09065097c79b938950c8618665100a80d571b8927ebe/ftfy-6.1.1-py3-none-any.whl future @ file:///home/void/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz gitdb @ file:///home/void/.cache/pypoetry/artifacts/90/22/b2/13e35fcefe45f6779d7507377070865acf263775bda5c08fff422c1a4d/gitdb-4.0.9-py3-none-any.whl GitPython @ file:///home/void/.cache/pypoetry/artifacts/c0/0a/c6/d3f006e6bd78795a03581a408b47eed8e5839bd35849453d0b65c005d5/GitPython-3.1.27-py3-none-any.whl google-api-core @ file:///home/void/.cache/pypoetry/artifacts/95/61/22/06e515ed29cde25f855277120efc36aed8d8c8b62bca201fb465362a3b/google_api_core-2.7.1-py3-none-any.whl google-auth @ file:///home/void/.cache/pypoetry/artifacts/8f/f9/4c/c7e8674f2d6e0c58b86e733066dda815b8d7bad12f35a6642d23feaaf3/google_auth-2.6.2-py2.py3-none-any.whl google-cloud-core @ file:///home/void/.cache/pypoetry/artifacts/1e/77/ed/a0d1979128eb7fcc25b759d57092dfd958a129b55d7df78167935e7bd2/google_cloud_core-2.2.3-py2.py3-none-any.whl google-cloud-storage @ file:///home/void/.cache/pypoetry/artifacts/d5/b1/01/e11f0598ae945199dc42faf92b242ebb2285f0e1c6fd980ba9f0cf1a70/google_cloud_storage-2.2.1-py2.py3-none-any.whl google-crc32c @ file:///home/void/.cache/pypoetry/artifacts/dc/73/5d/3e90ed7e68039176e9833fa7936e473a5542d1c38f19a3d63e42ef3e0a/google_crc32c-1.3.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl google-resumable-media @ file:///home/void/.cache/pypoetry/artifacts/39/3b/79/c4415ca7180677b2e557ed709639422e61f1b22966b311704abd49fef6/google_resumable_media-2.3.2-py2.py3-none-any.whl googleapis-common-protos @ file:///home/void/.cache/pypoetry/artifacts/77/03/40/c9ccd6f364aadbc110d86bf95cda9a0bb13f5a2e4d9d56cb5882faddc3/googleapis_common_protos-1.56.0-py2.py3-none-any.whl h5py @ file:///home/void/.cache/pypoetry/artifacts/f1/de/3b/b6b056d985b3b06bfcbb42eaa59871c2c47aae1687e406be20e01d1541/h5py-3.6.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl huggingface-hub @ file:///home/void/.cache/pypoetry/artifacts/24/8b/23/60f427cbd51495e2b236d213c7df704b754d21ff15a41f934a1ec82011/huggingface_hub-0.4.0-py3-none-any.whl identify @ file:///home/void/.cache/pypoetry/artifacts/43/96/84/cf181c7ebd84e914f9a0b2aef0b19fd311d3fe841e7fe02a6b481ca1f2/identify-2.4.12-py2.py3-none-any.whl idna @ file:///home/void/.cache/pypoetry/artifacts/d5/04/80/9c17fd3240a37d12ff2ef042b0306aeb1abd2d8b95f150fd60be938352/idna-3.3-py3-none-any.whl imagesize @ file:///home/void/.cache/pypoetry/artifacts/96/5c/68/c505397f41dd445fe3305ed42a0c203fe0f36fee2cbb522ef564d90d71/imagesize-1.3.0-py2.py3-none-any.whl importlib-metadata @ file:///home/void/.cache/pypoetry/artifacts/0e/bf/b8/5aa8328f4872f2f038b15e48abea8dbe20ba8b18cc0309ad17a127019e/importlib_metadata-4.11.3-py3-none-any.whl iniconfig @ file:///home/void/.cache/pypoetry/artifacts/fa/b0/c6/10cfac68c9e6de9d2a1678366ca89fd9292b362c1760dbe758e41691cb/iniconfig-1.1.1-py2.py3-none-any.whl isort @ file:///home/void/.cache/pypoetry/artifacts/9d/c0/d5/617ee6ec6b065b9a3c8c3f7a0b06604f1a47738904535923554e135a7d/isort-5.10.1-py3-none-any.whl jedi @ file:///home/void/.cache/pypoetry/artifacts/c0/16/7b/b208472f00204d5aaeb0895fcde8e681c56c250bdf8d106fa76cdf7b30/jedi-0.18.1-py2.py3-none-any.whl Jinja2 @ file:///home/void/.cache/pypoetry/artifacts/72/17/28/89e0501c998a9d7241d2c068d57e21d274caaa3ea52f338b31ba912c30/Jinja2-3.1.1-py3-none-any.whl jmespath @ file:///home/void/.cache/pypoetry/artifacts/9b/2b/69/20e4f26420d69eb0e0a57f76a4bec6cb3d6747f30e744483165c06fd47/jmespath-1.0.0-py3-none-any.whl joblib @ file:///home/void/.cache/pypoetry/artifacts/a6/ca/34/f3a58b34616787c399095242ae8633fc32e061f8f76debd987dcecb325/joblib-1.1.0-py2.py3-none-any.whl jsonnet @ file:///home/void/.cache/pypoetry/artifacts/06/9f/87/1e3f68863f77661cd42ff5091e9f05b1ca810d715d2a5401c971fedba0/jsonnet-0.18.0.tar.gz langcodes @ file:///home/void/.cache/pypoetry/artifacts/3c/ca/b7/0bd4d35b4be068935982424d5e77c73ead0ef8d16ec4d26e58c2fa7259/langcodes-3.3.0-py3-none-any.whl lmdb @ file:///home/void/.cache/pypoetry/artifacts/88/f7/6f/368763b557f08f461e9011d87b2bda57d49dfb4bd2acb3846ce1693767/lmdb-1.3.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl loguru @ file:///home/void/.cache/pypoetry/artifacts/af/c5/2e/90fa5f415cade87a24f804aa29454a5263fa93efb5f9568878194ebad1/loguru-0.6.0-py3-none-any.whl m2r2 @ file:///home/void/.cache/pypoetry/artifacts/13/22/0c/6840c0a42d0edbca824d90ebfdf23e3514d686ab1df51c393580ff1ca2/m2r2-0.3.2-py3-none-any.whl MarkupSafe @ file:///home/void/.cache/pypoetry/artifacts/f4/6d/d3/a9a915049dea17a65026c9ac187154834bfe6c9174f0dd25e602f28c45/MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl marshmallow @ file:///home/void/.cache/pypoetry/artifacts/b1/a8/f0/dc5ab8b9250d261e30d3dccf2712fdb1abdda469f9a8ed4b26317c411d/marshmallow-3.15.0-py3-none-any.whl marshmallow-polyfield @ file:///home/void/.cache/pypoetry/artifacts/15/54/1a/bf7b323a7e9bef7044335503b044f792374a95673e74fcf13d80690a10/marshmallow_polyfield-5.10-py3-none-any.whl mccabe @ file:///home/void/.cache/pypoetry/artifacts/96/5e/5f/21ae5296697ca7f94de4da6e21d4936d74029c352a35202e4c339a4253/mccabe-0.6.1-py2.py3-none-any.whl mistune @ file:///home/void/.cache/pypoetry/artifacts/33/31/4c/2d69dc65d06d1c8f8b00b8e995e24bae97fce2e1f8ec5d8d2d98e852da/mistune-0.8.4-py2.py3-none-any.whl more-itertools @ file:///home/void/.cache/pypoetry/artifacts/a6/67/cb/fbf4fb4bbfedae39080e351b12ca64287d47c839769eeaf91434f3274c/more_itertools-8.12.0-py3-none-any.whl multidict @ file:///home/void/.cache/pypoetry/artifacts/12/d0/ef/0b7b9499611ab1817c3e84fb92c039a4fb5ac1d0397a3b86f4c5283108/multidict-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl multiprocess @ file:///home/void/.cache/pypoetry/artifacts/5d/fb/ce/029cafd015834319024c43f708d975b006d115de466309cb8f4d8c1353/multiprocess-0.70.12.2-py39-none-any.whl murmurhash @ file:///home/void/.cache/pypoetry/artifacts/c5/62/3e/e9b9045eda9a62bac496552aaab7dca42fa7ed5eb8ad1e2e47334f2e7b/murmurhash-1.0.6-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl mypy @ file:///home/void/.cache/pypoetry/artifacts/f3/2e/4b/a6e385cc026a07a6e7f7ea6e399b95bf05152b89029fde1cbd9dc79777/mypy-0.931-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl mypy-extensions @ file:///home/void/.cache/pypoetry/artifacts/92/45/bf/1807ce854ff668d92602207a37bfa9316def2a3f257bd03c4c5be4bc9b/mypy_extensions-0.4.3-py2.py3-none-any.whl nitpick @ file:///home/void/.cache/pypoetry/artifacts/e2/e5/2d/cc41b378b4b17d511941cc4a8c6726d43ae20ab8d95c811b1ffd4a0366/nitpick-0.31.0-py3-none-any.whl nltk @ file:///home/void/.cache/pypoetry/artifacts/fa/3a/e7/039be37a8319e1d87eb9c6d4091d38c90ff404544574f52fe5fdf6a620/nltk-3.7-py3-none-any.whl numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1651020388495/work packaging @ file:///home/void/.cache/pypoetry/artifacts/47/3f/ce/b240169f7d8bef1ff24a0269b709721ce86543c2ec25e0b6adb2c2d7ac/packaging-21.3-py3-none-any.whl pandas @ file:///home/void/.cache/pypoetry/artifacts/31/04/ef/34f05c58bf930aa5a0a574177ab4757d1cdf3d8e1123ea306dcdb64739/pandas-1.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl parso @ file:///home/void/.cache/pypoetry/artifacts/4e/d3/c5/d91e200388f1ce44b74484eb39fbfd0797eb236741d3672a7d728668b2/parso-0.8.3-py2.py3-none-any.whl pathtools @ file:///home/void/.cache/pypoetry/artifacts/ce/ff/c7/31da76336d55d51d979a50868616c867c7b2ea6f2d2084b8c744726ae7/pathtools-0.1.2.tar.gz pathy @ file:///home/void/.cache/pypoetry/artifacts/50/12/78/e6a2a43271455193874ba977034f2eb9ce0c4993ce86418a8903962049/pathy-0.6.1-py3-none-any.whl pbr @ file:///home/void/.cache/pypoetry/artifacts/84/55/59/eb463369b9b26da119a7363f15941299731f5758e62a5b5d7d6e99b19b/pbr-5.8.1-py2.py3-none-any.whl pep8-naming @ file:///home/void/.cache/pypoetry/artifacts/c5/94/df/bcd60db349c70b5041763ab528cbf6864991142dc2a25426cc75f63068/pep8_naming-0.12.1-py2.py3-none-any.whl Pillow==9.2.0 pluggy @ file:///home/void/.cache/pypoetry/artifacts/81/78/ca/13f743a3628faf5a0b7f021efb45f2193acba3a13663d498f6b34bf02e/pluggy-1.0.0-py2.py3-none-any.whl preshed @ file:///home/void/.cache/pypoetry/artifacts/32/18/f6/d3ef3856138e839656c9506ee150937257b0be46dcefb42b33572b9a01/preshed-3.0.6-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl promise @ file:///home/void/.cache/pypoetry/artifacts/d6/c6/43/95f1e737b1dd79d3a5ac6cfb264a889716bab4cd9d28a9bc8c69591d53/promise-2.3.tar.gz protobuf @ file:///home/void/.cache/pypoetry/artifacts/10/0c/35/601d986797bc33b873a6d389873300c327b39a92a29a18f1af0fba64e2/protobuf-3.20.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl psutil @ file:///home/void/.cache/pypoetry/artifacts/19/ba/e2/d04bccece490beafb95d281e00c58ce8c6031bd5658389b25d0fee7b1f/psutil-5.9.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl py @ file:///home/void/.cache/pypoetry/artifacts/b3/5c/47/ba5a596e01a2b61fa2daa6a438252483ad8c04e6c99e5dc22eaf8a489a/py-1.11.0-py2.py3-none-any.whl py-rouge @ file:///home/void/.cache/pypoetry/artifacts/31/31/4f/cc7585fdf5aec32c5b688726f52c2238f959caba4f6a65950f1a932745/py_rouge-1.1-py3-none-any.whl pyarrow @ file:///home/void/.cache/pypoetry/artifacts/be/47/10/2402910ade8b5d730999bbd4d4d77a81edacf0c3c4c7ff9195a02eb368/pyarrow-7.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl pyasn1 @ file:///home/void/.cache/pypoetry/artifacts/7b/3a/54/42ce43b579bda01b9d79022fb733811594441e7a32e9f9a5a98f672bdc/pyasn1-0.4.8-py2.py3-none-any.whl pyasn1-modules @ file:///home/void/.cache/pypoetry/artifacts/dd/b8/4f/b56433e0354274a31074995e01b8671751e9f0ed0001f5254e5b03a54f/pyasn1_modules-0.2.8-py2.py3-none-any.whl pycodestyle @ file:///home/void/.cache/pypoetry/artifacts/80/25/e5/ced4effd2487693e96521aebb353ce58ad0d81417f71c480419ea8f44a/pycodestyle-2.8.0-py2.py3-none-any.whl pydantic @ file:///home/void/.cache/pypoetry/artifacts/e1/bf/c1/7cdd43a1b6ea4e9cfd1e0e9e5b3276ad83c6b57194b4449da3e64fe6cf/pydantic-1.8.2-cp39-cp39-manylinux2014_x86_64.whl pydocstyle @ file:///home/void/.cache/pypoetry/artifacts/75/e7/e5/1acad15a51efd39cf39259c7888c205fd787a92efea28f7afc5a9e315c/pydocstyle-6.1.1-py3-none-any.whl pyflakes @ file:///home/void/.cache/pypoetry/artifacts/ad/8e/6d/e46117c96aa3e955c40ed2f0d35b29910b0e557e327fbbba17fd8c390c/pyflakes-2.4.0-py2.py3-none-any.whl Pygments @ file:///home/void/.cache/pypoetry/artifacts/eb/89/d1/4faa7ff8e87642649c25d60a15480ea042f1cff3c237ad940149669cbf/Pygments-2.11.2-py3-none-any.whl pyparsing @ file:///home/void/.cache/pypoetry/artifacts/8e/18/e1/a47fbcc66e38a452c34d2a444d74d8a03fd4aa7c0b6e27c0bee1582107/pyparsing-3.0.7-py3-none-any.whl pytest @ file:///home/void/.cache/pypoetry/artifacts/0d/89/89/a248cc870a30ec312f23957c835d3c5d0a9a9b26cc85fab5453ac5a6da/pytest-7.1.1-py3-none-any.whl pytest-cov @ file:///home/void/.cache/pypoetry/artifacts/50/df/c5/4bd35027c6247daac4ba547aff90d97ccdc7daf4aa62e110d69cfd39de/pytest_cov-3.0.0-py3-none-any.whl pytest-randomly @ file:///home/void/.cache/pypoetry/artifacts/99/29/cd/a806a9ce497096d8fe2b4ef24aec6582ce95841657f5f9c574b99acd8f/pytest_randomly-3.11.0-py3-none-any.whl python-dateutil @ file:///home/void/.cache/pypoetry/artifacts/53/f8/2a/7d63ce15df7386e9536e83413453f8aa845b47fb425f05c4ca2fb231c3/python_dateutil-2.8.2-py2.py3-none-any.whl python-lsp-jsonrpc @ file:///home/void/.cache/pypoetry/artifacts/e6/0e/80/b725147ba93341249a4ba1659a62c1e5fdff33d830389c548e06f75065/python_lsp_jsonrpc-1.0.0-py3-none-any.whl python-lsp-server @ file:///home/void/.cache/pypoetry/artifacts/6f/98/ad/d14c9cd61a8edae1c4107c73d2bf27362f447c1ed82b93fbf17ba86c6f/python_lsp_server-1.4.1-py3-none-any.whl python-slugify @ file:///home/void/.cache/pypoetry/artifacts/26/60/f8/a441c9fdba2f1c5f992057fd43b6ccb5302a4886f17f4cde9d43a718f6/python_slugify-6.1.1-py2.py3-none-any.whl pytz @ file:///home/void/.cache/pypoetry/artifacts/fb/60/46/e704d81037c87ab74c6677ae79ab43dda85c31e3ae38f53c44486b593a/pytz-2022.1-py2.py3-none-any.whl PyYAML @ file:///home/void/.cache/pypoetry/artifacts/4e/12/e6/32a4a77023f06c3061d2fc6d4692aa531b8530e211b24ff1f77a39e6ee/PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl regex @ file:///home/void/.cache/pypoetry/artifacts/a0/4f/99/f2ff47e7e38a845ffad9089c618d421234de54d7dcd88287a05c9a6d39/regex-2022.3.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl requests @ file:///home/void/.cache/pypoetry/artifacts/ff/f3/bc/a6781f93c2f9488431db494169bb514a083a1d77f3c325a277d8699398/requests-2.27.1-py2.py3-none-any.whl responses @ file:///home/void/.cache/pypoetry/artifacts/dc/5e/50/a9a865ad978954b88e5eef7eb145cd6df70c17ab0eebcdb06139fdba71/responses-0.18.0-py3-none-any.whl restructuredtext-lint @ file:///home/void/.cache/pypoetry/artifacts/04/32/b5/e5036c2c17882570082f8f5147166579794efaa6ebc2fc7ab5a0aea9cb/restructuredtext_lint-1.4.0.tar.gz rouge-score==0.1.2 rsa @ file:///home/void/.cache/pypoetry/artifacts/29/ba/41/0ee0fcca877c94f32799d12775b513c1314a9712a5c2833dc5bacff2ab/rsa-4.8-py3-none-any.whl ruamel.yaml @ file:///home/void/.cache/pypoetry/artifacts/7c/2c/48/111028a9e95097a5ff404a2dc3c1990c911bec7802cb01366c06a9919d/ruamel.yaml-0.17.21-py3-none-any.whl ruamel.yaml.clib @ file:///home/void/.cache/pypoetry/artifacts/52/95/75/d1a001b469ea6ed66dd6f4194c9f7eb370223816cf6292be2a8213e9a4/ruamel.yaml.clib-0.2.6-cp39-cp39-manylinux1_x86_64.whl s3transfer @ file:///home/void/.cache/pypoetry/artifacts/37/ce/4f/6e340791b443e17e70194b28ffae5c244fefaba8c5ec34fe26803ae32c/s3transfer-0.5.2-py3-none-any.whl sacremoses @ file:///home/void/.cache/pypoetry/artifacts/fe/98/1e/44e689995403ecb78de0d342c0bd6b1ee909c7531b041787cfef263abd/sacremoses-0.0.49-py3-none-any.whl safety @ file:///home/void/.cache/pypoetry/artifacts/8f/cc/5a/1dc0196c65cdb3f015bad143c000d159aca6f838a21d7d10872b092b83/safety-1.10.3-py2.py3-none-any.whl scikit-learn @ file:///home/void/.cache/pypoetry/artifacts/c0/be/ed/effc57daf70ea24f0bf3ee842d9360a8d8b334360d6ede98084313f8ca/scikit_learn-1.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl scipy @ file:///home/void/.cache/pypoetry/artifacts/cf/54/79/6934fe0ee0382e865908197894c6ef46b1339d7fbe28cb76be7eba060c/scipy-1.6.1-cp39-cp39-manylinux1_x86_64.whl sentencepiece @ file:///home/void/.cache/pypoetry/artifacts/82/c3/b5/7dcf4928f5159deaba8f451549bd8b063b13e500873857524fc8f893c9/sentencepiece-0.1.96-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl sentry-sdk @ file:///home/void/.cache/pypoetry/artifacts/53/27/b4/436602342845428610293b503dba983d9445cc8201b396a7027cfdf0e2/sentry_sdk-1.5.8-py2.py3-none-any.whl setproctitle @ file:///home/void/.cache/pypoetry/artifacts/ad/8f/ef/df09cb137ec0407a725040b884910167a9a01e1c291f37466bfff3ed2f/setproctitle-1.2.2-cp39-cp39-manylinux1_x86_64.whl shortuuid @ file:///home/void/.cache/pypoetry/artifacts/1b/c2/26/0a03ab3637895180121fd749a44e5005e71b8300cdd863c20ddbf0d318/shortuuid-1.0.8-py3-none-any.whl six @ file:///home/void/.cache/pypoetry/artifacts/08/9f/47/c16ae03035fc69eaf100ea39657a49baaeef714e25a52575710c34cd48/six-1.16.0-py2.py3-none-any.whl smart-open @ file:///home/void/.cache/pypoetry/artifacts/90/9d/8f/b3121f6940407c06e50f8f81646a84f4551ec214bb239013488c3492e8/smart_open-5.2.1-py3-none-any.whl smmap @ file:///home/void/.cache/pypoetry/artifacts/76/3f/89/377e56f6e08e5b1fa88da762382b4c9c817c6dd24eae2e0e190898511d/smmap-5.0.0-py3-none-any.whl snowballstemmer @ file:///home/void/.cache/pypoetry/artifacts/a4/95/b0/c0f70d4b9bb0bac123e716da53ba9b012071cedf7c99bcf030757530f4/snowballstemmer-2.2.0-py2.py3-none-any.whl sortedcontainers @ file:///home/void/.cache/pypoetry/artifacts/b9/80/e1/4bdfa349488797fd308ecbe48f4fad57a3245777fb47c8741730583262/sortedcontainers-2.4.0-py2.py3-none-any.whl spacy @ file:///home/void/.cache/pypoetry/artifacts/ec/99/e6/5494087874800c47d8e2703077d94c4d68a5e48052742b0ad1a99925d8/spacy-3.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl spacy-legacy @ file:///home/void/.cache/pypoetry/artifacts/e6/1b/06/c503f010541d1e7a9def3bdb2ed43c3f7d7d28a5431d7b62f40bb68d6d/spacy_legacy-3.0.9-py2.py3-none-any.whl spacy-loggers @ file:///home/void/.cache/pypoetry/artifacts/4b/67/5c/cf7286e10b61c767780bbc5ec45dd12980a68d134bdc38f560e05deeb0/spacy_loggers-1.0.2-py3-none-any.whl Sphinx @ file:///home/void/.cache/pypoetry/artifacts/b8/21/89/626dfd895b30a559af2369438bbb53f9b97470130e07400845f5b02e7a/Sphinx-4.5.0-py3-none-any.whl sphinx-autodoc-typehints @ file:///home/void/.cache/pypoetry/artifacts/20/93/18/5a773d9276bc8ce1415caaeebec57d0a552e288507431207cd1dd05ce2/sphinx_autodoc_typehints-1.17.0-py3-none-any.whl sphinxcontrib-applehelp @ file:///home/void/.cache/pypoetry/artifacts/8d/eb/86/eec708bb3ff50c9780e78f36a9cb82cd9ff8030a90bd23b9a6f20aecca/sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl sphinxcontrib-devhelp @ file:///home/void/.cache/pypoetry/artifacts/56/a5/74/11ccaa7737f06a10422027e0595b24d243af7a7a1dc4982dec22044c28/sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl sphinxcontrib-htmlhelp @ file:///home/void/.cache/pypoetry/artifacts/66/d0/cb/7228297c74d9280e7246b52187704724b0b0881e2762cdef34e04be778/sphinxcontrib_htmlhelp-2.0.0-py2.py3-none-any.whl sphinxcontrib-jsmath @ file:///home/void/.cache/pypoetry/artifacts/d2/22/96/2076357e64b369910aa24a20d5b719beb24a1487146e4742476ee1e2d8/sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl sphinxcontrib-qthelp @ file:///home/void/.cache/pypoetry/artifacts/32/fc/a9/112a82396d53ec629c1450253a6ded4d94d7ffffd63acd49879543ece9/sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl sphinxcontrib-serializinghtml @ file:///home/void/.cache/pypoetry/artifacts/e6/9a/17/830e357f3aee36549c613a2d660b5cf38d70c27ecb7c218d15c7bfffe1/sphinxcontrib_serializinghtml-1.1.5-py2.py3-none-any.whl sqgen==0.1.0 srsly @ file:///home/void/.cache/pypoetry/artifacts/89/a3/39/8c42cc259a1c485abc0d750eab45ade4628e27862bbc13e1b55e08fdcc/srsly-2.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl stevedore @ file:///home/void/.cache/pypoetry/artifacts/42/17/61/45c6b9d8b0d45dd19994d1f96aa784b4e9792c968eb24e21b0816a258b/stevedore-3.5.0-py3-none-any.whl tensorboardX @ file:///home/void/.cache/pypoetry/artifacts/92/47/7e/7ec08edf2a5dd1d263fe84464c711bf81fb5ca0750298924e7cd25bfb1/tensorboardX-2.5-py2.py3-none-any.whl termcolor @ file:///home/void/.cache/pypoetry/artifacts/a2/5d/c7/e4ccb3b3bb8d3e3aff995fb6ffb12cfc78bbc8affa283907ee5eb5a5a5/termcolor-1.1.0.tar.gz testfixtures @ file:///home/void/.cache/pypoetry/artifacts/42/0d/ed/2ebf061a3336990c5502f46fcc245856d3b4410e3d4534faf557ff708e/testfixtures-6.18.5-py2.py3-none-any.whl text-unidecode @ file:///home/void/.cache/pypoetry/artifacts/34/f9/c2/484c44b08bab89d472229bbd257fcc1d1c6273ee027f01cb08c4e3c309/text_unidecode-1.3-py2.py3-none-any.whl thinc @ file:///home/void/.cache/pypoetry/artifacts/50/36/77/e54de1f1570d8fe6d5663eedf7dd2a586d09214798e14e0812d3da7a31/thinc-8.0.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl threadpoolctl @ file:///home/void/.cache/pypoetry/artifacts/a1/99/8f/521268b618d08b18c198e13411a00fa56ff22bc2129adfbec84c73bc21/threadpoolctl-3.1.0-py3-none-any.whl tokenizers @ file:///home/void/.cache/pypoetry/artifacts/76/b7/1e/e41606a4aef09b047a3994b5bd08e7a6fe647f5e1dcfaa1acde0c74661/tokenizers-0.12.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl toml @ file:///home/void/.cache/pypoetry/artifacts/6b/6a/c9/53b19f7870a77d855e8b05ecdc98193944e5d246dafe11bbcad850ecba/toml-0.10.2-py2.py3-none-any.whl tomli @ file:///home/void/.cache/pypoetry/artifacts/73/7c/d9/9f2752fc5b05f9176c6f3adc6484be1cec75a68925b8c5fe39d6493a07/tomli-2.0.1-py3-none-any.whl tomlkit @ file:///home/void/.cache/pypoetry/artifacts/94/40/ed/49ed2b2bc80640b90ddc4d96266c9961325337428d29b7f21cd69c3e54/tomlkit-0.10.1-py3-none-any.whl torch==1.10.2 torch-model-archiver @ file:///home/void/.cache/pypoetry/artifacts/40/42/a1/7dbfd7727f05615e362db3ba13d0e16e7d230058f7d61ad64edf24d35e/torch_model_archiver-0.5.3-1-py2.py3-none-any.whl torchvision==0.11.3 tqdm @ file:///home/void/.cache/pypoetry/artifacts/4f/cb/23/ea1b6f00ee018ed18b62a59f236afc58992bea37bed1c52392070e0c20/tqdm-4.63.2-py2.py3-none-any.whl transformers @ file:///home/void/.cache/pypoetry/artifacts/73/70/cc/f69c484ede493bcdc65a51165110ec666c9b103ab8916e9b237380fca5/transformers-4.17.0-py3-none-any.whl typer @ file:///home/void/.cache/pypoetry/artifacts/6a/ab/b8/3c3bc10517987597baf2339967f15315a3b0fe479e307e387c032eb97d/typer-0.4.1-py3-none-any.whl typing_extensions @ file:///home/void/.cache/pypoetry/artifacts/9b/2e/e0/f2f5524348ab0c59b31fc2526543d60a53583608f35a4574a4978f5358/typing_extensions-4.1.1-py3-none-any.whl ujson @ file:///home/void/.cache/pypoetry/artifacts/46/87/2c/1be282cb0f7a001871d3b96036b7b5297671ba93fbfaf0aa196d408579/ujson-5.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl urllib3 @ file:///home/void/.cache/pypoetry/artifacts/86/4a/ad/37e91024d94f8218fc54b1fbb58b71888f53c1636daeb28593fa254ed2/urllib3-1.26.9-py2.py3-none-any.whl wandb @ file:///home/void/.cache/pypoetry/artifacts/87/2d/6e/f52d6900867bd9439ac8c2322abb363e8c2db0ee332596cff4a435fcf3/wandb-0.12.11-py2.py3-none-any.whl wasabi @ file:///home/void/.cache/pypoetry/artifacts/10/b9/ba/03f58e92aec7231ab5ce047f473f003232225cea36167ba6e5ff1d7e36/wasabi-0.9.1-py3-none-any.whl wcwidth @ file:///home/void/.cache/pypoetry/artifacts/7d/f4/60/0737157bb9711fec72c70dff523aa54491eef317e0d586cf5388ff0908/wcwidth-0.2.5-py2.py3-none-any.whl wemake-python-styleguide @ file:///home/void/.cache/pypoetry/artifacts/47/a2/24/991460310cb6782b1c07d99c5499aa92bd5a21148d808ab6be91764993/wemake_python_styleguide-0.16.1-py3-none-any.whl word2number @ file:///home/void/.cache/pypoetry/artifacts/91/7b/91/fd4e6b1580eb2a2f0bb8b725ba137628acb0adb21522a3ff9d69e6f5e1/word2number-1.1.zip xxhash @ file:///home/void/.cache/pypoetry/artifacts/6b/b3/33/e526ffaa46a1fe44c8214380090bf8e2e0d50d76f3d5dfc437b174227c/xxhash-3.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl yarl @ file:///home/void/.cache/pypoetry/artifacts/a3/6b/bb/af40f3ac1dfb8650848efd34d43176c01039c7203c8f8f615543baf9d2/yarl-1.7.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl yaspin @ file:///home/void/.cache/pypoetry/artifacts/08/96/72/a995df36daf20ca713ef9c496ef7cc2595253341a2ed64056cb8ddb82c/yaspin-2.1.0-py3-none-any.whl zipp @ file:///home/void/.cache/pypoetry/artifacts/8a/08/12/1ce534dd356211524f826d50d2a2bfe2e99a8f3cac7355cac56639b06b/zipp-3.8.0-py3-none-any.whl ```

Steps to reproduce

Particularly I am measuring the time it takes to call rouge here like this.

start = timer()            
self._rouge(predictions, target_ids)            
end = timer()            
print('GPU Rouge timing:', end - start)

For a batch size of 64

This takes: GPU Rouge timing: 187.72676797099984 # That's more than 3 minutes.

I noticed that the rouge computation is done on the GPU and maybe that was slowing things down, so I decided to check

start = timer()            
self._rouge(predictions.to('cpu'), target_ids.to('cpu'))            
end = timer()            
print('CPU Rouge timing:', end - start)

This takes: CPU Rouge timing: 64.7368660709999 # Much faster but still very slow

So I went ahead and used the rouge implementations from the dataset library which is basically a wrapper around rouge_score by Google.

from datasets import load_dataset, load_metric
metric = load_metric("rouge")
start = timer()
pred_text = self._indexer._tokenizer.batch_decode(
    predictions.tolist(), skip_special_tokens=True
)
ref_text = self._indexer._tokenizer.batch_decode(
    target_ids.tolist(), skip_special_tokens=True
)
metric.compute(predictions=pred_text, references=ref_text, use_stemmer=True)
end = timer()
print('HFT Rouge timing:', end - start)

This takes:

HFT Rouge timing: 1.1103893849999622 # That's just one second

The rouge implementation used in allennlp here is 180 times slower than huggingface even though it doesn't have to perform the decoding step or perform stemming or compute the RougeLSum score.

I have used this model before, but I don't remember it being this slow. It seems like a recent thing.

Calling rouge for a single batch of size 64 should not be taking 3 minutes no matter what is going on internally.

AkshitaB commented 2 years ago

@vikigenius

I wrote a small test script, but cannot reproduce the latency that you're observing.

import torch
from allennlp.training.metrics import ROUGE

from allennlp.data.token_indexers.pretrained_transformer_indexer import PretrainedTransformerIndexer
token_indexer = PretrainedTransformerIndexer("facebook/bart-base", namespace="tokens")

vocab_size = 50265
batch_size = 64
seq_len = 10

predictions = torch.randint(0, vocab_size, (batch_size, seq_len))
targets = torch.randint(0, vocab_size, (batch_size, seq_len))

metric = ROUGE(exclude_indices={0, 1, 2})
st = time.time()
metric(predictions, targets)
en = time.time()
print(f"allennlp Rouge timing: {en-st} seconds.")

from datasets import load_dataset, load_metric
metric = load_metric("rouge")

pred_text = token_indexer._tokenizer.batch_decode(
    predictions.tolist(), skip_special_tokens=True
)
ref_text = token_indexer._tokenizer.batch_decode(
    targets.tolist(), skip_special_tokens=True
)
start = time.time()
metric.compute(predictions=pred_text, references=ref_text, use_stemmer=True)
end = time.time()
print('HFT Rouge timing:', end - start)

Yields:

allennlp Rouge timing: 0.1593489646911621 seconds.
HFT Rouge timing: 0.2746100425720215

Can you share more details on the setup that you're observing this on? Are you evaluating a specific bart model?

vikigenius commented 2 years ago

@AkshitaB I get similar timings with your inputs

But if you change your input shapes to this

vocab_size = 50265
batch_size = 64
pred_seq_len = 150
tgt_seq_len = 787

predictions = torch.randint(0, vocab_size, (batch_size, pred_seq_len))
targets = torch.randint(0, vocab_size, (batch_size, tgt_seq_len))

I get

allennlp Rouge timing: 105.64900302886963 seconds.
HFT Rouge timing: 3.820307731628418

AkshitaB commented 2 years ago

@vikigenius I can reproduce the above. I'll look into it.