alphagov / govuk-content-metadata

GovNER: an encoder-based language model (RoBERTa) fine-tuned to perform Named Entity Recognition (NER) on GOV.UK content

MIT License

4 stars 1 forks source link

Unschedule bulk inference #102

Closed exfalsoquodlibet closed 1 year ago

exfalsoquodlibet commented 1 year ago

Summary

The purpose of this PR:

update the bulk inference pipeline to use CUDA 11.7 (since pytorch no longer supports CUDA 11.2)
better document the pipeline after it is no longer scheduled to run every two weeks; it has now moved to be a on has been replaced by the daily vertex-ai batch job
streamline setting up of VM instances

Checklists

This pull/merge request meets the following requirements:

[x] code runs
[x] developments are ethical and secure
[x] you have made proportionate checks that the code works correctly
[x] test suite passes
[x] developments adhere to AQA plan (see docs/aqa/aqa_plan.md)
[ ] data log updated (see docs/aqa/data_log.md), if necessary
[ ] assumptions, and caveats log updated (see docs/aqa/assumptions_caveats.md), if necessary
[x] minimum usable documentation written in the docs folder

Comments have been added below around the incomplete checks.