Optimise title truncation

Living-with-machines / lwmdb

A django-based library for managing the Living with Machines newspapers metadata database schema

https://living-with-machines.github.io/lwmdb/

MIT License

2 stars 0 forks source link

Open griff-rees opened 1 year ago

griff-rees commented 1 year ago

The callable_on_chunk function in lwmdb.utils could be improved by one or more of the following:

increase batch_size (currently set to 2000)
refactor to use bulk_update rather than save
async functions/methods (like asave and abulk_update)
split data to even portions from records, managing each portion as a subprocess
limit the number of fields saved/updated to just those changing. Specifically:
- title_word_count
- title_char_count
- title_truncated

griff-rees commented 1 year ago

Feature provided in #170 is an example of using bulk_update and could be general enough to apply here.