UNSDG text classifier predict - Githubissues

nestauk / old_nesta_daps

[archived]

MIT License

18 stars 5 forks source link

UNSDG text classifier predict #38

Open georgerichardson opened 5 years ago

georgerichardson commented 5 years ago

Create routines and tasks to apply a pre-trained classifier in text fields in order to predict relevant sustainable UNSDG labels.

Misc

[x] Create S3 bucket for data and model
[x] Create fields in ES for model version and UNSDG labels

Logic

[x] Write function to get most recent file from s3
[ ] ~Lemmatizer~ (leaving this for now)
[x] Predict labels with classifier pipe
[ ] ~Format output labels~ (will be done in batchable)

Prepare

[x] Query ES where model ver < current
[x] Get latest model date from s3
[x] Batch returned IDs into files and put in S3 intermediate bucket
[x] Generate list of dicts for batchable params (S3 file locations and model location(s))

Batchable

[x] Collect latest model from s3
[x] Query ES for text using doc id
[x] Preprocess text
[x] Apply prediction
[x] Insert predicted labels to terms_unsdg_abstract and update date_unsdg_model

georgerichardson commented 5 years ago

Main functionality written and working. Now need to move items to packages and write tests.