search

counterdata-network / story-processor

Story discovery engine for the Counterdata Network. Grabs relevant stories from various APIs, runs them against bespoke classifier models, post results to a central server.
Apache License 2.0
0 stars 2 forks source link

Worker to Download New Models? #88

Open math4humanities opened 1 week ago

math4humanities commented 1 week ago

Currently, worker only restarts and redownloads models after new version releases. This means when new models are added, workers continue to run with the outdated model list. Is there a way for the celery worker to download_models on every run, or when there are new updates to the model list?

rahulbot commented 1 week ago

The queue worker gets batches of stories, where all stories are in the same project, right? Is all the project/model metadata passed to the queue alongside the stories? Hopefully because that's where the thresholding info should come from since it is up to date at time of content fetch.

If so, then is there some way the worker could look at the project/model listed and realize when it is new, referencing some model it doesn't know about... and then it could pause all jobs and download that model? Not sure if that is possible, but it would be "self healing" in a neat way.