search

alephdata / memorious

Lightweight web scraping toolkit for documents and structured data.
https://docs.alephdata.org/developers/memorious
MIT License
309 stars 59 forks source link

Load new crawlers without a restart #7

Closed rhiaro closed 6 years ago

rhiaro commented 7 years ago

If new crawlers are added to the CONFIG_PATH the celery workers can't see them without being turned off and on again.

Something about dynamic code reloading.

pudo commented 7 years ago

cf. https://docs.python.org/3.4/library/importlib.html#importlib.reload

pudo commented 7 years ago

Is this still relevant now that we have dev mode, which inlines all crawler execution and is based on many small invocations of the shell?

rhiaro commented 7 years ago

It'd still be nice to be able to add new production crawlers without restarting wouldn't it?

pudo commented 6 years ago

Yeah, but this raises a pretty weird question about live-updating docker containers in production, which I don't think we have a strong enough need for to figure out. My vote would be to close this for the moment.

pudo commented 6 years ago

OK, I don't think we'll come up with a fix for this. Closing unless we have an actual idea for how to wiggle it :)