new tests/celery task for syncing NEW resources from crawler's DB to metadata.models.Page

AusDTO / disco_layer

Code, outputs and Information relevant to the discovery layer.

1 stars 5 forks source link

Closed monkeypants closed 9 years ago

monkeypants commented 9 years ago

related to #30

How should this go:

[x] trigger periodic job (60s?) from celerybeat?
[x] job pulls (limit 10K?) new resources from crawler, dispatches create task to queue
[x] task creates new Resource into metadata
[x] per #28, index maintenance should trigger automatically...

monkeypants commented 9 years ago

Need tests for this!

disco_service docker image should be able to run as a celerybeat node.