cgueret
commented
8 years ago We could add one boolean parameter "same-origin" and one integer parameter "radius":
- The first some taking a value of true for restricting adding to the queue items with the same origin of the one being currently added via the tool "crawler-add". When adding a (set of) URI to be crawled this will allow limiting the span of crawling to the same dataset. We will have to account for the fact that, eg, adding http://dbpedia.org/resource/Cardiff results in adding http://dbpedia.org/data/Cardiff.xml which in turns add all the resources pointed to in this description. So to match the expected behaviour from a user point of view the "same-origin=yes" will have to apply to the information resource rather than the non information one. But unless data providers redirect to a different origin we should be fine. That can be implemented in the code using the "is_same_origin" check already there.
- The radius parameter will limit the extent to which recursion is done. -1 would mean no limit, 0 don't recurse, 1 include the objects, 2 the objects pointed to by those objects etc. Same remark as with the previous case. We should ensure that 0 still allows for http://dbpedia.org/data/Cardiff.xml to be added to the queue. This can be implemented by adding an extra column to the DB set to 0 when ingesting using "crawler-add" and incremented when allowed.
On a side note, Anansi currently adds to the queue the objects of rdf:type and then crawls vocabularies. We could try to avoid crawling vocabularies and focus on resources via an additional parameters though this will have to be extended to deal not only with avoiding "rdf:type".
nevali
For instance a parameter "radius=1" to restrict exploration to the direct object properties of the resources explicitly added to the queue.
Internal tracking: RESDATA-962