search

dbpedia / links

A repo that contains outgoing links from DBpedia
50 stars 42 forks source link

fix the viaf.org script #7

Closed m1ci closed 7 years ago

m1ci commented 8 years ago

https://github.com/dbpedia/links/blob/master/links/dbpedia.org/viaf.org/scripts/makeLinks.sh

RPOD commented 7 years ago

Changed the resource link of the viaf.org script in 5b93cc145b54767200388d12001ded5c2a86b07b which is reachable. However the script will download a 9.1GB zip file. Should this be changed or this much traffic acceptable?

m1ci commented 7 years ago

I think 9GB is fine. Actually, we will always first need to approve a contribution, so I would not set some general requirement on the traffic, but decide on this at a contribution level.

RPOD commented 7 years ago

What is the maximum size of a file? I would update the How To wiki page to insure that each contributor is aware of it and does not create scripts which generate an unnecessary amount of network traffic

m1ci commented 7 years ago

for a maximum size of a file, please look at the current files and set the size maximum size to a number where most of the current linksets fit in. 0.5GB, 1GB? dont know. As a note you can write smth like: "If your linkset generation needs to download more the 1GB of data, please contact the maintainers." What do you think?

RPOD commented 7 years ago

Sounds reasonable I will add that right away.

kurzum commented 7 years ago
  1. users can provide a download link
  2. we can keep large stable files in http://downloads.dbpedia.org/links/resources/
  3. maximum file size should be 50MB that is already a lot to handle for Git, each change will be logged indefinitely, so if a 50mb file gets updated 10 times it is 500 through all revisions, which you need to fully download on git clone.