search

inspirehep / hepcrawl

Scrapy project for feeds into INSPIRE-HEP
http://inspirehep.net
Other
17 stars 30 forks source link

post-enhancement: complete CC-license information #172

Open fschwenn opened 7 years ago

fschwenn commented 7 years ago

Create a workflow task to complete creativ commons license information which could be added to POSTENHANCE_RECORD for HEP records.

Expected Behavior

In the HEP schema the license contains the license and the url. For CC licenses this is redundant and often only one of the two is contained in the original metadata. Instead of completing it in all the individual crawlers this task could be done in a central place.

michamos commented 7 years ago

I think the literature builder https://github.com/inspirehep/inspire-schemas/blob/36bb1791b4df5890e5445f850c59ed9c5ee9b7c9/inspire_schemas/builders/literature.py#L493-L519 is a better place for this, but I agree in the principle of centralizing this normalization.

fschwenn commented 7 years ago

Obviously you know the system better than me. Does every record 'pass' literature.py? Also user suggestions and new records by BibEdit?

michamos commented 6 years ago

all new records go through the builder. Migrations from legacy and manual edits using the record editor don't. Do you think it's needed there too?

kaplun commented 6 years ago

all new records go through the builder

@michamos besides records we will gather from DESY interim harvester and CDS.

kaplun commented 6 years ago

Maybe that at some point we should ditch inspire-dojson and write a driver that transform bibrec from MARCXML using the Builder.

jacquerie commented 6 years ago

Maybe that at some point we should ditch inspire-dojson and write a driver that transform bibrec from MARCXML using the Builder.

Probably not, because then you will have to reimplement all the normalization that handles anomalies in Legacy's data (you have to ensure that you don't regress on the ~500 test cases in inspire-dojson).

michamos commented 6 years ago

The direction URL -> license name has been added to the builder in inspirehep/inspire-schemas#244 and inspirehep/inspire-schemas#245. The other direction has not been implemented yet.