set_schema overwritting user data coming from inspirehep

cern-sis / issues-inspire

0 stars 0 forks source link

Closed DonHaul closed 2 months ago

DonHaul commented 2 months ago

We have to set the author JSONSchema, currently we do it in airflow but it's wrong.

TECH NOTES:

[x] Remove the set_schema task from airflow here
[x] On author create set the schema "https://inspirehep.net/schemas/records/authors.json" in data.$schema

DonHaul commented 2 months ago

@drjova The workflow data is indeed being passed to, so if you think its a cleaner approach we can add this schema in an airflow stage as well