Have direct ingest send url link

[spacemansteve]

Direct ingest must send the url to the arxiv paper to master. Without this, we risk not having a link on Bumblebee when the myADS email is sent.

[spacemansteve]

Direct ingest must send the url to the arxiv paper to master It must be sent so master can merge it with links data from the nonbib pipeline.

Direct ingest could send a DataLinksRow or DataLinksRowList protobuf. Since nonbib does not send this record (it only sends a NonBibRecord), master will know where this message originated. Master could store this field in a new column in the Records table. These links would be merged into the Solr record.

Other options include:

• Create a new protobuf similar to DataLinksRow and send it and the BibRecord to master for each direct ingest bibcode
• Create a new protobuf that combines the BibRecord and DataLinksRow and send this one protobuf for each direct ingest bibcode
• Add an origin field to DataLinksRow for clarity

[romanchyla]

if i understand correctly:

direct ingest pipeline is producing information that normally is generated by two distinct pipelines

• nonbib
• metadata (bib)

you say 'master must merge this info with other links coming from nonbib pipeline'

is there another pipeline involved in generating data for this field? If it was, we clearly will need new 'origin' - and won't be able to rely on the type of the protobuf; though this doesn't seem to be the case yet

this would be our preferred way:

• generate all relevant data in direct ingest (and send multiple protobufs if necessary)
• modify master pipeline to merge (decide whether to replace/update existing values) if it encounters protobuf that updates something that already exists (note: we keep this local to a protobuf - to one column; we do not contaminate values between multiple columns this way)

new column in the table is not what we want; this datapoint seems not to justify that expense (timestamp etc) - can the decision be made simply using some heuristic rules of nature as