search

CivicSpleen / ambry

A comprehensive data package manager
BSD 2-Clause "Simplified" License
4 stars 5 forks source link

Sources: Change 'url' and 'generator' to 'ref' #127

Closed ericbusboom closed 8 years ago

ericbusboom commented 8 years ago

There are three values in the DataSource table for referencing the object that produces rows: url, ref and generator. These three should be combined into 'ref'.

Then, the 'urltype' should be changed 'reftype'

nmb10 commented 8 years ago

What are the choices for ref field? What are the choices for reftype field?

ericbusboom commented 8 years ago

Ref can be anything. It is a URL for files with a URL, a partition name or partition id for partitions, a Google Doc ID for google docs. For generator objects, it is the generator class name.

The reftype value is not strongly defined, but there are some values for it in orm.source.DataSourceBase#is_downloadable, which is used to determine if the file should be downloaded during ingest. Then, bundle.bundle.Bundle#_ingest has a special check for a reftype of 'generator'.

The values in orm.source.DataSourceBase#is_downloadable should be moved to a class constant. Currently they are hard coded in the method.

However, I thought I had assigned this one to myself. I've already done part of it, so I'll take you off and finish it up.