Make code for writing tables require all the expected columns

[taldcroft]

If I'm understanding correctly this code worries me:

        dtype = DTYPES[table_name]
        names = [n for n in dtype.names if n in data.dtype.names]
        if len(names) == 0:
            raise Exception('Input data has no columns in common with table in DB')
        b = np.zeros(len(data), dtype=dtype)
        b[names] = data[names].as_array().astype(dtype[names])
        data = b

It basically says that the calling function can supply a data table that does not have the expected columns and the missing columns will silently be filled with zeros. I think that finding the expected columns should be required. The following code does this (I think):

dtype = DTYPES[table_name]
data = data[dtype.names].as_array().astype(dtype)

[javierggt]

I added code to check this (in my local copy) and this brings up another issue.

The fields in the dtypes were set more or less following the previous astromon, but at some point we decided we do not care about being compatible. The following fields are currently not being set, so I should set them to default values, set them to proper values or remove the column altogether:

• astromon_obs
  • fids. Not set now. Do we need this here?
  • process_status. Not set right now, but it might be useful if we have a kind if V&V process where we check the sources and matches.
• astromon_xray_src (I think we do not need them)
  • name
  • double_id
  • status_id

[taldcroft]

My inclination is cutting the column names that are not defined. Keep it simple and don't put in scaffolding that we don't obviously need.