Modifying DB schema to support new image format

[Helioviewer-Kirill]

I'm having an issue with modifying the database schema to include a KCOR test image.

Do we have any information about how the datasources are defined in

Specifically the in the create_datasource_table and create_datasource_property_table functions on lines 181 and 298 respectively.

There are other data sources present in that file which are inserted on database creation, and I have tried to mimic their layout to no avail.

I cannot seem to avoid crashing the db creation installer when it comes time to parse the source object in on line 102.

I'm getting a KeyError: 'COSMO_K-Coronagraph' before and after trying to use that as the value for many of the definition fields in the schema.

The xmlbox in the jp2 file I'm attempting to import does not contain that value in any field, the most similar value is: `COSMO K-Coronagraph

COSMO K-Coronagraph` (note: no underscore present) I'm not sure what could be causing this error, whether it's the schema definition or the image attributes stored in XML.

[wafels]

have you added a new datasource to the end of the existing datasources in create_datasource_table ?

[Helioviewer-Kirill]

Yes, I've tried to add (83, 'COSMO_K-Coronagraph', 'COSMO_K-Coronagraph', 'DN', 2, 0, '', 1, 0, 0, 0) to the datasources table as well as (83, 'Observatory', 'COSMO_K-Coronagraph', 'COSMO_K-Coronagraph', 'COSMO_K-Coronagraph', '2'), (83, 'Instrument', 'COSMO_K-Coronagraph', 'COSMO_K-Coronagraph', 'COSMO_K-Coronagraph', '2'), (83, 'Measurement', 'white-light-pB', 'white-light-pB', 'MLSO', '2'), to the datasource_property table. This still gives me the KeyError mentioned above.

I am attempting to do the same but with spaces instead of underscores now, to match the xml box metadata in the image.

[Helioviewer-Kirill]

Using spaces rather than underscores did not work either. I've tried many combinations and none of them seem to work.

I'm confused how the datasource should be defined. What are the guidelines for formatting the datasource and datasource_property db entries?

[wafels]

There is some information on the wiki - http://wiki.helioviewer.org/wiki/Adding_Support_for_New_Data

[Helioviewer-Kirill]

Thanks, but unfortunately that page does not provide any useful info on how to format the new db entries, it simply states the files which need to be modified.

Also, the first part of that article talks about modifying sunpy and I am unable to get the sunpy functions read_header() or make_map() to invoke at all. It's not clear how this part of the process is related to adding the datasource.

[ddialer]

Kirill,

Could you post your updated db.py file here with new data sources you trying to add?

[Helioviewer-Kirill]

I can't seem to be able to attach python files here, so I renamed it to .txt db.txt

The only changes to this file are highlighted in my second post on this thread located on line 282 in create_datasource_table and on lines 603-605 in create_datasource_property_table.

I used a source id of 83 as a test for these entries.

[ddialer]

I just executed scripts without any issues.

A you trying to run install script as a new installation or trying to update existing database?

[Helioviewer-Kirill]

I've noticed the inconsistency extends into how get_datasources is querying the data from the db and then parsing it into a sources object.

The returned sources object does not contain the data that I've entered formatted similarly to the other datasources.

Running the sql query generated by the get_datasources function manually and inspecting the output I realized that the uiOrder field is the key to my mistake.

[Helioviewer-Kirill]

I was attempting to run the install script as a new installation when receiving the keyerror

[Helioviewer-Kirill]

The entries for the datasource_property table should have read: (83, 'Observatory', 'COSMO_K-Coronagraph', 'COSMO_K-Coronagraph', 'COSMO_K-Coronagraph', '1'), (83, 'Instrument', 'COSMO', 'COSMO', 'COSMO', '2'), (83, 'Detector', '', '', '', '3'), (83, 'Measurement', '735', '735', '735', '4'),

Where my mistake was the last number in each line, the uiOrder field.

On locating the property values: I could not locate these values by using jp2dump in the image header. I followed the breadcrumbs in the crash logs to attain the correct keys for each of those fields ["observatory", "instrument", "detector", "measurement"].

[wafels]

Hello @ddialer !

[ddialer]

Hello @wafels ! Had a moment and decided to help, but looks like Kirill solved the issue)