Create SQLAlchemy ORM models for Climsoft CDMS

isedwards commented 3 years ago

This issue will begin to add support for the Climsoft CDMS. You will need an instance of MySQL 8+ installed and then to restore the Climsoft 4.1.1 SQL DDL file here.

When the Climsoft software is installed, the SQL DDL creates two databases:

CREATE DATABASE IF NOT EXISTS `mariadb_climsoft_db_v4`
CREATE DATABASE IF NOT EXISTS `mariadb_climsoft_test_db_v4`

The first is used as the actual climate database whilst the second contains some test data that is used for training with tutorials.

We only need to create models for mariadb_climsoft_db_v4. The models should be created in /opencdms/models/climsoft/v4_1_1.py.

Once we have the models, we should try using the get_schema_diff function from opencdms.models to check our new models against the second database schema (mariadb_climsoft_test_db_v4) to see whether it is identical. It's possible that there may be minor differences.

There is one final complication: some of the tables in Climsoft do not have primary keys explicitly specified - this prevents SQLAlchemy/sqlacodegen from creating ORM models (see docs here). The attached PDF describes where primary keys should be specified.

We should discuss our options for working around the primary key issue. We can either:

Submit a change request to the Climsoft project to ALTER TABLE and add the missing primary keys, this will then appear in future releases of Climsoft
Alter our local copy of the schema to make it appear that primary keys are specified. If my understanding is correct, we should then be able to use our models (that explicitly set the primary keys) to connect to current versions of the database where many primary keys are not explicitly defined, but everything will work correctly.

I'm very happy to discuss this more - either in GitHub comments, messaging or scheduling a call... Climsoft Data Model Tables and Keys.pdf

Shaibujnr commented 3 years ago

@isedwards Concerning the primary key issue, I'd rather we go with option 2 at the moment. However I am having difficulty understanding the table in the pdf. It would be nice to have a more detailed explanation. We could also discuss in a call.

isedwards commented 3 years ago

@Shaibujnr I have a bit more information on this from https://github.com/climsoft/climsoft-database/issues/2#issuecomment-908196231

We should start by only creating models for the 24 main tables that are considered to "constitute the relational data model". These 24 tables are listed in the PDF file above. The other tables should be ignored for now.

These 24 tables either have a 'single column' primary key already specified - or they have a unique index on a composite key that we can use as the primary key.

Taking the first two tables from the PDF, these have:

UNIQUE INDEX `obsFinalIdentification` (`recordedFrom`, `describedBy`, `obsDatetime`)
UNIQUE INDEX `obsInitialIdentification` (`recordedFrom`, `describedBy`, `obsDatetime`, `qcStatus`, `acquisitionType`)

We can use @faysal-ishtiaq 's ALTER TABLE commands to add the primary keys for the tables in the PDF that need them.

So...

Run the ALTER TABLE commands on a local copy of the database - just for those of the 24 tables in the PDF file that need a composite primary key added
Find a way to get sqlacodegen to generate models for just the 24 tables in the PDF
Run get_schema_diff() against the automatically generated models for these 24 tables that have had PKs added and paste the diff for us all to see in the comments
Run alembic revision --autogenerate to generate a migration file for the difference between the current database schema and the defined SQLAlchemy models (as here)
On a separate commit, manually change our models so that they match the output of the alembic migration
Run get_schema_diff() again and report any differences
Can any remaining differences be fixed (or explicitly ignored, like we did with ignore_tables in CliDE so that views were not reported)

Shaibujnr commented 3 years ago

Got it @isedwards , I have been able to install mysql locally and run the DDL file, I now have the main database and test database as well.

I am about to create the primary keys for the tables with the unique identifiers. I observed that in the pdf, for table observationfinal, the columns recordedFrom, describedBy, obsDatetime, obsLevel were listed as the unique keys, but on viewing the table properties of my db locally, the columns recordedFrom, describedBy, obsDatetime were the only unique constraints. Is this an error with the pdf or is my db not created properly?

isedwards commented 3 years ago

In production, the unique indexes on the Climsoft observationinitial and observationfinal tables are currently missing the obslevel field.

In the case of observationfinal this means that only one 'observation level' can be recorded for each unique [recordedFrom, describedBy, obsDatetime] row and similarly with the observationinitial table.

In practice, I think obslevel is currently not implemented and therefore it always has the default value of NULL (different 'observation levels' are currently captured by having multiple describedBy values that also capture the 'level' in addition to other characteristics).

In OpenCDMS, we should include obslevel in the primary key for correctness. The risk is if the OpenCDMS software was used alonside Climsoft - and OpenCDMS allowed users to record multiple 'observation levels' - then this could lead to the Climsoft software overwritting data because the software does not recognised the obslevel field until a fix is implemented.

Although our model will include obslevel in the primary key, the actual database implementation will not include this in the unique index until a fix is implemented in Climsoft. This means that when OpenCDMS queries an older version of the Climsoft database - the obslevel field will not be indexed. However, I doubt there would be any performance issues given there is only one obslevel per unique row and the value is NULL.

isedwards commented 3 years ago

After applying the alter tables commands locally:

ALTER TABLE mariadb_climsoft_db_v4.observationfinal ADD PRIMARY KEY (recordedFrom, describedBy, obsDatetime, obsLevel);
ALTER TABLE mariadb_climsoft_db_v4.observationinitial ADD PRIMARY KEY (recordedFrom, describedBy, obsDatetime, qcStatus,acquisitionType, obsLevel);
ALTER TABLE ...

we should then generate the models with sqlacodegen for the altered version and, given the output of alembic, manually update the models (on a separate commit so we can see the diff). We're then looking for whether:

get_schema_diff() reports no differences with our locally altered database
How to control the output of get_schema_diff() so that it does not report the known difference between our model PKs and the original Climsoft database's unique indexes (whilst ensuring that it continues to report any unexpected differences).

Shaibujnr commented 3 years ago

I have been able to generate the models for the tables listed in the pdf, all 24 (25 actually) of them. 25 because one of the tables has a relationship to a table not listed in the pdf and therefore sqlacodegen automatically generated that for us. However, I am currently facing a dilemma with the get_schema_diff.

If I run get_schema_diff as is, with only the 24 models defined, against all the tables in the db, it will report all differences from all the tables, including/especially those for which the models have not been generated. Is this the output you'd like to see? It's actually a lot. @isedwards

I have currently updated the get_schema_diff method to accept include_tables and exclude_tables arguments. exclude_tables is exactly the same as ignore_tables. Think of it as blacklisting and whitelisting. exclude_tables will blacklist the tables in the list and therefore not report any operation difference for those tables in the result of get_schema_diff. include_tables on the other hand will ignore every operation not related to the listed tables.

I believe this is the output you'd like to see, am I correct? @isedwards

To describe the issue, i'd first like to describe the output of alembic's compare_metadata. It's simply a list of tuples. The first item in the tuple is a string describing the operation to be performed e.g add_table, remove_table, add_index, remove_index and the second item is the sqlalchemy object for that operation. e.g for the add_table operation, the second item in the tuple will be a Table object that should be added, similarly for remove_table but for operations like add_index the second object in the tuple would be an sqlalchemy Index object that has a table as it's attribute.

Now the issue. To whitelist/blacklist the report i.e constraint the diff to only report differences concerning a specified list of tables like the 24 we have pre-selected. I have to identify what table every operation is targeted at and filter by that table. For operations like add_table, remove_table I can easily pick the second element in the tuple and just filter. But for operations like add_index or remove_index I have to get the table name from the index and then filter. Meaning, the filter logic would be different for every operation group. It's difficult to tell all the operation commands alembic currently supports and the logic to handle them, and therefore there's a high chance we'd get inaccurate results if we don't handle all the possible operations properly.

Shaibujnr commented 3 years ago

This is diff report generated for climsoft without whitelisting or blacklisting.

[   (   'remove_table',
        Table('tm_309052', MetaData(), Column('order', INTEGER(), table=<tm_309052>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x11fe44d90>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_309052>), Column('CREX_Template', VARCHAR(length=50), table=<tm_309052>), Column('Sequence_Descriptor1', VARCHAR(length=255), table=<tm_309052>), Column('Sequence_Descriptor0', VARCHAR(length=255), table=<tm_309052>), Column('Bufr_Element', VARCHAR(length=255), table=<tm_309052>), Column('Crex_Element', VARCHAR(length=50), table=<tm_309052>), Column('Climsoft_Element', VARCHAR(length=50), table=<tm_309052>), Column('Element_Name', VARCHAR(length=255), table=<tm_309052>), Column('Crex_Unit', VARCHAR(length=25), table=<tm_309052>), Column('Crex_Scale', VARCHAR(length=25), table=<tm_309052>), Column('Crex_DataWidth', VARCHAR(length=25), table=<tm_309052>), Column('Bufr_Unit', VARCHAR(length=255), table=<tm_309052>), Column('Bufr_Scale', INTEGER(), table=<tm_309052>), Column('Bufr_RefValue', BIGINT(), table=<tm_309052>), Column('Bufr_DataWidth_Bits', INTEGER(), table=<tm_309052>), Column('Selected', TINYINT(), table=<tm_309052>), Column('Observation', VARCHAR(length=255), table=<tm_309052>), Column('Crex_Data', VARCHAR(length=30), table=<tm_309052>), Column('Bufr_Data', VARCHAR(length=255), table=<tm_309052>), schema=None)),
    (   'remove_table',
        Table('tm_307086', MetaData(), Column('order', INTEGER(), table=<tm_307086>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x120655160>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_307086>), Column('Crex_Template', VARCHAR(length=50), table=<tm_307086>), Column('Sequence_Descriptor1', VARCHAR(length=50), table=<tm_307086>), Column('Sequence_Descriptor0', VARCHAR(length=50), table=<tm_307086>), Column('Bufr_Element', VARCHAR(length=50), table=<tm_307086>), Column('Crex_Element', VARCHAR(length=255), table=<tm_307086>), Column('Climsoft_Element', VARCHAR(length=255), table=<tm_307086>), Column('Element_Name', VARCHAR(length=255), table=<tm_307086>), Column('synop_code', VARCHAR(length=255), table=<tm_307086>), Column('unit', VARCHAR(length=255), table=<tm_307086>), Column('scale', VARCHAR(length=255), table=<tm_307086>), Column('width', VARCHAR(length=255), table=<tm_307086>), Column('units_length_scale', VARCHAR(length=255), table=<tm_307086>), Column('data_type', VARCHAR(length=255), table=<tm_307086>), Column('selected', TINYINT(), table=<tm_307086>), Column('observation', VARCHAR(length=255), table=<tm_307086>), Column('Crex_Data', VARCHAR(length=255), table=<tm_307086>), Column('Bufr_Data', VARCHAR(length=50), table=<tm_307086>), schema=None)),
    (   'remove_table',
        Table('tm_307084', MetaData(), Column('order', INTEGER(), table=<tm_307084>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x12065e280>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_307084>), Column('Crex_Template', VARCHAR(length=50), table=<tm_307084>), Column('Sequence_Descriptor1', VARCHAR(length=50), table=<tm_307084>), Column('Sequence_Descriptor0', VARCHAR(length=50), table=<tm_307084>), Column('Bufr_Element', VARCHAR(length=50), table=<tm_307084>), Column('Crex_Element', VARCHAR(length=255), table=<tm_307084>), Column('Climsoft_Element', VARCHAR(length=255), table=<tm_307084>), Column('Element_Name', VARCHAR(length=255), table=<tm_307084>), Column('synop_code', VARCHAR(length=255), table=<tm_307084>), Column('unit', VARCHAR(length=255), table=<tm_307084>), Column('scale', VARCHAR(length=255), table=<tm_307084>), Column('width', VARCHAR(length=255), table=<tm_307084>), Column('units_length_scale', VARCHAR(length=255), table=<tm_307084>), Column('data_type', VARCHAR(length=255), table=<tm_307084>), Column('selected', TINYINT(), table=<tm_307084>), Column('observation', VARCHAR(length=255), table=<tm_307084>), Column('Crex_Data', VARCHAR(length=255), table=<tm_307084>), Column('Bufr_Data', VARCHAR(length=50), table=<tm_307084>), schema=None)),
    (   'remove_index',
        Index('obsInitialIdentification', Column('StationId', VARCHAR(length=15), table=<qcabslimits>, nullable=False), Column('ElementId', BIGINT(), table=<qcabslimits>), Column('Datetime', DATETIME(), table=<qcabslimits>))),
    (   'remove_table',
        Table('qcabslimits', MetaData(), Column('StationId', VARCHAR(length=15), table=<qcabslimits>, nullable=False), Column('ElementId', BIGINT(), table=<qcabslimits>), Column('Datetime', DATETIME(), table=<qcabslimits>), Column('YYYY', INTEGER(), table=<qcabslimits>), Column('mm', TINYINT(), table=<qcabslimits>), Column('dd', TINYINT(), table=<qcabslimits>), Column('hh', TINYINT(), table=<qcabslimits>), Column('obsValue', VARCHAR(length=10), table=<qcabslimits>), Column('limitValue', VARCHAR(length=10), table=<qcabslimits>), Column('qcStatus', INTEGER(), table=<qcabslimits>), Column('acquisitionType', INTEGER(), table=<qcabslimits>), Column('obsLevel', VARCHAR(length=255), table=<qcabslimits>), Column('capturedBy', VARCHAR(length=255), table=<qcabslimits>), Column('dataForm', VARCHAR(length=255), table=<qcabslimits>), schema=None)),
    (   'remove_table',
        Table('climsoftusers', MetaData(), Column('userName', VARCHAR(length=50), table=<climsoftusers>, primary_key=True, nullable=False), Column('userRole', VARCHAR(length=50), table=<climsoftusers>, nullable=False), schema=None)),
    (   'remove_table',
        Table('seq_month_day_synoptime', MetaData(), Column('mm', INTEGER(), table=<seq_month_day_synoptime>, primary_key=True, nullable=False), Column('dd', INTEGER(), table=<seq_month_day_synoptime>, primary_key=True, nullable=False), Column('hh', INTEGER(), table=<seq_month_day_synoptime>, primary_key=True, nullable=False), schema=None)),
    (   'remove_table',
        Table('seq_month_day', MetaData(), Column('mm', INTEGER(), table=<seq_month_day>, primary_key=True, nullable=False), Column('dd', INTEGER(), table=<seq_month_day>, primary_key=True, nullable=False), schema=None)),
    (   'remove_table',
        Table('qc_interelement_relationship_definition', MetaData(), Column('elementId_1', INTEGER(), table=<qc_interelement_relationship_definition>, primary_key=True, nullable=False), Column('relationship', VARCHAR(length=50), table=<qc_interelement_relationship_definition>, primary_key=True, nullable=False), Column('elementId_2', INTEGER(), table=<qc_interelement_relationship_definition>, primary_key=True, nullable=False), schema=None)),
    (   'remove_table',
        Table('seq_month_day_element', MetaData(), Column('mm', INTEGER(), table=<seq_month_day_element>, nullable=False), Column('dd', INTEGER(), table=<seq_month_day_element>, nullable=False), Column('elementId', INTEGER(), table=<seq_month_day_element>, nullable=False), schema=None)),
    (   'remove_table',
        Table('aws_stations', MetaData(), Column('aws_id', VARCHAR(length=50), table=<aws_stations>, primary_key=True, nullable=False), Column('national_id', VARCHAR(length=50), table=<aws_stations>, primary_key=True, nullable=False), Column('station_name', VARCHAR(length=50), table=<aws_stations>), schema=None)),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_lsi>, nullable=False))),
    (   'remove_table',
        Table('aws_lsi', MetaData(), Column('Cols', INTEGER(), table=<aws_lsi>, nullable=False), Column('Element_abbreviation', VARCHAR(length=20), table=<aws_lsi>), Column('Element_Name', VARCHAR(length=20), table=<aws_lsi>), Column('Element_Details', VARCHAR(length=25), table=<aws_lsi>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_lsi>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_lsi>), Column('unit', VARCHAR(length=15), table=<aws_lsi>), Column('lower_limit', VARCHAR(length=10), table=<aws_lsi>), Column('upper_limit', VARCHAR(length=10), table=<aws_lsi>), Column('obsv', VARCHAR(length=25), table=<aws_lsi>), schema=None)),
    (   'remove_table',
        Table('tm_307073', MetaData(), Column('order', DOUBLE(asdecimal=True), table=<tm_307073>), Column('Bufr_Template', VARCHAR(length=255), table=<tm_307073>), Column('CREX_Template', VARCHAR(length=255), table=<tm_307073>), Column('Sequence_Descriptor1', VARCHAR(length=255), table=<tm_307073>), Column('Sequence_Descriptor0', VARCHAR(length=255), table=<tm_307073>), Column('Bufr_Element', VARCHAR(length=255), table=<tm_307073>), Column('Crex_Element', VARCHAR(length=255), table=<tm_307073>), Column('Climsoft_Element', VARCHAR(length=255), table=<tm_307073>), Column('Element_Name', VARCHAR(length=255), table=<tm_307073>), Column('Crex_Unit', VARCHAR(length=255), table=<tm_307073>), Column('Crex_Scale', VARCHAR(length=255), table=<tm_307073>), Column('Crex_DataWidth', VARCHAR(length=255), table=<tm_307073>), Column('Bufr_Unit', VARCHAR(length=255), table=<tm_307073>), Column('Bufr_Scale', DOUBLE(asdecimal=True), table=<tm_307073>), Column('Bufr_RefValue', DOUBLE(asdecimal=True), table=<tm_307073>), Column('Bufr_DataWidth_Bits', DOUBLE(asdecimal=True), table=<tm_307073>), Column('Selected', TINYINT(), table=<tm_307073>), Column('Observation', VARCHAR(length=255), table=<tm_307073>), Column('Crex_Data', VARCHAR(length=255), table=<tm_307073>), Column('Bufr_Data', VARCHAR(length=255), table=<tm_307073>), schema=None)),
    (   'remove_table',
        Table('bufr_indicators', MetaData(), Column('Tmplate', VARCHAR(length=50), table=<bufr_indicators>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x12069bdc0>, for_update=False)), Column('Msg_Header', VARCHAR(length=50), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a6160>, for_update=False)), Column('BUFR_Edition', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a6250>, for_update=False)), Column('Originating_Centre', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a6340>, for_update=False)), Column('Originating_SubCentre', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a6430>, for_update=False)), Column('Update_Sequence', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a6520>, for_update=False)), Column('Optional_Section', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a6610>, for_update=False)), Column('Data_Category', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a6700>, for_update=False)), Column('Intenational_Data_SubCategory', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a67f0>, for_update=False)), Column('Local_Data_SubCategory', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a68e0>, for_update=False)), Column('Master_table', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a69d0>, for_update=False)), Column('Local_Table', VARCHAR(length=10), table=<bufr_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206a6ac0>, for_update=False)), schema=None)),
    (   'remove_table',
        Table('aws_sasscal1', MetaData(), Column('Cols', INTEGER(), table=<aws_sasscal1>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206ad400>, for_update=False)), Column('Element_Abbreviation', VARCHAR(length=50), table=<aws_sasscal1>), Column('Element_Name', VARCHAR(length=50), table=<aws_sasscal1>), Column('Element_Details', VARCHAR(length=50), table=<aws_sasscal1>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_sasscal1>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_sasscal1>), Column('unit', VARCHAR(length=15), table=<aws_sasscal1>), Column('lower_limit', VARCHAR(length=50), table=<aws_sasscal1>), Column('upper_limit', VARCHAR(length=50), table=<aws_sasscal1>), Column('obsv', VARCHAR(length=50), table=<aws_sasscal1>), schema=None)),
    (   'remove_table',
        Table('tm_307081', MetaData(), Column('Nos', INTEGER(), table=<tm_307081>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206b8130>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_307081>), Column('Crex_Template', VARCHAR(length=50), table=<tm_307081>), Column('Sequence_Descriptor1', VARCHAR(length=50), table=<tm_307081>), Column('Sequence_Descriptor0', VARCHAR(length=50), table=<tm_307081>), Column('Bufr_Element', VARCHAR(length=50), table=<tm_307081>), Column('Crex_Element', VARCHAR(length=255), table=<tm_307081>), Column('Climsoft_Element', VARCHAR(length=255), table=<tm_307081>), Column('Element_Name', VARCHAR(length=255), table=<tm_307081>), Column('synop_code', VARCHAR(length=255), table=<tm_307081>), Column('unit', VARCHAR(length=255), table=<tm_307081>), Column('scale', VARCHAR(length=255), table=<tm_307081>), Column('width', VARCHAR(length=255), table=<tm_307081>), Column('units_length_scale', VARCHAR(length=255), table=<tm_307081>), Column('data_type', VARCHAR(length=255), table=<tm_307081>), Column('selected', TINYINT(), table=<tm_307081>), Column('Observation', VARCHAR(length=255), table=<tm_307081>), Column('Crex_Data', VARCHAR(length=255), table=<tm_307081>), Column('Bufr_Data', VARCHAR(length=50), table=<tm_307081>), schema=None)),
    (   'remove_index',
        Index('identification', Column('No', INTEGER(), table=<aws1>, nullable=False))),
    (   'remove_table',
        Table('aws1', MetaData(), Column('No', INTEGER(), table=<aws1>, nullable=False), Column('Element_abbreviation', TEXT(), table=<aws1>, nullable=False), Column('Element_Name', TEXT(), table=<aws1>, nullable=False), Column('Element_Details', TEXT(), table=<aws1>, nullable=False), Column('Climsoft_Element', TEXT(), table=<aws1>, nullable=False), Column('Bufr_Element', TEXT(), table=<aws1>, nullable=False), Column('unit', TEXT(), table=<aws1>, nullable=False), Column('lower_limit', TEXT(), table=<aws1>, nullable=False), Column('upper_limit', TEXT(), table=<aws1>, nullable=False), Column('obsv', TEXT(), table=<aws1>, nullable=False), schema=None)),
    (   'remove_table',
        Table('tm_307080', MetaData(), Column('order', INTEGER(), table=<tm_307080>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206c2fd0>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_307080>), Column('Crex_Template', VARCHAR(length=50), table=<tm_307080>), Column('Sequence_Descriptor1', VARCHAR(length=50), table=<tm_307080>), Column('Sequence_Descriptor0', VARCHAR(length=50), table=<tm_307080>), Column('Bufr_Element', VARCHAR(length=50), table=<tm_307080>), Column('Crex_Element', VARCHAR(length=255), table=<tm_307080>), Column('Climsoft_Element', VARCHAR(length=255), table=<tm_307080>), Column('Element_Name', VARCHAR(length=255), table=<tm_307080>), Column('synop_code', VARCHAR(length=255), table=<tm_307080>), Column('unit', VARCHAR(length=255), table=<tm_307080>), Column('scale', VARCHAR(length=255), table=<tm_307080>), Column('width', VARCHAR(length=255), table=<tm_307080>), Column('units_length_scale', VARCHAR(length=255), table=<tm_307080>), Column('data_type', VARCHAR(length=255), table=<tm_307080>), Column('selected', TINYINT(), table=<tm_307080>), Column('observation', VARCHAR(length=255), table=<tm_307080>), Column('Crex_Data', VARCHAR(length=255), table=<tm_307080>), Column('Bufr_Data', VARCHAR(length=50), table=<tm_307080>), schema=None)),
    (   'remove_table',
        Table('stationnetworkdefinition', MetaData(), Column('networkAbbreviation', VARCHAR(length=255), table=<stationnetworkdefinition>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206d81c0>, for_update=False)), Column('networkFullName', VARCHAR(length=255), table=<stationnetworkdefinition>), schema=None)),
    (   'remove_table',
        Table('seq_year', MetaData(), Column('yyyy', INTEGER(), table=<seq_year>), schema=None)),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_lsi1>, nullable=False))),
    (   'remove_table',
        Table('aws_lsi1', MetaData(), Column('Cols', INTEGER(), table=<aws_lsi1>, nullable=False), Column('Element_abbreviation', VARCHAR(length=20), table=<aws_lsi1>), Column('Element_Name', VARCHAR(length=20), table=<aws_lsi1>), Column('Element_Details', VARCHAR(length=25), table=<aws_lsi1>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_lsi1>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_lsi1>), Column('unit', VARCHAR(length=15), table=<aws_lsi1>), Column('lower_limit', VARCHAR(length=10), table=<aws_lsi1>), Column('upper_limit', VARCHAR(length=10), table=<aws_lsi1>), Column('obsv', VARCHAR(length=25), table=<aws_lsi1>), schema=None)),
    (   'remove_table',
        Table('seq_month_day_leap_yr', MetaData(), Column('mm', INTEGER(), table=<seq_month_day_leap_yr>, primary_key=True, nullable=False), Column('dd', INTEGER(), table=<seq_month_day_leap_yr>, primary_key=True, nullable=False), schema=None)),
    (   'remove_table',
        Table('tblproducts', MetaData(), Column('productId', VARCHAR(length=10), table=<tblproducts>, primary_key=True, nullable=False), Column('productName', VARCHAR(length=50), table=<tblproducts>), Column('prDetails', VARCHAR(length=50), table=<tblproducts>), Column('prCategory', VARCHAR(length=50), table=<tblproducts>), schema=None)),
    (   'remove_table',
        Table('aws_toa5_mg2', MetaData(), Column('Cols', BIGINT(), table=<aws_toa5_mg2>, primary_key=True, nullable=False), Column('Element_Abbreviation', VARCHAR(length=50), table=<aws_toa5_mg2>), Column('Element_Name', VARCHAR(length=50), table=<aws_toa5_mg2>), Column('Element_Details', VARCHAR(length=50), table=<aws_toa5_mg2>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_toa5_mg2>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_toa5_mg2>), Column('unit', VARCHAR(length=15), table=<aws_toa5_mg2>), Column('lower_limit', VARCHAR(length=50), table=<aws_toa5_mg2>), Column('upper_limit', VARCHAR(length=50), table=<aws_toa5_mg2>), Column('obsv', VARCHAR(length=50), table=<aws_toa5_mg2>), schema=None)),
    (   'remove_table',
        Table('aws_tahmo', MetaData(), Column('Cols', INTEGER(), table=<aws_tahmo>, primary_key=True, nullable=False), Column('Element_abbreviation', VARCHAR(length=50), table=<aws_tahmo>), Column('Element_Name', VARCHAR(length=50), table=<aws_tahmo>), Column('Element_Details', VARCHAR(length=50), table=<aws_tahmo>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_tahmo>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_tahmo>), Column('unit', VARCHAR(length=15), table=<aws_tahmo>), Column('lower_limit', VARCHAR(length=50), table=<aws_tahmo>), Column('upper_limit', VARCHAR(length=50), table=<aws_tahmo>), Column('obsv', VARCHAR(length=50), table=<aws_tahmo>), schema=None)),
    (   'remove_table',
        Table('seq_month_day_synoptime_leap_yr', MetaData(), Column('mm', INTEGER(), table=<seq_month_day_synoptime_leap_yr>, primary_key=True, nullable=False), Column('dd', INTEGER(), table=<seq_month_day_synoptime_leap_yr>, primary_key=True, nullable=False), Column('hh', INTEGER(), table=<seq_month_day_synoptime_leap_yr>, primary_key=True, nullable=False), schema=None)),
    (   'remove_table',
        Table('aws_rwanda4', MetaData(), Column('Cols', INTEGER(), table=<aws_rwanda4>, primary_key=True, nullable=False), Column('Element_Name', VARCHAR(length=20), table=<aws_rwanda4>), Column('Element_Abbreviation', VARCHAR(length=20), table=<aws_rwanda4>), Column('Element_Details', VARCHAR(length=25), table=<aws_rwanda4>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_rwanda4>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_rwanda4>), Column('unit', VARCHAR(length=15), table=<aws_rwanda4>), Column('lower_limit', VARCHAR(length=10), table=<aws_rwanda4>), Column('upper_limit', VARCHAR(length=10), table=<aws_rwanda4>), Column('obsv', VARCHAR(length=25), table=<aws_rwanda4>), schema=None)),
    (   'remove_index',
        Index('identification', Column('Element_Abbreviation', VARCHAR(length=50), table=<aws_toa5_bw1>))),
    (   'remove_table',
        Table('aws_toa5_bw1', MetaData(), Column('Cols', INTEGER(), table=<aws_toa5_bw1>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x12070f610>, for_update=False)), Column('Element_Abbreviation', VARCHAR(length=50), table=<aws_toa5_bw1>), Column('Element_Name', VARCHAR(length=50), table=<aws_toa5_bw1>), Column('Element_Details', VARCHAR(length=50), table=<aws_toa5_bw1>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_toa5_bw1>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_toa5_bw1>), Column('unit', VARCHAR(length=15), table=<aws_toa5_bw1>), Column('lower_limit', VARCHAR(length=50), table=<aws_toa5_bw1>), Column('upper_limit', VARCHAR(length=50), table=<aws_toa5_bw1>), Column('obsv', VARCHAR(length=50), table=<aws_toa5_bw1>), schema=None)),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<testing_aws>, nullable=False))),
    (   'remove_table',
        Table('testing_aws', MetaData(), Column('Cols', INTEGER(), table=<testing_aws>, nullable=False), Column('Element_abbreviation', VARCHAR(length=20), table=<testing_aws>), Column('Element_Name', VARCHAR(length=20), table=<testing_aws>), Column('Element_Details', VARCHAR(length=25), table=<testing_aws>), Column('Climsoft_Element', VARCHAR(length=6), table=<testing_aws>), Column('Bufr_Element', VARCHAR(length=6), table=<testing_aws>), Column('unit', VARCHAR(length=15), table=<testing_aws>), Column('lower_limit', VARCHAR(length=10), table=<testing_aws>), Column('upper_limit', VARCHAR(length=10), table=<testing_aws>), Column('obsv', VARCHAR(length=25), table=<testing_aws>), schema=None)),
    (   'remove_table',
        Table('seq_month', MetaData(), Column('mm', VARCHAR(length=50), table=<seq_month>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x120722310>, for_update=False)), schema=None)),
    (   'remove_table',
        Table('missing_data', MetaData(), Column('STN_ID', VARCHAR(length=255), table=<missing_data>, primary_key=True, nullable=False), Column('OBS_DATE', DATE(), table=<missing_data>, primary_key=True, nullable=False), Column('ELEM', BIGINT(), table=<missing_data>, primary_key=True, nullable=False), schema=None)),
    (   'remove_index',
        Index('identification', Column('Element_Name', VARCHAR(length=50), table=<aws_rwanda1>))),
    (   'remove_table',
        Table('aws_rwanda1', MetaData(), Column('Cols', INTEGER(), table=<aws_rwanda1>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1207296a0>, for_update=False)), Column('Element_abbreviation', VARCHAR(length=50), table=<aws_rwanda1>), Column('Element_Name', VARCHAR(length=50), table=<aws_rwanda1>), Column('Element_Details', VARCHAR(length=50), table=<aws_rwanda1>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_rwanda1>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_rwanda1>), Column('unit', VARCHAR(length=15), table=<aws_rwanda1>), Column('lower_limit', VARCHAR(length=50), table=<aws_rwanda1>), Column('upper_limit', VARCHAR(length=50), table=<aws_rwanda1>), Column('obsv', VARCHAR(length=50), table=<aws_rwanda1>), schema=None)),
    (   'remove_index',
        Index('synop_code', Column('Element_Name', VARCHAR(length=255), table=<tm_307089>))),
    (   'remove_index',
        Index('synop_code1', Column('synop_code', VARCHAR(length=50), table=<tm_307089>))),
    (   'remove_table',
        Table('tm_307089', MetaData(), Column('order', INTEGER(), table=<tm_307089>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x120732550>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_307089>), Column('Crex_Template', VARCHAR(length=50), table=<tm_307089>), Column('Sequence_Descriptor1', VARCHAR(length=10), table=<tm_307089>), Column('Sequence_Descriptor0', VARCHAR(length=10), table=<tm_307089>), Column('Bufr_Element', VARCHAR(length=50), table=<tm_307089>), Column('Crex_Element', VARCHAR(length=255), table=<tm_307089>), Column('Climsoft_Element', VARCHAR(length=50), table=<tm_307089>), Column('Element_Name', VARCHAR(length=255), table=<tm_307089>), Column('synop_code', VARCHAR(length=50), table=<tm_307089>), Column('unit', VARCHAR(length=50), table=<tm_307089>), Column('scale', VARCHAR(length=50), table=<tm_307089>), Column('width', VARCHAR(length=50), table=<tm_307089>), Column('units_length_scale', VARCHAR(length=255), table=<tm_307089>), Column('data_type', VARCHAR(length=255), table=<tm_307089>), Column('selected', TINYINT(), table=<tm_307089>), Column('observation', VARCHAR(length=100), table=<tm_307089>), Column('crex', VARCHAR(length=25), table=<tm_307089>), Column('Crex_Data', VARCHAR(length=255), table=<tm_307089>), Column('Bufr_Data', VARCHAR(length=50), table=<tm_307089>), schema=None)),
    (   'remove_index',
        Index('Identification', Column('Element_Name', VARCHAR(length=50), table=<aws_malawi1>))),
    (   'remove_table',
        Table('aws_malawi1', MetaData(), Column('Cols', INTEGER(), table=<aws_malawi1>, primary_key=True, nullable=False), Column('Element_abbreviation', VARCHAR(length=50), table=<aws_malawi1>), Column('Element_Name', VARCHAR(length=50), table=<aws_malawi1>), Column('Element_Details', VARCHAR(length=50), table=<aws_malawi1>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_malawi1>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_malawi1>), Column('unit', VARCHAR(length=15), table=<aws_malawi1>), Column('lower_limit', VARCHAR(length=50), table=<aws_malawi1>), Column('upper_limit', VARCHAR(length=50), table=<aws_malawi1>), Column('obsv', VARCHAR(length=50), table=<aws_malawi1>), schema=None)),
    (   'remove_table',
        Table('language_translation', MetaData(), Column('tagID', VARCHAR(length=50), table=<language_translation>, primary_key=True, nullable=False), Column('en', VARCHAR(length=100), table=<language_translation>), Column('fr', VARCHAR(length=100), table=<language_translation>), Column('de', VARCHAR(length=100), table=<language_translation>), Column('pt', VARCHAR(length=100), table=<language_translation>), schema=None)),
    (   'remove_index',
        Index('report', Column('reportClass', VARCHAR(length=255), table=<routinereporttransmission>), Column('reportDatetime', DATETIME(), table=<routinereporttransmission>), Column('reportedFrom', VARCHAR(length=255), table=<routinereporttransmission>))),
    (   'remove_table',
        Table('routinereporttransmission', MetaData(), Column('reportClass', VARCHAR(length=255), ForeignKey('routinereportdefinition.reportClass'), table=<routinereporttransmission>), Column('reportDatetime', DATETIME(), table=<routinereporttransmission>), Column('receivedDatetime', DATETIME(), table=<routinereporttransmission>), Column('reportedFrom', VARCHAR(length=255), ForeignKey('station.stationId'), table=<routinereporttransmission>), schema=None)),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_rwanda_rain>, nullable=False))),
    (   'remove_table',
        Table('aws_rwanda_rain', MetaData(), Column('Cols', INTEGER(), table=<aws_rwanda_rain>, nullable=False), Column('Element_abbreviation', VARCHAR(length=20), table=<aws_rwanda_rain>), Column('Element_Name', VARCHAR(length=20), table=<aws_rwanda_rain>), Column('Element_Details', VARCHAR(length=25), table=<aws_rwanda_rain>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_rwanda_rain>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_rwanda_rain>), Column('unit', VARCHAR(length=15), table=<aws_rwanda_rain>), Column('lower_limit', VARCHAR(length=10), table=<aws_rwanda_rain>), Column('upper_limit', VARCHAR(length=10), table=<aws_rwanda_rain>), Column('obsv', VARCHAR(length=25), table=<aws_rwanda_rain>), schema=None)),
    (   'remove_index',
        Index('report_code', Column('reportCode', VARCHAR(length=255), table=<routinereportdefinition>))),
    (   'remove_table',
        Table('routinereportdefinition', MetaData(), Column('reportClass', VARCHAR(length=255), table=<routinereportdefinition>, primary_key=True, nullable=False), Column('reportSchedule', VARCHAR(length=255), table=<routinereportdefinition>), Column('reportCode', VARCHAR(length=255), table=<routinereportdefinition>), Column('reportDescription', VARCHAR(length=255), table=<routinereportdefinition>), schema=None)),
    (   'remove_table',
        Table('qc_interelement_2', MetaData(), Column('stationId_2', VARCHAR(length=50), table=<qc_interelement_2>, primary_key=True, nullable=False), Column('elementId_2', INTEGER(), table=<qc_interelement_2>, primary_key=True, nullable=False), Column('obsDatetime_2', DATETIME(), table=<qc_interelement_2>, primary_key=True, nullable=False), Column('obsValue_2', INTEGER(), table=<qc_interelement_2>), Column('qcStatus_2', INTEGER(), table=<qc_interelement_2>), Column('acquisitionType_2', INTEGER(), table=<qc_interelement_2>), Column('obsLevel_2', VARCHAR(length=50), table=<qc_interelement_2>), Column('capturedBy_2', VARCHAR(length=50), table=<qc_interelement_2>), Column('dataForm_2', VARCHAR(length=50), table=<qc_interelement_2>), schema=None)),
    (   'remove_table',
        Table('form_hourlywind', MetaData(), Column('stationId', VARCHAR(length=255), table=<form_hourlywind>, primary_key=True, nullable=False), Column('yyyy', INTEGER(), table=<form_hourlywind>, primary_key=True, nullable=False), Column('mm', INTEGER(), table=<form_hourlywind>, primary_key=True, nullable=False), Column('dd', INTEGER(), table=<form_hourlywind>, primary_key=True, nullable=False), Column('elem_112_00', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_01', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_02', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_03', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_04', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_05', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_06', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_07', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_08', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_09', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_10', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_11', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_12', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_13', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_14', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_15', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_16', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_17', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_18', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_19', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_20', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_21', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_22', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_112_23', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag00', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag01', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag02', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag03', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag04', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag05', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag06', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag07', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag08', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag09', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag10', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag11', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag12', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag13', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag14', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag15', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag16', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag17', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag18', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag19', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag20', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag21', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag22', VARCHAR(length=255), table=<form_hourlywind>), Column('ddflag23', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_00', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_01', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_02', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_03', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_04', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_05', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_06', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_07', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_08', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_09', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_10', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_11', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_12', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_13', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_14', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_15', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_16', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_17', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_18', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_19', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_20', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_21', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_22', VARCHAR(length=255), table=<form_hourlywind>), Column('elem_111_23', VARCHAR(length=255), table=<form_hourlywind>), Column('total', VARCHAR(length=50), table=<form_hourlywind>), Column('signature', VARCHAR(length=50), table=<form_hourlywind>), Column('entryDatetime', DATETIME(), table=<form_hourlywind>), schema=None)),
    (   'remove_table',
        Table('tm_307092', MetaData(), Column('order', INTEGER(), table=<tm_307092>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1207a9400>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_307092>), Column('CREX_Template', VARCHAR(length=50), table=<tm_307092>), Column('Sequence_Descriptor1', VARCHAR(length=255), table=<tm_307092>), Column('Sequence_Descriptor0', VARCHAR(length=255), table=<tm_307092>), Column('Bufr_Element', VARCHAR(length=255), table=<tm_307092>), Column('Crex_Element', VARCHAR(length=50), table=<tm_307092>), Column('Climsoft_Element', VARCHAR(length=50), table=<tm_307092>), Column('Element_Name', VARCHAR(length=255), table=<tm_307092>), Column('Crex_Unit', VARCHAR(length=25), table=<tm_307092>), Column('Crex_Scale', VARCHAR(length=25), table=<tm_307092>), Column('Crex_DataWidth', VARCHAR(length=25), table=<tm_307092>), Column('Bufr_Unit', VARCHAR(length=255), table=<tm_307092>), Column('Bufr_Scale', INTEGER(), table=<tm_307092>), Column('Bufr_RefValue', BIGINT(), table=<tm_307092>), Column('Bufr_DataWidth_Bits', INTEGER(), table=<tm_307092>), Column('Selected', TINYINT(), table=<tm_307092>), Column('Observation', VARCHAR(length=255), table=<tm_307092>), Column('Crex_Data', VARCHAR(length=30), table=<tm_307092>), Column('Bufr_Data', VARCHAR(length=255), table=<tm_307092>), schema=None)),
    (   'remove_table',
        Table('form_hourly', MetaData(), Column('stationId', VARCHAR(length=50), table=<form_hourly>, primary_key=True, nullable=False), Column('elementId', INTEGER(), table=<form_hourly>, primary_key=True, nullable=False), Column('yyyy', INTEGER(), table=<form_hourly>, primary_key=True, nullable=False), Column('mm', INTEGER(), table=<form_hourly>, primary_key=True, nullable=False), Column('dd', INTEGER(), table=<form_hourly>, primary_key=True, nullable=False), Column('hh_00', VARCHAR(length=50), table=<form_hourly>), Column('hh_01', VARCHAR(length=50), table=<form_hourly>), Column('hh_02', VARCHAR(length=50), table=<form_hourly>), Column('hh_03', VARCHAR(length=50), table=<form_hourly>), Column('hh_04', VARCHAR(length=50), table=<form_hourly>), Column('hh_05', VARCHAR(length=50), table=<form_hourly>), Column('hh_06', VARCHAR(length=50), table=<form_hourly>), Column('hh_07', VARCHAR(length=50), table=<form_hourly>), Column('hh_08', VARCHAR(length=50), table=<form_hourly>), Column('hh_09', VARCHAR(length=50), table=<form_hourly>), Column('hh_10', VARCHAR(length=50), table=<form_hourly>), Column('hh_11', VARCHAR(length=50), table=<form_hourly>), Column('hh_12', VARCHAR(length=50), table=<form_hourly>), Column('hh_13', VARCHAR(length=50), table=<form_hourly>), Column('hh_14', VARCHAR(length=50), table=<form_hourly>), Column('hh_15', VARCHAR(length=50), table=<form_hourly>), Column('hh_16', VARCHAR(length=50), table=<form_hourly>), Column('hh_17', VARCHAR(length=50), table=<form_hourly>), Column('hh_18', VARCHAR(length=50), table=<form_hourly>), Column('hh_19', VARCHAR(length=50), table=<form_hourly>), Column('hh_20', VARCHAR(length=50), table=<form_hourly>), Column('hh_21', VARCHAR(length=50), table=<form_hourly>), Column('hh_22', VARCHAR(length=50), table=<form_hourly>), Column('hh_23', VARCHAR(length=50), table=<form_hourly>), Column('flag00', VARCHAR(length=50), table=<form_hourly>), Column('flag01', VARCHAR(length=50), table=<form_hourly>), Column('flag02', VARCHAR(length=50), table=<form_hourly>), Column('flag03', VARCHAR(length=50), table=<form_hourly>), Column('flag04', VARCHAR(length=50), table=<form_hourly>), Column('flag05', VARCHAR(length=50), table=<form_hourly>), Column('flag06', VARCHAR(length=50), table=<form_hourly>), Column('flag07', VARCHAR(length=50), table=<form_hourly>), Column('flag08', VARCHAR(length=50), table=<form_hourly>), Column('flag09', VARCHAR(length=50), table=<form_hourly>), Column('flag10', VARCHAR(length=50), table=<form_hourly>), Column('flag11', VARCHAR(length=50), table=<form_hourly>), Column('flag12', VARCHAR(length=50), table=<form_hourly>), Column('flag13', VARCHAR(length=50), table=<form_hourly>), Column('flag14', VARCHAR(length=50), table=<form_hourly>), Column('flag15', VARCHAR(length=50), table=<form_hourly>), Column('flag16', VARCHAR(length=50), table=<form_hourly>), Column('flag17', VARCHAR(length=50), table=<form_hourly>), Column('flag18', VARCHAR(length=50), table=<form_hourly>), Column('flag19', VARCHAR(length=50), table=<form_hourly>), Column('flag20', VARCHAR(length=50), table=<form_hourly>), Column('flag21', VARCHAR(length=50), table=<form_hourly>), Column('flag22', VARCHAR(length=50), table=<form_hourly>), Column('flag23', VARCHAR(length=50), table=<form_hourly>), Column('total', VARCHAR(length=50), table=<form_hourly>), Column('signature', VARCHAR(length=50), table=<form_hourly>), Column('entryDatetime', DATETIME(), table=<form_hourly>), schema=None)),
    (   'remove_table',
        Table('tm_307091', MetaData(), Column('Rec', INTEGER(), table=<tm_307091>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1207d0460>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_307091>), Column('CREX_Template', VARCHAR(length=50), table=<tm_307091>), Column('Sequence_Descriptor1', VARCHAR(length=255), table=<tm_307091>), Column('Sequence_Descriptor0', VARCHAR(length=255), table=<tm_307091>), Column('Bufr_Element', VARCHAR(length=255), table=<tm_307091>), Column('Crex_Element', VARCHAR(length=50), table=<tm_307091>), Column('Climsoft_Element', VARCHAR(length=50), table=<tm_307091>), Column('Element_Name', VARCHAR(length=255), table=<tm_307091>), Column('Crex_Unit', VARCHAR(length=25), table=<tm_307091>), Column('Crex_Scale', VARCHAR(length=25), table=<tm_307091>), Column('Crex_DataWidth', VARCHAR(length=25), table=<tm_307091>), Column('Bufr_Unit', VARCHAR(length=255), table=<tm_307091>), Column('Bufr_Scale', INTEGER(), table=<tm_307091>), Column('Bufr_RefValue', BIGINT(), table=<tm_307091>), Column('Bufr_DataWidth_Bits', INTEGER(), table=<tm_307091>), Column('Selected', TINYINT(), table=<tm_307091>), Column('Observation', VARCHAR(length=255), table=<tm_307091>), Column('Crex_Data', VARCHAR(length=30), table=<tm_307091>), Column('Bufr_Data', VARCHAR(length=255), table=<tm_307091>), schema=None)),
    (   'remove_table',
        Table('form_hourly_time_selection', MetaData(), Column('hh', INTEGER(), table=<form_hourly_time_selection>, primary_key=True, nullable=False), Column('hh_selection', TINYINT(), table=<form_hourly_time_selection>), schema=None)),
    (   'remove_table',
        Table('seq_leap_year', MetaData(), Column('yyyy', INTEGER(), table=<seq_leap_year>), schema=None)),
    (   'remove_index',
        Index('stationid_alias_identification', Column('idAlias', VARCHAR(length=255), table=<stationidalias>))),
    (   'remove_index',
        Index('stationstationidAlias', Column('belongsTo', VARCHAR(length=255), table=<stationidalias>))),
    (   'remove_table',
        Table('stationidalias', MetaData(), Column('idAlias', VARCHAR(length=255), table=<stationidalias>), Column('refersTo', VARCHAR(length=255), table=<stationidalias>), Column('belongsTo', VARCHAR(length=255), ForeignKey('station.stationId'), table=<stationidalias>), Column('idAliasBeginDate', VARCHAR(length=50), table=<stationidalias>), Column('idAliasEndDate', VARCHAR(length=50), table=<stationidalias>), schema=None)),
    (   'remove_table',
        Table('seq_month_day_element_leap_yr', MetaData(), Column('mm', INTEGER(), table=<seq_month_day_element_leap_yr>, nullable=False), Column('dd', INTEGER(), table=<seq_month_day_element_leap_yr>, nullable=False), Column('elementId', INTEGER(), table=<seq_month_day_element_leap_yr>, nullable=False), schema=None)),
    (   'remove_table',
        Table('aws_mss', MetaData(), Column('ftpId', VARCHAR(length=50), table=<aws_mss>, primary_key=True, nullable=False), Column('inputFolder', VARCHAR(length=20), table=<aws_mss>, nullable=False), Column('userName', VARCHAR(length=15), table=<aws_mss>, nullable=False), Column('ftpMode', VARCHAR(length=10), table=<aws_mss>), Column('password', VARCHAR(length=15), table=<aws_mss>), schema=None)),
    (   'remove_table',
        Table('seq_hour', MetaData(), Column('hh', INTEGER(), table=<seq_hour>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1207f0be0>, for_update=False)), schema=None)),
    (   'remove_table',
        Table('form_daily2', MetaData(), Column('stationId', VARCHAR(length=50), table=<form_daily2>, primary_key=True, nullable=False), Column('elementId', INTEGER(), table=<form_daily2>, primary_key=True, nullable=False), Column('yyyy', INTEGER(), table=<form_daily2>, primary_key=True, nullable=False), Column('mm', INTEGER(), table=<form_daily2>, primary_key=True, nullable=False), Column('hh', INTEGER(), table=<form_daily2>, primary_key=True, nullable=False), Column('day01', VARCHAR(length=45), table=<form_daily2>), Column('day02', VARCHAR(length=45), table=<form_daily2>), Column('day03', VARCHAR(length=45), table=<form_daily2>), Column('day04', VARCHAR(length=45), table=<form_daily2>), Column('day05', VARCHAR(length=45), table=<form_daily2>), Column('day06', VARCHAR(length=45), table=<form_daily2>), Column('day07', VARCHAR(length=45), table=<form_daily2>), Column('day08', VARCHAR(length=45), table=<form_daily2>), Column('day09', VARCHAR(length=45), table=<form_daily2>), Column('day10', VARCHAR(length=45), table=<form_daily2>), Column('day11', VARCHAR(length=45), table=<form_daily2>), Column('day12', VARCHAR(length=45), table=<form_daily2>), Column('day13', VARCHAR(length=45), table=<form_daily2>), Column('day14', VARCHAR(length=45), table=<form_daily2>), Column('day15', VARCHAR(length=45), table=<form_daily2>), Column('day16', VARCHAR(length=45), table=<form_daily2>), Column('day17', VARCHAR(length=45), table=<form_daily2>), Column('day18', VARCHAR(length=45), table=<form_daily2>), Column('day19', VARCHAR(length=45), table=<form_daily2>), Column('day20', VARCHAR(length=45), table=<form_daily2>), Column('day21', VARCHAR(length=45), table=<form_daily2>), Column('day22', VARCHAR(length=45), table=<form_daily2>), Column('day23', VARCHAR(length=45), table=<form_daily2>), Column('day24', VARCHAR(length=45), table=<form_daily2>), Column('day25', VARCHAR(length=45), table=<form_daily2>), Column('day26', VARCHAR(length=45), table=<form_daily2>), Column('day27', VARCHAR(length=45), table=<form_daily2>), Column('day28', VARCHAR(length=45), table=<form_daily2>), Column('day29', VARCHAR(length=45), table=<form_daily2>), Column('day30', VARCHAR(length=45), table=<form_daily2>), Column('day31', VARCHAR(length=45), table=<form_daily2>), Column('flag01', VARCHAR(length=1), table=<form_daily2>), Column('flag02', VARCHAR(length=1), table=<form_daily2>), Column('flag03', VARCHAR(length=1), table=<form_daily2>), Column('flag04', VARCHAR(length=1), table=<form_daily2>), Column('flag05', VARCHAR(length=1), table=<form_daily2>), Column('flag06', VARCHAR(length=1), table=<form_daily2>), Column('flag07', VARCHAR(length=1), table=<form_daily2>), Column('flag08', VARCHAR(length=1), table=<form_daily2>), Column('flag09', VARCHAR(length=1), table=<form_daily2>), Column('flag10', VARCHAR(length=1), table=<form_daily2>), Column('flag11', VARCHAR(length=1), table=<form_daily2>), Column('flag12', VARCHAR(length=1), table=<form_daily2>), Column('flag13', VARCHAR(length=1), table=<form_daily2>), Column('flag14', VARCHAR(length=1), table=<form_daily2>), Column('flag15', VARCHAR(length=1), table=<form_daily2>), Column('flag16', VARCHAR(length=1), table=<form_daily2>), Column('flag17', VARCHAR(length=1), table=<form_daily2>), Column('flag18', VARCHAR(length=1), table=<form_daily2>), Column('flag19', VARCHAR(length=1), table=<form_daily2>), Column('flag20', VARCHAR(length=1), table=<form_daily2>), Column('flag21', VARCHAR(length=1), table=<form_daily2>), Column('flag22', VARCHAR(length=1), table=<form_daily2>), Column('flag23', VARCHAR(length=1), table=<form_daily2>), Column('flag24', VARCHAR(length=1), table=<form_daily2>), Column('flag25', VARCHAR(length=1), table=<form_daily2>), Column('flag26', VARCHAR(length=1), table=<form_daily2>), Column('flag27', VARCHAR(length=1), table=<form_daily2>), Column('flag28', VARCHAR(length=1), table=<form_daily2>), Column('flag29', VARCHAR(length=1), table=<form_daily2>), Column('flag30', VARCHAR(length=1), table=<form_daily2>), Column('flag31', VARCHAR(length=1), table=<form_daily2>), Column('period01', VARCHAR(length=45), table=<form_daily2>), Column('period02', VARCHAR(length=45), table=<form_daily2>), Column('period03', VARCHAR(length=45), table=<form_daily2>), Column('period04', VARCHAR(length=45), table=<form_daily2>), Column('period05', VARCHAR(length=45), table=<form_daily2>), Column('period06', VARCHAR(length=45), table=<form_daily2>), Column('period07', VARCHAR(length=45), table=<form_daily2>), Column('period08', VARCHAR(length=45), table=<form_daily2>), Column('period09', VARCHAR(length=45), table=<form_daily2>), Column('period10', VARCHAR(length=45), table=<form_daily2>), Column('period11', VARCHAR(length=45), table=<form_daily2>), Column('period12', VARCHAR(length=45), table=<form_daily2>), Column('period13', VARCHAR(length=45), table=<form_daily2>), Column('period14', VARCHAR(length=45), table=<form_daily2>), Column('period15', VARCHAR(length=45), table=<form_daily2>), Column('period16', VARCHAR(length=45), table=<form_daily2>), Column('period17', VARCHAR(length=45), table=<form_daily2>), Column('period18', VARCHAR(length=45), table=<form_daily2>), Column('period19', VARCHAR(length=45), table=<form_daily2>), Column('period20', VARCHAR(length=45), table=<form_daily2>), Column('period21', VARCHAR(length=45), table=<form_daily2>), Column('period22', VARCHAR(length=45), table=<form_daily2>), Column('period23', VARCHAR(length=45), table=<form_daily2>), Column('period24', VARCHAR(length=45), table=<form_daily2>), Column('period25', VARCHAR(length=45), table=<form_daily2>), Column('period26', VARCHAR(length=45), table=<form_daily2>), Column('period27', VARCHAR(length=45), table=<form_daily2>), Column('period28', VARCHAR(length=45), table=<form_daily2>), Column('period29', VARCHAR(length=45), table=<form_daily2>), Column('period30', VARCHAR(length=45), table=<form_daily2>), Column('period31', VARCHAR(length=45), table=<form_daily2>), Column('total', VARCHAR(length=45), table=<form_daily2>), Column('signature', VARCHAR(length=45), table=<form_daily2>), Column('entryDatetime', DATETIME(), table=<form_daily2>), Column('temperatureUnits', VARCHAR(length=45), table=<form_daily2>), Column('precipUnits', VARCHAR(length=45), table=<form_daily2>), Column('cloudHeightUnits', VARCHAR(length=45), table=<form_daily2>), Column('visUnits', VARCHAR(length=45), table=<form_daily2>), schema=None)),
    (   'remove_index',
        Index('identification', Column('No', INTEGER(), table=<aws_malawi12>, nullable=False))),
    (   'remove_table',
        Table('aws_malawi12', MetaData(), Column('No', INTEGER(), table=<aws_malawi12>, nullable=False), Column('Element_abbreviation', TEXT(), table=<aws_malawi12>, nullable=False), Column('Element_Name', TEXT(), table=<aws_malawi12>, nullable=False), Column('Element_Details', TEXT(), table=<aws_malawi12>, nullable=False), Column('Climsoft_Element', TEXT(), table=<aws_malawi12>, nullable=False), Column('Bufr_Element', TEXT(), table=<aws_malawi12>, nullable=False), Column('unit', TEXT(), table=<aws_malawi12>, nullable=False), Column('lower_limit', TEXT(), table=<aws_malawi12>, nullable=False), Column('upper_limit', TEXT(), table=<aws_malawi12>, nullable=False), Column('obsv', TEXT(), table=<aws_malawi12>, nullable=False), schema=None)),
    (   'remove_index',
        Index('ftpId_UNIQUE', Column('ftpId', VARCHAR(length=50), table=<aws_basestation>, primary_key=True, nullable=False))),
    (   'remove_table',
        Table('aws_basestation', MetaData(), Column('ftpId', VARCHAR(length=50), table=<aws_basestation>, primary_key=True, nullable=False), Column('inputFolder', VARCHAR(length=20), table=<aws_basestation>, nullable=False), Column('ftpMode', VARCHAR(length=10), table=<aws_basestation>), Column('userName', VARCHAR(length=15), table=<aws_basestation>, nullable=False), Column('password', VARCHAR(length=15), table=<aws_basestation>, nullable=False), schema=None)),
    (   'remove_table',
        Table('aws_elements', MetaData(), Column('aws_element', VARCHAR(length=50), table=<aws_elements>, primary_key=True, nullable=False), Column('climsoft_element', VARCHAR(length=50), table=<aws_elements>, primary_key=True, nullable=False), Column('element_description', VARCHAR(length=50), table=<aws_elements>, nullable=False), schema=None)),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_test>, nullable=False))),
    (   'remove_table',
        Table('aws_test', MetaData(), Column('Cols', INTEGER(), table=<aws_test>, nullable=False), Column('Element_abbreviation', VARCHAR(length=20), table=<aws_test>), Column('Element_Name', VARCHAR(length=20), table=<aws_test>), Column('Element_Details', VARCHAR(length=25), table=<aws_test>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_test>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_test>), Column('unit', VARCHAR(length=15), table=<aws_test>), Column('lower_limit', VARCHAR(length=10), table=<aws_test>), Column('upper_limit', VARCHAR(length=10), table=<aws_test>), Column('obsv', VARCHAR(length=25), table=<aws_test>), schema=None)),
    (   'remove_table',
        Table('form_monthly', MetaData(), Column('stationId', VARCHAR(length=255), table=<form_monthly>, primary_key=True, nullable=False), Column('elementId', INTEGER(), table=<form_monthly>, primary_key=True, nullable=False), Column('yyyy', INTEGER(), table=<form_monthly>, primary_key=True, nullable=False), Column('mm_01', VARCHAR(length=255), table=<form_monthly>), Column('mm_02', VARCHAR(length=255), table=<form_monthly>), Column('mm_03', VARCHAR(length=255), table=<form_monthly>), Column('mm_04', VARCHAR(length=255), table=<form_monthly>, nullable=False), Column('mm_05', VARCHAR(length=255), table=<form_monthly>), Column('mm_06', VARCHAR(length=255), table=<form_monthly>), Column('mm_07', VARCHAR(length=255), table=<form_monthly>), Column('mm_08', VARCHAR(length=255), table=<form_monthly>), Column('mm_09', VARCHAR(length=255), table=<form_monthly>), Column('mm_10', VARCHAR(length=255), table=<form_monthly>), Column('mm_11', VARCHAR(length=255), table=<form_monthly>), Column('mm_12', VARCHAR(length=255), table=<form_monthly>), Column('flag01', VARCHAR(length=255), table=<form_monthly>), Column('flag02', VARCHAR(length=255), table=<form_monthly>), Column('flag03', VARCHAR(length=255), table=<form_monthly>), Column('flag04', VARCHAR(length=255), table=<form_monthly>), Column('flag05', VARCHAR(length=255), table=<form_monthly>), Column('flag06', VARCHAR(length=255), table=<form_monthly>), Column('flag07', VARCHAR(length=255), table=<form_monthly>), Column('flag08', VARCHAR(length=255), table=<form_monthly>), Column('flag09', VARCHAR(length=255), table=<form_monthly>), Column('flag10', VARCHAR(length=255), table=<form_monthly>), Column('flag11', VARCHAR(length=255), table=<form_monthly>), Column('flag12', VARCHAR(length=255), table=<form_monthly>), Column('period01', VARCHAR(length=255), table=<form_monthly>), Column('period02', VARCHAR(length=255), table=<form_monthly>), Column('period03', VARCHAR(length=255), table=<form_monthly>), Column('period04', VARCHAR(length=255), table=<form_monthly>), Column('period05', VARCHAR(length=255), table=<form_monthly>), Column('period06', VARCHAR(length=255), table=<form_monthly>), Column('period07', VARCHAR(length=255), table=<form_monthly>), Column('period08', VARCHAR(length=255), table=<form_monthly>), Column('period09', VARCHAR(length=255), table=<form_monthly>), Column('period10', VARCHAR(length=255), table=<form_monthly>), Column('period11', VARCHAR(length=255), table=<form_monthly>), Column('period12', VARCHAR(length=255), table=<form_monthly>), Column('signature', VARCHAR(length=50), table=<form_monthly>), Column('entryDatetime', DATETIME(), table=<form_monthly>), schema=None)),
    (   'remove_table',
        Table('ccitt', MetaData(), Column('Characters', VARCHAR(length=25), table=<ccitt>), Column('MostSignificant', INTEGER(), table=<ccitt>), Column('LeastSignificant', INTEGER(), table=<ccitt>), schema=None)),
    (   'remove_index',
        Index('Identification', Column('stationId', VARCHAR(length=10), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('yyyy', BIGINT(), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('mm', BIGINT(), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('dd', BIGINT(), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('hh', VARCHAR(length=5), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False))),
    (   'remove_table',
        Table('form_synoptic2_tdcf', MetaData(), Column('stationId', VARCHAR(length=10), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('yyyy', BIGINT(), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('mm', BIGINT(), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('dd', BIGINT(), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('hh', VARCHAR(length=5), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('106', VARCHAR(length=6), table=<form_synoptic2_tdcf>), Column('107', VARCHAR(length=6), table=<form_synoptic2_tdcf>), Column('399', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('301', VARCHAR(length=8), table=<form_synoptic2_tdcf>), Column('185', VARCHAR(length=6), table=<form_synoptic2_tdcf>), Column('101', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('103', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('105', VARCHAR(length=50), table=<form_synoptic2_tdcf>), Column('110', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('114', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('115', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('168', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('192', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('169', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('170', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('171', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('119', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('116', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('117', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('118', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('123', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('120', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('121', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('122', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('127', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('124', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('125', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('126', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('131', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('128', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('129', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('130', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('167', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('197', VARCHAR(length=50), table=<form_synoptic2_tdcf>), Column('193', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('18', VARCHAR(length=6), table=<form_synoptic2_tdcf>), Column('532', VARCHAR(length=6), table=<form_synoptic2_tdcf>), Column('132', VARCHAR(length=6), table=<form_synoptic2_tdcf>), Column('5', VARCHAR(length=6), table=<form_synoptic2_tdcf>), Column('174', VARCHAR(length=50), table=<form_synoptic2_tdcf>), Column('3', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('2', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('112', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('111', VARCHAR(length=5), table=<form_synoptic2_tdcf>), Column('85', VARCHAR(length=50), table=<form_synoptic2_tdcf>), Column('flag1', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag2', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag3', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag4', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag5', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag6', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag7', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag8', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag9', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag10', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag11', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag12', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag13', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag14', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag15', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag16', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag17', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag18', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag19', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag20', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag21', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag22', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag23', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag24', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag25', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag26', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag27', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag28', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag29', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag30', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag31', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag32', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag33', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag34', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag35', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag36', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag37', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag38', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag39', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag40', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag41', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag42', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag43', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag44', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('flag45', VARCHAR(length=1), table=<form_synoptic2_tdcf>), Column('signature', VARCHAR(length=50), table=<form_synoptic2_tdcf>), Column('entryDatetime', DATETIME(), table=<form_synoptic2_tdcf>), schema=None)),
    (   'remove_table',
        Table('seq_daily_element', MetaData(), Column('seq', BIGINT(), table=<seq_daily_element>, primary_key=True, nullable=False), Column('elementId', BIGINT(), table=<seq_daily_element>, nullable=False), schema=None)),
    (   'remove_table',
        Table('seq_day', MetaData(), Column('dd', VARCHAR(length=50), table=<seq_day>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208995e0>, for_update=False)), schema=None)),
    (   'remove_table',
        Table('userrecords', MetaData(), Column('username', VARCHAR(length=255), table=<userrecords>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x120899fd0>, for_update=False)), Column('recsexpt', INTEGER(), table=<userrecords>), Column('recsdone', VARCHAR(length=255), table=<userrecords>), schema=None)),
    (   'remove_table',
        Table('seq_monthly_element', MetaData(), Column('seq', BIGINT(), table=<seq_monthly_element>, primary_key=True, nullable=False), Column('elementId', BIGINT(), table=<seq_monthly_element>, nullable=False), schema=None)),
    (   'remove_table',
        Table('qc_interelement_1', MetaData(), Column('stationId_1', VARCHAR(length=50), table=<qc_interelement_1>, primary_key=True, nullable=False), Column('elementId_1', INTEGER(), table=<qc_interelement_1>, primary_key=True, nullable=False), Column('obsDatetime_1', DATETIME(), table=<qc_interelement_1>, primary_key=True, nullable=False), Column('obsValue_1', INTEGER(), table=<qc_interelement_1>), Column('qcStatus_1', INTEGER(), table=<qc_interelement_1>), Column('acquisitionType_1', INTEGER(), table=<qc_interelement_1>), Column('obsLevel_1', VARCHAR(length=50), table=<qc_interelement_1>), Column('capturedBy_1', VARCHAR(length=50), table=<qc_interelement_1>), Column('dataForm_1', VARCHAR(length=50), table=<qc_interelement_1>), schema=None)),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<ss>, nullable=False))),
    (   'remove_table',
        Table('ss', MetaData(), Column('Cols', INTEGER(), table=<ss>, nullable=False), Column('Element_abbreviation', VARCHAR(length=20), table=<ss>), Column('Element_Name', VARCHAR(length=20), table=<ss>), Column('Element_Details', VARCHAR(length=25), table=<ss>), Column('Climsoft_Element', VARCHAR(length=6), table=<ss>), Column('Bufr_Element', VARCHAR(length=6), table=<ss>), Column('unit', VARCHAR(length=15), table=<ss>), Column('lower_limit', VARCHAR(length=10), table=<ss>), Column('upper_limit', VARCHAR(length=10), table=<ss>), Column('obsv', VARCHAR(length=25), table=<ss>), schema=None)),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<testing_aws1>, nullable=False))),
    (   'remove_table',
        Table('testing_aws1', MetaData(), Column('Cols', INTEGER(), table=<testing_aws1>, nullable=False), Column('Element_abbreviation', VARCHAR(length=20), table=<testing_aws1>), Column('Element_Name', VARCHAR(length=20), table=<testing_aws1>), Column('Element_Details', VARCHAR(length=25), table=<testing_aws1>), Column('Climsoft_Element', VARCHAR(length=6), table=<testing_aws1>), Column('Bufr_Element', VARCHAR(length=6), table=<testing_aws1>), Column('unit', VARCHAR(length=15), table=<testing_aws1>), Column('lower_limit', VARCHAR(length=10), table=<testing_aws1>), Column('upper_limit', VARCHAR(length=10), table=<testing_aws1>), Column('obsv', VARCHAR(length=25), table=<testing_aws1>), schema=None)),
    (   'remove_table',
        Table('aws_process_parameters', MetaData(), Column('RetrieveInterval', INTEGER(), table=<aws_process_parameters>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208b9c40>, for_update=False)), Column('HourOffset', INTEGER(), table=<aws_process_parameters>, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208b9fa0>, for_update=False)), Column('RetrievePeriod', INTEGER(), table=<aws_process_parameters>, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208c20d0>, for_update=False)), Column('RetrieveTimeout', INTEGER(), table=<aws_process_parameters>, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208c21c0>, for_update=False)), Column('DelinputFile', TINYINT(), table=<aws_process_parameters>, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208c22b0>, for_update=False)), Column('UTCDiff', TINYINT(display_width=2, unsigned=True, zerofill=True), table=<aws_process_parameters>, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208c23a0>, for_update=False)), schema=None)),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_rema1>, nullable=False))),
    (   'remove_table',
        Table('aws_rema1', MetaData(), Column('Cols', INTEGER(), table=<aws_rema1>, nullable=False), Column('Element_abbreviation', VARCHAR(length=20), table=<aws_rema1>), Column('Element_Name', VARCHAR(length=20), table=<aws_rema1>), Column('Element_Details', VARCHAR(length=25), table=<aws_rema1>), Column('Climsoft_Element', VARCHAR(length=6), table=<aws_rema1>), Column('Bufr_Element', VARCHAR(length=6), table=<aws_rema1>), Column('unit', VARCHAR(length=15), table=<aws_rema1>), Column('lower_limit', VARCHAR(length=10), table=<aws_rema1>), Column('upper_limit', VARCHAR(length=10), table=<aws_rema1>), Column('obsv', VARCHAR(length=25), table=<aws_rema1>), schema=None)),
    (   'remove_table',
        Table('gaps', MetaData(), Column('Missing_STNID', VARCHAR(length=255), table=<gaps>, primary_key=True, nullable=False), Column('Missing_ELEM', BIGINT(), table=<gaps>, primary_key=True, nullable=False), Column('Missing_Date', DATE(), table=<gaps>, primary_key=True, nullable=False), schema=None)),
    (   'remove_table',
        Table('flagtable', MetaData(), Column('Bufr_Descriptor', VARCHAR(length=6), table=<flagtable>, primary_key=True, nullable=False), Column('Crex_Descriptor', VARCHAR(length=6), table=<flagtable>), Column('Details', VARCHAR(length=255), table=<flagtable>), Column('Widths', INTEGER(), table=<flagtable>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208d2730>, for_update=False)), Column('Missing', INTEGER(), table=<flagtable>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208d2820>, for_update=False)), schema=None)),
    (   'remove_table',
        Table('tm_307083', MetaData(), Column('order', INTEGER(), table=<tm_307083>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208da190>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_307083>), Column('Crex_Template', VARCHAR(length=50), table=<tm_307083>), Column('Sequence_Descriptor1', VARCHAR(length=50), table=<tm_307083>), Column('Sequence_Descriptor0', VARCHAR(length=50), table=<tm_307083>), Column('Bufr_Element', VARCHAR(length=50), table=<tm_307083>), Column('Crex_Element', VARCHAR(length=255), table=<tm_307083>), Column('Climsoft_Element', VARCHAR(length=255), table=<tm_307083>), Column('Element_Name', VARCHAR(length=255), table=<tm_307083>), Column('synop_code', VARCHAR(length=255), table=<tm_307083>), Column('unit', VARCHAR(length=255), table=<tm_307083>), Column('scale', VARCHAR(length=255), table=<tm_307083>), Column('width', VARCHAR(length=255), table=<tm_307083>), Column('units_length_scale', VARCHAR(length=255), table=<tm_307083>), Column('data_type', VARCHAR(length=255), table=<tm_307083>), Column('selected', TINYINT(), table=<tm_307083>), Column('observation', VARCHAR(length=255), table=<tm_307083>), Column('Crex_Data', VARCHAR(length=255), table=<tm_307083>), Column('Bufr_Data', VARCHAR(length=50), table=<tm_307083>), schema=None)),
    (   'remove_index',
        Index('identification', Column('No', INTEGER(), table=<abc>, nullable=False))),
    (   'remove_table',
        Table('abc', MetaData(), Column('No', INTEGER(), table=<abc>, nullable=False), Column('Element_abbreviation', TEXT(), table=<abc>, nullable=False), Column('Element_Name', TEXT(), table=<abc>, nullable=False), Column('Element_Details', TEXT(), table=<abc>, nullable=False), Column('Climsoft_Element', TEXT(), table=<abc>, nullable=False), Column('Bufr_Element', TEXT(), table=<abc>, nullable=False), Column('unit', TEXT(), table=<abc>, nullable=False), Column('lower_limit', TEXT(), table=<abc>, nullable=False), Column('upper_limit', TEXT(), table=<abc>, nullable=False), Column('obsv', TEXT(), table=<abc>, nullable=False), schema=None)),
    (   'remove_table',
        Table('code_flag', MetaData(), Column('FXY', VARCHAR(length=255), table=<code_flag>, primary_key=True, nullable=False), Column('Fxyyy', VARCHAR(length=50), table=<code_flag>), Column('Description', VARCHAR(length=255), table=<code_flag>), Column('Bufr_DataWidth_Bits', INTEGER(), table=<code_flag>), Column('Crex_DataWidth', VARCHAR(length=25), table=<code_flag>), Column('Bufr_Unit', VARCHAR(length=255), table=<code_flag>), Column('Bufr_Value', VARCHAR(length=50), table=<code_flag>), Column('Crex_Value', VARCHAR(length=10), table=<code_flag>), schema=None)),
    (   'remove_table',
        Table('tdcf_indicators', MetaData(), Column('CREX_Edition', INTEGER(), table=<tdcf_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208efc70>, for_update=False)), Column('CREX_Table', INTEGER(), table=<tdcf_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208f8070>, for_update=False)), Column('BUFR_Table', INTEGER(), table=<tdcf_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208f8160>, for_update=False)), Column('Local_Table', INTEGER(), table=<tdcf_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208f8250>, for_update=False)), Column('Data_Category', INTEGER(), table=<tdcf_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208f8340>, for_update=False)), Column('Data_SubCategory', INTEGER(), table=<tdcf_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208f8430>, for_update=False)), Column('Originating_Centre', INTEGER(), table=<tdcf_indicators>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208f8520>, for_update=False)), schema=None)),
    (   'remove_table',
        Table('form_agro1', MetaData(), Column('stationId', VARCHAR(length=50), table=<form_agro1>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1208f8e20>, for_update=False)), Column('yyyy', INTEGER(), table=<form_agro1>, primary_key=True, nullable=False), Column('mm', INTEGER(), table=<form_agro1>, primary_key=True, nullable=False), Column('dd', INTEGER(), table=<form_agro1>, primary_key=True, nullable=False), Column('Val_Elem101', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem102', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem103', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem105', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem002', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem003', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem099', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem072', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem073', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem074', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem554', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem075', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem076', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem561', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem562', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem563', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem513', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem005', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem504', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem532', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem137', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem018', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem518', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem511', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem512', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem503', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem515', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem564', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem565', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem566', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem531', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem530', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem541', VARCHAR(length=6), table=<form_agro1>), Column('Val_Elem542', VARCHAR(length=6), table=<form_agro1>), Column('Flag101', VARCHAR(length=1), table=<form_agro1>), Column('Flag102', VARCHAR(length=1), table=<form_agro1>), Column('Flag103', VARCHAR(length=1), table=<form_agro1>), Column('Flag105', VARCHAR(length=1), table=<form_agro1>), Column('Flag002', VARCHAR(length=1), table=<form_agro1>), Column('Flag003', VARCHAR(length=1), table=<form_agro1>), Column('Flag099', VARCHAR(length=1), table=<form_agro1>), Column('Flag072', VARCHAR(length=1), table=<form_agro1>), Column('Flag073', VARCHAR(length=1), table=<form_agro1>), Column('Flag074', VARCHAR(length=1), table=<form_agro1>), Column('Flag554', VARCHAR(length=1), table=<form_agro1>), Column('Flag075', VARCHAR(length=1), table=<form_agro1>), Column('Flag076', VARCHAR(length=1), table=<form_agro1>), Column('Flag561', VARCHAR(length=1), table=<form_agro1>), Column('Flag562', VARCHAR(length=1), table=<form_agro1>), Column('Flag563', VARCHAR(length=1), table=<form_agro1>), Column('Flag513', VARCHAR(length=1), table=<form_agro1>), Column('Flag005', VARCHAR(length=1), table=<form_agro1>), Column('Flag504', VARCHAR(length=1), table=<form_agro1>), Column('Flag532', VARCHAR(length=1), table=<form_agro1>), Column('Flag137', VARCHAR(length=1), table=<form_agro1>), Column('Flag018', VARCHAR(length=1), table=<form_agro1>), Column('Flag518', VARCHAR(length=1), table=<form_agro1>), Column('Flag511', VARCHAR(length=1), table=<form_agro1>), Column('Flag512', VARCHAR(length=1), table=<form_agro1>), Column('Flag503', VARCHAR(length=1), table=<form_agro1>), Column('Flag515', VARCHAR(length=1), table=<form_agro1>), Column('Flag564', VARCHAR(length=1), table=<form_agro1>), Column('Flag565', VARCHAR(length=1), table=<form_agro1>), Column('Flag566', VARCHAR(length=1), table=<form_agro1>), Column('Flag531', VARCHAR(length=1), table=<form_agro1>), Column('Flag530', VARCHAR(length=1), table=<form_agro1>), Column('Flag541', VARCHAR(length=1), table=<form_agro1>), Column('Flag542', VARCHAR(length=1), table=<form_agro1>), Column('signature', VARCHAR(length=45), table=<form_agro1>), Column('entryDatetime', DATETIME(), table=<form_agro1>), schema=None)),
    (   'remove_table',
        Table('bufr_crex_master', MetaData(), Column('Bufr_FXY', VARCHAR(length=6), table=<bufr_crex_master>, nullable=False), Column('Crex_Fxxyyy', VARCHAR(length=255), table=<bufr_crex_master>, nullable=False), Column('ElementName', VARCHAR(length=255), table=<bufr_crex_master>, nullable=False), Column('Bufr_Unit', VARCHAR(length=255), table=<bufr_crex_master>, nullable=False), Column('Bufr_Scale', VARCHAR(length=25), table=<bufr_crex_master>, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1209217c0>, for_update=False)), Column('Bufr_RefValue', VARCHAR(length=50), table=<bufr_crex_master>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1209218b0>, for_update=False)), Column('Bufr_DataWidth_Bits', VARCHAR(length=50), table=<bufr_crex_master>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1209219a0>, for_update=False)), Column('Crex_Unit', VARCHAR(length=25), table=<bufr_crex_master>), Column('Crex_Scale', VARCHAR(length=25), table=<bufr_crex_master>, nullable=False), Column('Crex_DataWidth', VARCHAR(length=25), table=<bufr_crex_master>, nullable=False), Column('Observation', VARCHAR(length=255), table=<bufr_crex_master>), Column('Crex_Data', VARCHAR(length=30), table=<bufr_crex_master>), Column('Bufr_Data', VARCHAR(length=255), table=<bufr_crex_master>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x120921b80>, for_update=False)), schema=None)),
    (   'remove_index',
        Index('strID_UNIQUE', Column('strID', INTEGER(), table=<aws_structures>, nullable=False))),
    (   'remove_table',
        Table('aws_structures', MetaData(), Column('strID', INTEGER(), table=<aws_structures>, nullable=False), Column('strName', VARCHAR(length=20), table=<aws_structures>, primary_key=True, nullable=False), Column('data_delimiter', VARCHAR(length=10), table=<aws_structures>, nullable=False), Column('hdrRows', INTEGER(), table=<aws_structures>, nullable=False), Column('txtQualifier', VARCHAR(length=5), table=<aws_structures>), schema=None)),
    (   'remove_table',
        Table('tm_307082', MetaData(), Column('order', INTEGER(), table=<tm_307082>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x120932130>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<tm_307082>), Column('Crex_Template', VARCHAR(length=50), table=<tm_307082>), Column('Sequence_Descriptor1', VARCHAR(length=50), table=<tm_307082>), Column('Sequence_Descriptor0', VARCHAR(length=50), table=<tm_307082>), Column('Bufr_Element', VARCHAR(length=50), table=<tm_307082>), Column('Crex_Element', VARCHAR(length=255), table=<tm_307082>), Column('Climsoft_Element', VARCHAR(length=255), table=<tm_307082>), Column('Element_Name', VARCHAR(length=255), table=<tm_307082>), Column('synop_code', VARCHAR(length=255), table=<tm_307082>), Column('unit', VARCHAR(length=255), table=<tm_307082>), Column('scale', VARCHAR(length=255), table=<tm_307082>), Column('width', VARCHAR(length=255), table=<tm_307082>), Column('units_length_scale', VARCHAR(length=255), table=<tm_307082>), Column('data_type', VARCHAR(length=255), table=<tm_307082>), Column('selected', TINYINT(), table=<tm_307082>), Column('observation', VARCHAR(length=255), table=<tm_307082>), Column('Crex_Data', VARCHAR(length=255), table=<tm_307082>), Column('Bufr_Data', VARCHAR(length=50), table=<tm_307082>), schema=None)),
    (   'remove_table',
        Table('missing_stats', MetaData(), Column('STN_ID', VARCHAR(length=255), table=<missing_stats>, primary_key=True, nullable=False), Column('ELEM', BIGINT(), table=<missing_stats>, primary_key=True, nullable=False), Column('MISSING', BIGINT(), table=<missing_stats>), Column('Closing_Date', DATE(), table=<missing_stats>, primary_key=True, nullable=False), Column('Opening_Date', DATE(), table=<missing_stats>, primary_key=True, nullable=False), schema=None)),
    (   'remove_index',
        Index('SiteID_UNIQUE', Column('SiteID', VARCHAR(length=20), table=<aws_sites>, primary_key=True, nullable=False))),
    (   'remove_table',
        Table('aws_sites', MetaData(), Column('SiteID', VARCHAR(length=20), table=<aws_sites>, primary_key=True, nullable=False), Column('SiteName', VARCHAR(length=50), table=<aws_sites>), Column('InputFile', VARCHAR(length=255), table=<aws_sites>), Column('FilePrefix', VARCHAR(length=50), table=<aws_sites>), Column('chkPrefix', TINYINT(display_width=1), table=<aws_sites>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x120947130>, for_update=False)), Column('DataStructure', VARCHAR(length=50), table=<aws_sites>), Column('MissingDataFlag', VARCHAR(length=20), table=<aws_sites>), Column('awsServerIP', VARCHAR(length=50), table=<aws_sites>), Column('OperationalStatus', TINYINT(display_width=1, unsigned=True, zerofill=True), table=<aws_sites>, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1209472b0>, for_update=False)), Column('GTSEncode', TINYINT(display_width=1), table=<aws_sites>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1209473a0>, for_update=False)), Column('GTSHeader', VARCHAR(length=20), table=<aws_sites>), schema=None)),
    (   'remove_table',
        Table('form_synoptic_2_ra1', MetaData(), Column('stationId', VARCHAR(length=50), table=<form_synoptic_2_ra1>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x120947e50>, for_update=False)), Column('yyyy', INTEGER(), table=<form_synoptic_2_ra1>, primary_key=True, nullable=False), Column('mm', INTEGER(), table=<form_synoptic_2_ra1>, primary_key=True, nullable=False), Column('dd', INTEGER(), table=<form_synoptic_2_ra1>, primary_key=True, nullable=False), Column('hh', INTEGER(), table=<form_synoptic_2_ra1>, primary_key=True, nullable=False), Column('Val_Elem106', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem107', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem400', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem814', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem399', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem301', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem185', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem101', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem102', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem103', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem105', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem192', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem110', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem114', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem112', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem111', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem167', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem197', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem193', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem115', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem168', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem169', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem170', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem171', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem119', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem116', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem117', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem118', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem123', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem120', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem121', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem122', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem127', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem124', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem125', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem126', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem131', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem128', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem129', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem130', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem002', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem003', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem099', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem018', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem084', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem132', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem005', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem174', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Val_Elem046', VARCHAR(length=6), table=<form_synoptic_2_ra1>), Column('Flag106', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag107', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag400', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag814', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag399', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag301', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag185', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag101', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag102', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag103', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag105', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag192', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag110', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag114', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag112', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag111', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag167', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag197', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag193', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag115', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag168', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag169', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag170', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag171', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag119', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag116', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag117', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag118', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag123', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag120', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag121', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag122', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag127', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag124', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag125', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag126', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag131', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag128', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag129', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag130', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag002', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag003', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag099', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag018', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag084', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag132', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag005', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag174', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('Flag046', VARCHAR(length=1), table=<form_synoptic_2_ra1>), Column('signature', VARCHAR(length=45), table=<form_synoptic_2_ra1>), Column('entryDatetime', DATETIME(), table=<form_synoptic_2_ra1>), schema=None)),
    (   'remove_index',
        Index('identification', Column('seq', BIGINT(), table=<seq_element>, primary_key=True, nullable=False))),
    (   'remove_table',
        Table('seq_element', MetaData(), Column('seq', BIGINT(), table=<seq_element>, primary_key=True, nullable=False), Column('element_code', VARCHAR(length=50), table=<seq_element>), schema=None)),
    (   'remove_table',
        Table('bufr_crex_data', MetaData(), Column('nos', INTEGER(), table=<bufr_crex_data>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x120981bb0>, for_update=False)), Column('Bufr_Template', VARCHAR(length=50), table=<bufr_crex_data>), Column('Crex_Template', VARCHAR(length=50), table=<bufr_crex_data>), Column('Sequence_Descriptor1', VARCHAR(length=50), table=<bufr_crex_data>), Column('Sequence_Descriptor0', VARCHAR(length=50), table=<bufr_crex_data>), Column('Bufr_Element', VARCHAR(length=50), table=<bufr_crex_data>), Column('Crex_Element', VARCHAR(length=255), table=<bufr_crex_data>), Column('Climsoft_Element', VARCHAR(length=255), table=<bufr_crex_data>), Column('Element_Name', VARCHAR(length=255), table=<bufr_crex_data>), Column('Crex_Unit', VARCHAR(length=25), table=<bufr_crex_data>), Column('Crex_Scale', VARCHAR(length=25), table=<bufr_crex_data>, nullable=False), Column('Crex_DataWidth', VARCHAR(length=25), table=<bufr_crex_data>, nullable=False), Column('Bufr_Unit', VARCHAR(length=255), table=<bufr_crex_data>, nullable=False), Column('Bufr_Scale', VARCHAR(length=25), table=<bufr_crex_data>, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x12098b370>, for_update=False)), Column('Bufr_RefValue', VARCHAR(length=50), table=<bufr_crex_data>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x12098b460>, for_update=False)), Column('Bufr_DataWidth_Bits', VARCHAR(length=50), table=<bufr_crex_data>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x12098b550>, for_update=False)), Column('selected', TINYINT(), table=<bufr_crex_data>), Column('Observation', VARCHAR(length=255), table=<bufr_crex_data>), Column('Crex_Data', VARCHAR(length=30), table=<bufr_crex_data>), Column('Bufr_Data', VARCHAR(length=255), table=<bufr_crex_data>, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x12098b6d0>, for_update=False)), schema=None)),
    (   'add_index',
        Index('ix_faultresolution_associatedWith', Column('associatedWith', BigInteger(), table=<faultresolution>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('code', Column('instrumentId', VARCHAR(length=255), table=<instrument>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_instrument_installedAt', Column('installedAt', String(length=255), table=<instrument>))),
    (   'add_index',
        Index('ix_instrument_instrumentId', Column('instrumentId', String(length=255), table=<instrument>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('report_id', Column('reportId', BIGINT(), table=<instrumentfaultreport>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_instrumentfaultreport_reportId', Column('reportId', BigInteger(), table=<instrumentfaultreport>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_instrumentfaultreport_reportedFrom', Column('reportedFrom', String(length=255), table=<instrumentfaultreport>))),
    (   'add_index',
        Index('ix_instrumentinspection_performedAt', Column('performedAt', String(length=255), table=<instrumentinspection>))),
    (   'remove_index',
        Index('elementCode', Column('elementId', BIGINT(), table=<obselement>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x12069b820>, for_update=False)))),
    (   'add_index',
        Index('ix_obselement_elementId', Column('elementId', BigInteger(), table=<obselement>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x11fd03eb0>, for_update=False)))),
    (   'remove_index',
        Index('obsElementObservationInitial', Column('describedBy', BIGINT(), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationObservationInitial', Column('recordedFrom', VARCHAR(length=255), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationfinal_describedBy', Column('describedBy', BigInteger(), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationfinal_recordedFrom', Column('recordedFrom', String(length=255), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('obsElementObservationInitial', Column('describedBy', BIGINT(), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationObservationInitial', Column('recordedFrom', VARCHAR(length=255), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationinitial_describedBy', Column('describedBy', BigInteger(), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationinitial_recordedFrom', Column('recordedFrom', String(length=255), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('scheduleClassIdeification', Column('scheduleClass', VARCHAR(length=255), table=<obsscheduleclass>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1206d8a00>, for_update=False)))),
    (   'add_index',
        Index('ix_obsscheduleclass_refersTo', Column('refersTo', String(length=255), table=<obsscheduleclass>))),
    (   'add_index',
        Index('ix_obsscheduleclass_scheduleClass', Column('scheduleClass', String(length=255), table=<obsscheduleclass>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x11fd90910>, for_update=False)))),
    (   'add_index',
        Index('ix_paperarchive_classifiedInto', Column('classifiedInto', String(length=50), table=<paperarchive>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('paperarchivedef', Column('formId', VARCHAR(length=50), table=<paperarchivedefinition>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_paperarchivedefinition_formId', Column('formId', String(length=50), table=<paperarchivedefinition>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('physicalFeatureidentification_idx', Column('classifiedInto', VARCHAR(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationfeature', Column('associatedWith', VARCHAR(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeature_associatedWith', Column('associatedWith', String(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeature_classifiedInto', Column('classifiedInto', String(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationFeatureClass', Column('featureClass', VARCHAR(length=255), table=<physicalfeatureclass>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeatureclass_featureClass', Column('featureClass', String(length=255), table=<physicalfeatureclass>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeatureclass_refersTo', Column('refersTo', String(length=255), table=<physicalfeatureclass>))),
    (   'remove_index',
        Index('StationStationId', Column('stationId', VARCHAR(length=255), table=<station>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_station_stationId', Column('stationId', String(length=255), table=<station>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('obsElementobservationInitial', Column('describedBy', BIGINT(), table=<stationelement>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationobservationInitial', Column('recordedFrom', VARCHAR(length=255), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_describedBy', Column('describedBy', BigInteger(), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_recordedFrom', Column('recordedFrom', String(length=255), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_recordedWith', Column('recordedWith', String(length=255), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_scheduledFor', Column('scheduledFor', String(length=255), table=<stationelement>))),
    (   'remove_index',
        Index('stationQualifierIdentification', Column('belongsTo', VARCHAR(length=255), table=<stationqualifier>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationqualifier_belongsTo', Column('belongsTo', String(length=255), table=<stationqualifier>, primary_key=True, nullable=False)))]

Shaibujnr commented 3 years ago

This is the output when the report is whitelisted to only the defined models.

[   (   'remove_index',
        Index('report_code', Column('reportCode', VARCHAR(length=255), table=<routinereportdefinition>))),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_test>, nullable=False))),
    (   'remove_index',
        Index('SiteID_UNIQUE', Column('SiteID', VARCHAR(length=20), table=<aws_sites>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationid_alias_identification', Column('idAlias', VARCHAR(length=255), table=<stationidalias>))),
    (   'remove_index',
        Index('stationstationidAlias', Column('belongsTo', VARCHAR(length=255), table=<stationidalias>))),
    (   'remove_index',
        Index('identification', Column('Element_Abbreviation', VARCHAR(length=50), table=<aws_toa5_bw1>))),
    (   'remove_index',
        Index('synop_code', Column('Element_Name', VARCHAR(length=255), table=<tm_307089>))),
    (   'remove_index',
        Index('synop_code1', Column('synop_code', VARCHAR(length=50), table=<tm_307089>))),
    (   'remove_index',
        Index('identification', Column('No', INTEGER(), table=<aws1>, nullable=False))),
    (   'remove_index',
        Index('ftpId_UNIQUE', Column('ftpId', VARCHAR(length=50), table=<aws_basestation>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('obsInitialIdentification', Column('StationId', VARCHAR(length=15), table=<qcabslimits>, nullable=False), Column('ElementId', BIGINT(), table=<qcabslimits>), Column('Datetime', DATETIME(), table=<qcabslimits>))),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_lsi1>, nullable=False))),
    (   'remove_index',
        Index('identification', Column('seq', BIGINT(), table=<seq_element>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_lsi>, nullable=False))),
    (   'remove_index',
        Index('identification', Column('Element_Name', VARCHAR(length=50), table=<aws_rwanda1>))),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<testing_aws1>, nullable=False))),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_rwanda_rain>, nullable=False))),
    (   'remove_index',
        Index('Identification', Column('Element_Name', VARCHAR(length=50), table=<aws_malawi1>))),
    (   'remove_index',
        Index('strID_UNIQUE', Column('strID', INTEGER(), table=<aws_structures>, nullable=False))),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<ss>, nullable=False))),
    (   'remove_index',
        Index('identification', Column('No', INTEGER(), table=<abc>, nullable=False))),
    (   'remove_index',
        Index('identification', Column('No', INTEGER(), table=<aws_malawi12>, nullable=False))),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<testing_aws>, nullable=False))),
    (   'remove_index',
        Index('Identification', Column('stationId', VARCHAR(length=10), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('yyyy', BIGINT(), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('mm', BIGINT(), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('dd', BIGINT(), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False), Column('hh', VARCHAR(length=5), table=<form_synoptic2_tdcf>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('report', Column('reportClass', VARCHAR(length=255), table=<routinereporttransmission>), Column('reportDatetime', DATETIME(), table=<routinereporttransmission>), Column('reportedFrom', VARCHAR(length=255), table=<routinereporttransmission>))),
    (   'remove_index',
        Index('identification', Column('Cols', INTEGER(), table=<aws_rema1>, nullable=False))),
    (   'add_index',
        Index('ix_faultresolution_associatedWith', Column('associatedWith', BigInteger(), table=<faultresolution>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('code', Column('instrumentId', VARCHAR(length=255), table=<instrument>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_instrument_installedAt', Column('installedAt', String(length=255), table=<instrument>))),
    (   'add_index',
        Index('ix_instrument_instrumentId', Column('instrumentId', String(length=255), table=<instrument>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('report_id', Column('reportId', BIGINT(), table=<instrumentfaultreport>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_instrumentfaultreport_reportId', Column('reportId', BigInteger(), table=<instrumentfaultreport>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_instrumentfaultreport_reportedFrom', Column('reportedFrom', String(length=255), table=<instrumentfaultreport>))),
    (   'add_index',
        Index('ix_instrumentinspection_performedAt', Column('performedAt', String(length=255), table=<instrumentinspection>))),
    (   'remove_index',
        Index('elementCode', Column('elementId', BIGINT(), table=<obselement>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x114b07dc0>, for_update=False)))),
    (   'add_index',
        Index('ix_obselement_elementId', Column('elementId', BigInteger(), table=<obselement>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x114032ee0>, for_update=False)))),
    (   'remove_index',
        Index('obsElementObservationInitial', Column('describedBy', BIGINT(), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationObservationInitial', Column('recordedFrom', VARCHAR(length=255), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationfinal_describedBy', Column('describedBy', BigInteger(), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationfinal_recordedFrom', Column('recordedFrom', String(length=255), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('obsElementObservationInitial', Column('describedBy', BIGINT(), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationObservationInitial', Column('recordedFrom', VARCHAR(length=255), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationinitial_describedBy', Column('describedBy', BigInteger(), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationinitial_recordedFrom', Column('recordedFrom', String(length=255), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('scheduleClassIdeification', Column('scheduleClass', VARCHAR(length=255), table=<obsscheduleclass>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x114b41b20>, for_update=False)))),
    (   'add_index',
        Index('ix_obsscheduleclass_refersTo', Column('refersTo', String(length=255), table=<obsscheduleclass>))),
    (   'add_index',
        Index('ix_obsscheduleclass_scheduleClass', Column('scheduleClass', String(length=255), table=<obsscheduleclass>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x1140bf940>, for_update=False)))),
    (   'add_index',
        Index('ix_paperarchive_classifiedInto', Column('classifiedInto', String(length=50), table=<paperarchive>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('paperarchivedef', Column('formId', VARCHAR(length=50), table=<paperarchivedefinition>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_paperarchivedefinition_formId', Column('formId', String(length=50), table=<paperarchivedefinition>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('physicalFeatureidentification_idx', Column('classifiedInto', VARCHAR(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationfeature', Column('associatedWith', VARCHAR(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeature_associatedWith', Column('associatedWith', String(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeature_classifiedInto', Column('classifiedInto', String(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationFeatureClass', Column('featureClass', VARCHAR(length=255), table=<physicalfeatureclass>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeatureclass_featureClass', Column('featureClass', String(length=255), table=<physicalfeatureclass>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeatureclass_refersTo', Column('refersTo', String(length=255), table=<physicalfeatureclass>))),
    (   'remove_index',
        Index('StationStationId', Column('stationId', VARCHAR(length=255), table=<station>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_station_stationId', Column('stationId', String(length=255), table=<station>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('obsElementobservationInitial', Column('describedBy', BIGINT(), table=<stationelement>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationobservationInitial', Column('recordedFrom', VARCHAR(length=255), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_describedBy', Column('describedBy', BigInteger(), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_recordedFrom', Column('recordedFrom', String(length=255), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_recordedWith', Column('recordedWith', String(length=255), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_scheduledFor', Column('scheduledFor', String(length=255), table=<stationelement>))),
    (   'remove_index',
        Index('stationQualifierIdentification', Column('belongsTo', VARCHAR(length=255), table=<stationqualifier>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationqualifier_belongsTo', Column('belongsTo', String(length=255), table=<stationqualifier>, primary_key=True, nullable=False)))]

Shaibujnr commented 3 years ago

The whitelist algorithm basically, takes the second item in the list, checks if it's a Table object then checks the name of the Table object against the list of the tables and filters accordingly.

For the second output, it looks like all we have left is add_index and remove_index operations. I can easily extend the algorithm to check for index objects and then check the table name of the index objects then filter accordingly too. And for every operation we encounter we can keep extending the algorithm to support it.

So let me know what you think @isedwards

isedwards commented 3 years ago

Yes, it makes sense. Let's add a capability to ignore the add_index and remove_index operations for the tables that we're not interested in.

Let's refer to these 25 tables as climsoft/v4_1_1_core.py.

Once we have an empty result from get_schema_diff() for these 25 tables when comparing to the database schema then it may make sense to try and see whether we can represent more of the tables with a larger complete set of models called climsoft/v4_1_1.

Shaibujnr commented 3 years ago

output after whitelisting indexes too.

[   (   'add_index',
        Index('ix_faultresolution_associatedWith', Column('associatedWith', BigInteger(), table=<faultresolution>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('code', Column('instrumentId', VARCHAR(length=255), table=<instrument>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_instrument_installedAt', Column('installedAt', String(length=255), table=<instrument>))),
    (   'add_index',
        Index('ix_instrument_instrumentId', Column('instrumentId', String(length=255), table=<instrument>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('report_id', Column('reportId', BIGINT(), table=<instrumentfaultreport>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_instrumentfaultreport_reportId', Column('reportId', BigInteger(), table=<instrumentfaultreport>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_instrumentfaultreport_reportedFrom', Column('reportedFrom', String(length=255), table=<instrumentfaultreport>))),
    (   'add_index',
        Index('ix_instrumentinspection_performedAt', Column('performedAt', String(length=255), table=<instrumentinspection>))),
    (   'remove_index',
        Index('elementCode', Column('elementId', BIGINT(), table=<obselement>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x11d2b69d0>, for_update=False)))),
    (   'add_index',
        Index('ix_obselement_elementId', Column('elementId', BigInteger(), table=<obselement>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x11c99f100>, for_update=False)))),
    (   'remove_index',
        Index('obsElementObservationInitial', Column('describedBy', BIGINT(), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationObservationInitial', Column('recordedFrom', VARCHAR(length=255), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationfinal_describedBy', Column('describedBy', BigInteger(), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationfinal_recordedFrom', Column('recordedFrom', String(length=255), table=<observationfinal>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('obsElementObservationInitial', Column('describedBy', BIGINT(), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationObservationInitial', Column('recordedFrom', VARCHAR(length=255), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationinitial_describedBy', Column('describedBy', BigInteger(), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_observationinitial_recordedFrom', Column('recordedFrom', String(length=255), table=<observationinitial>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('scheduleClassIdeification', Column('scheduleClass', VARCHAR(length=255), table=<obsscheduleclass>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x11d308640>, for_update=False)))),
    (   'add_index',
        Index('ix_obsscheduleclass_refersTo', Column('refersTo', String(length=255), table=<obsscheduleclass>))),
    (   'add_index',
        Index('ix_obsscheduleclass_scheduleClass', Column('scheduleClass', String(length=255), table=<obsscheduleclass>, primary_key=True, nullable=False, server_default=DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x11ca0ee80>, for_update=False)))),
    (   'add_index',
        Index('ix_paperarchive_classifiedInto', Column('classifiedInto', String(length=50), table=<paperarchive>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('paperarchivedef', Column('formId', VARCHAR(length=50), table=<paperarchivedefinition>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_paperarchivedefinition_formId', Column('formId', String(length=50), table=<paperarchivedefinition>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('physicalFeatureidentification_idx', Column('classifiedInto', VARCHAR(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationfeature', Column('associatedWith', VARCHAR(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeature_associatedWith', Column('associatedWith', String(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeature_classifiedInto', Column('classifiedInto', String(length=255), table=<physicalfeature>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationFeatureClass', Column('featureClass', VARCHAR(length=255), table=<physicalfeatureclass>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeatureclass_featureClass', Column('featureClass', String(length=255), table=<physicalfeatureclass>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_physicalfeatureclass_refersTo', Column('refersTo', String(length=255), table=<physicalfeatureclass>))),
    (   'remove_index',
        Index('StationStationId', Column('stationId', VARCHAR(length=255), table=<station>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_station_stationId', Column('stationId', String(length=255), table=<station>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('obsElementobservationInitial', Column('describedBy', BIGINT(), table=<stationelement>, primary_key=True, nullable=False))),
    (   'remove_index',
        Index('stationobservationInitial', Column('recordedFrom', VARCHAR(length=255), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_describedBy', Column('describedBy', BigInteger(), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_recordedFrom', Column('recordedFrom', String(length=255), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_recordedWith', Column('recordedWith', String(length=255), table=<stationelement>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationelement_scheduledFor', Column('scheduledFor', String(length=255), table=<stationelement>))),
    (   'remove_index',
        Index('stationQualifierIdentification', Column('belongsTo', VARCHAR(length=255), table=<stationqualifier>, primary_key=True, nullable=False))),
    (   'add_index',
        Index('ix_stationqualifier_belongsTo', Column('belongsTo', String(length=255), table=<stationqualifier>, primary_key=True, nullable=False)))]

Shaibujnr commented 3 years ago

After generating the migration script with alembic revision --autogenerate I have manually updated the models to align more with the database. Right now, get_schema_diff returns [] for the core tables, however the alembic migration script still detects the below.

def upgrade():
    # ### commands auto generated by Alembic - please adjust! ###
    op.alter_column('featuregeographicalposition', 'latitude',
               existing_type=mysql.DOUBLE(precision=11, scale=6, asdecimal=True),
               type_=sa.Float(precision=11, asdecimal=True),
               existing_nullable=True)
    op.alter_column('featuregeographicalposition', 'longitude',
               existing_type=mysql.DOUBLE(precision=11, scale=6, asdecimal=True),
               type_=sa.Float(precision=11, asdecimal=True),
               existing_nullable=True)
    op.alter_column('station', 'latitude',
               existing_type=mysql.DOUBLE(precision=11, scale=6, asdecimal=True),
               type_=sa.Float(precision=11, asdecimal=True),
               existing_nullable=True)
    op.alter_column('station', 'longitude',
               existing_type=mysql.DOUBLE(precision=11, scale=6, asdecimal=True),
               type_=sa.Float(precision=11, asdecimal=True),
               existing_nullable=True)
    op.alter_column('stationlocationhistory', 'latitude',
               existing_type=mysql.DOUBLE(precision=11, scale=6, asdecimal=True),
               type_=sa.Float(precision=11, asdecimal=True),
               existing_nullable=True)
    op.alter_column('stationlocationhistory', 'longitude',
               existing_type=mysql.DOUBLE(precision=11, scale=6, asdecimal=True),
               type_=sa.Float(precision=11, asdecimal=True),
               existing_nullable=True)
    # ### end Alembic commands ###

For some reason, compare_metadata doesn't output alter_column statements. These columns are of type double in the database and that type is only specific to mysql. SQLACodegen generated the columns as float.

how do we resolve this? @isedwards

isedwards commented 3 years ago

Thank you @Shaibujnr - it's odd that compare_metadata doesn't pick this up - I thought alembic used compare_metadata in order to produce_migrations (autogenerate api).

It looks like the best solution would be to manually update the output of sqlacodegen to use mysql.DOUBLE (again in a separate commit to make it easier to document and reference the changes)? Presumably get_schema_diff would still return [] if we do this but the model would more accurately represent the database.

Shaibujnr commented 3 years ago

@isedwards https://github.com/opencdms/pyopencdms/pull/28

opencdms-dev / pyopencdms-old

Create SQLAlchemy ORM models for Climsoft CDMS #2