SpatialTransformer - handle vast amounts of aspatial data

[nesdolya]

Two issues need to be addressed:

1. Need to determine how we are going to handle the vast amounts of aspatial data. 2BT data team did not separate the spatial files from the aspatial files, so there will be tonnes of aspatial records in the raw_data_tracker.
2. Need to determine how RecordsReviser will display only the most relevant information - e.g. dropped = False and planting year = YYYY. Or some sort of sorting filtering in the GUI itself.

[nesdolya]

Addressing point #1: We decided to add a event_type field with a spatial/aspatial flag To address point #2: @AnthonyRodway - How difficult would it be to filter the records reviser to only display the records that were input within the last 24 hours? This would ONLY apply to when the records reviser automatically pops up at the end of the SpatialTransformer run.

[AnthonyRodway]

Changes were made in this commit: https://github.com/cat-cfs/twobilliontoolkit/commit/c97d71018ca74810367038626a15b2a6dab668d0

Changes:

• The record reviser (function) can now take a filter dictionary with some special functionality for the created_at column. If called from spatial transformer there is a filter already applied to only include entries that were created on the same day. Some other functionality may need to be implemented in the future for other filters but the framework is there.
• The record reviser table can now be sorted by each column either ascending or descending. When the reset or save button is clicked it will automatically go back to sorting the project_spatial_id ascending so it is consistent with the dataframe which saves having to redo alot of the code (could be a future job to rework the tool)

[AnthonyRodway]

The entry type was made quite a while ago in a previous commit but forgot to document which one, sorry about that. It is implemented in the new spatial logic branch

[nesdolya]

Tested and functioning