bcgov / FIT_opendatadownloader

Monitor open data, report when changes are detected
Apache License 2.0
0 stars 0 forks source link

only write to s3 if data has changed #10

Open smnorris opened 2 months ago

smnorris commented 2 months ago

Updated requirement - it is not necessary to retain all downloads indefinitely - only write download to s3 if something has happened.

A file structure something like this should work:

/MVRD/surrey/<layer>.gdb.zip                                                            
/MVRD/surrey/<layer>_changes/<layer>.gdb.zip               
/MVRD/surrey/<layer>_changes/change_report.xlsx      
/MVRD/surrey/<layer>_changes/change_report.gdb.zip
smnorris commented 2 months ago

Definitions of data in the changes folder:

  1. Spatial layers representing the diff
    • additions
    • deletions
    • modifications - geometry only
    • modifications - attribute only (with old/new attribute diff)
    • modifications - geometry + attribute (with old/new attribute diff)

All modification layers to use the latest geometries.

2 Report summarizing changes with columns: