index scans bulk update

adambreznicky commented 5 years ago

Issue by adambreznicky Thursday Mar 28, 2019 at 18:36 GMT Originally opened as https://github.com/TNRIS/data.tnris.org/issues/175

the following are the tasks which need to be completed for the index scans to be 'completed' and in s3 for public availability through the dataHub:

[x] printed list of 350+ scanned indexes which exist in s3 but cannot be tied to any collection in the dataHub database: these need to be reviewed and have their dates updated either in the filename or in the database to be correct and relational @LaurenKirk @csidenblad
[x] @csidenblad has a post-it note list of remaining single-county txdot indexes which need to be uploaded to s3 via cloudberry @LaurenKirk
[x] multi-county indexes uploaded to s3 via cloudberry @csidenblad @LaurenKirk
[x] @adambreznicky train RDC about new table 'Photo Index Scanned LS4 Links' as shown at bottom of collection form in admin console. holler when y'all wanna chat about it
[x] @adambreznicky re-run index_scans_etl.py to update photo_index_scanned_ls4_link table so links are updated for frontend app
[x] @adambreznicky re-generate list of scanned indexes for tnris.org/mapserver
[x] @adambreznicky delete etl_scanned_Index database view as it is only used by the index_scans_etl.py script

these don't need to be done in this order necessarily. the script to update the database with the current s3 uploads can run over and over without harm so updates can happen at any increment of RDC's choosing

adambreznicky commented 5 years ago

Comment by adambreznicky Tuesday Apr 23, 2019 at 16:24 GMT

updated scanned index list on tnris.org/mapserver to be dynamic from the database so that it's always matching what is managed in the dataHub admin console

adambreznicky commented 5 years ago

Comment by adambreznicky Tuesday Apr 23, 2019 at 16:39 GMT

temp view deleted. should be good to go for rdc long term maintenance. i do'nt plan to run the bulk update again.

will leave the issue open until all checkboxes are complete so that at that point we can officially say all indexes are completed and in the database.

adambreznicky commented 5 years ago

Comment by adambreznicky Tuesday Apr 23, 2019 at 16:48 GMT

insofar as training on the new table 'Photo Index Scanned LS4 Links', i already went over this briefly with @csidenblad so whenever @LaurenKirk wants a rundown just let me know. that ball is in your court. if @csidenblad wants a refresher at that time she can join, of course!

adambreznicky commented 5 years ago

resolved ac95efa50a53ef74aaf444cc59819c680ca6e33f

TNRIS / api.tnris.org

index scans bulk update #2