cityofaustin / atd-data-tech

Austin Transportation Data & Technology Services
17 stars 2 forks source link

Move the data lake publisher scripts from data01 to the new servers #17875

Open frankhereford opened 2 weeks ago

frankhereford commented 2 weeks ago

We're waiting to hear back from Ken if he and his team are still using or relying on the cron'd ETLs related to the data lake. Based on that the outcome of that conversation, or CTM wanting to destroy the old on-prem servers, this issue is to track moving those programs over to the new server.

cc @chiaberry @mddilley

johnclary commented 6 days ago

I reached out to the MMC about the GRIDSMART datasets, and Allyson's response was:

We aren’t readily using this dataset anymore. A majority of the devices that fed this dataset have been removed or are defunct; we wouldn’t categorize the inbound data today as ‘representative’ of volumes or other items we’d generally use for high-level representation.

I think we should not migrate these ETLs to our new servers, just as Frank had originally suggested. I have updated the public dataset's description to indicate that it is no longer updated: https://datahub.austintexas.gov/Transportation-and-Mobility/Camera-Traffic-Counts/sh59-i6y9/about_data

Next steps: communicate this to AMD and to Ken P. cc @chiaberry @Charlie-Henry @ChristinaTremel

chiaberry commented 5 days ago

This has been communicated to AMD, @johnclary do you want to send the email to Ken, or would you like me to?

frankhereford commented 1 day ago

Do you all think this is ready to close? 👍 from me, but I don't want to rush it.