damonmcc commented 3 months ago

Project Description

We've set up the scaffolding to automate the push to Open Data and do validation on datasets. Now we need to implement this for all our (and GIS's) products.

Outcomes

The deployments to Socrata are entirely automated for all products.

Timing

This will require a lot of coordination, and modification to existing datasets to fix issues. This will take months.

Tasks

Tracking here (last column)

800

Motivations

...

Notes

...

damonmcc commented 2 months ago

notes from 4/23

let's use the progress indicators from the GDE biweekly
having files in edm-distributions bucket is important (whether it's for an API or manual distribution to Socrata)
Socrata dataset types
- old map types block API use
- to deprecate, we'd like to make an old dataset "useless" and point to the dataset that replaces it
- prepend title with something like DEPRECATED
- change description to point to new dataset
- update with row-less data
datasets to prioritize
- those that GIS is scheduled to update soon and/or monthly

next steps

ask GIS about their upcoming dataset updates
confirm automation progress goals and track (possibly in Open Data excel sheet)
ask GIS to choose a project lead to streamline comms and planning
add a column to the Open Data excel sheet for "dataset type" (@damonmcc)

alexrichey commented 2 months ago

Also adding: goal of automations is 80% by end of Q2 (July 1). 100% by Q3, per Strategic Planning

alexrichey commented 2 months ago

@croswell81 FYI, this is our main issue for the project.

croswell81 commented 2 months ago

[like] Matthew Croswell (DCP) reacted to your message:

From: Alex Richey @.> Sent: Wednesday, April 24, 2024 9:12:34 PM To: NYCPlanning/data-engineering @.> Cc: Matthew Croswell (DCP) @.>; Mention @.> Subject: [EXTERNAL] Re: [NYCPlanning/data-engineering] DE<>GIS Automations - Phase 2 (Issue #663)

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Forward suspect email to @.**@.> as an attachment (Click the More button, then forward as attachment).

@croswell81https://github.com/croswell81 FYI, this is our main issue for the project.

— Reply to this email directly, view it on GitHubhttps://github.com/NYCPlanning/data-engineering/issues/663#issuecomment-2075853066, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AICWQSV3OXM3XIHSA34TWRTY7AN4FAVCNFSM6AAAAABEM2OKJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVHA2TGMBWGY. You are receiving this because you were mentioned.Message ID: @.***>

NYCPlanning / data-engineering

DE<>GIS Automations - Phase 2 #663