search

cityofaustin / atd-data-tech

Austin Transportation Data & Technology Services
18 stars 2 forks source link

Add Signal LPI attributes to ArcGIS Online and Socrata #1165

Open johnclary opened 4 years ago

amenity commented 4 years ago

@johnclary - should this be broken out into separate issues for Geo and Dev? Want to make sure it gets sized and in the correct team's backlog.

johnclary commented 4 years ago

yes, converting to epic and documenting this process. Involves Apps as well.

johnclary commented 4 years ago

@amenity @dianamartin i started on docs for this process. Would one of you walk me through how to put them in the new wiki?

How to Add New Fields to Existing Socrata Datasets

This doc describes the process for adding new fields to an existing Socrata dataset whose source is a Knack application.

  1. Locate the API view in the Knack application that serves as the source for the Socrata dataset. Each of our Knack applications will have a series of views (aka, pages) which we call "API Views" and are purpose-built for publishing data from Knack to external sources (Socrata, ArcGIS Online, etc.).

If you're unsure which view is the source for a dataset, you'll need a developer to review the ETL script which publishes the Knack data to the destination system. They will be able to confrim the page and scene which serves as the source.

The majority of our Socrata and AGOL publishing scripts live in the atd-data-publshing repo, and are defined in this configuration file.

  1. Field naming: In Knack parlance, field names are called "Labels". There are very few restrictions in Knack around how you can name a field. It can have spaces, numbers, mixed case, etc. This is because Knack maintains it's own unique field identifiers which you as a builder cannot modify, e.g. field_101, etc.

Crucially, we use the user-defined Knack field label as the field name in the downstream system. In Socrata, these field names are called the "API Field Names", and they must be lower case, with no spaces or special characters. In ArcGIS Online, these fields may be upper or lower case, with no spaces or special characters.

As such, the field "label" of any Knack field which is to be published to an external system should follow these same conventions. Typically, we follow the practice of labeling these fields in upper case with an underscore in leiu of spaces.

Our Knack > Socrata publishing script will convert any Knack field name to lowercase to match Socrata's expected field format.

  1. Once you've located the API view in the Knack builder, first verify (1) that the fields are named appropriately (as described above) and (2) if the columns you want to include in the destination dataset are already present in the view. If they fields are not present, add them. Otherwise, continue to the next step.

  2. Login to data.austintexas.gov and navigate to the "Primer" page of the dataset you wish to modify. Scroll down to the "Columns in this Dataset" section and verify that the fields you wish to add are not present in the dataset. To add new fields, click on the blue "Edit" button in the top right corner of the Primer page.

  3. Click on the button below "Review & Configure Data". From the configuration editor, click the "Add Column" button. Fill out the form fields as follows:

  1. Click "Save" to save your changes. Return to the Primer page, and click the blue "Update" button in the top-right of the page to publish the dataset changes.

  2. In order to refresh the Socrata dataset with values for the new columns, a dev will need to manually run the dataset publishing script with a --replace command. Contact a dev for help.

dianamartin commented 4 years ago

@johnclary I can show you how to edit the GitBook wiki when you're free

johnclary commented 4 years ago

added my docs: https://app.gitbook.com/@atd-data-and-technology-services/s/austin-transportation-knack-guides/knack/open-data-portal-socrata-integration

johnclary commented 4 years ago

@amenity FYI this should be done by thursday and is worth updating AMD about