Task: draft data documation and/or dictionary

[nlebovits]

As part of our documentation for the website, we need dataset documentation It should include:

1. Summary
   • Provide a brief overview of the dataset, its purpose, and its significance.
2. Introduction
   • Detailed background information about the dataset.
   • Overview of its scope and potential applications.
3. How Data Were Chosen
   • Explain the criteria and methodology used for selecting the data.
   • Discuss any filters or selection processes applied.
4. Data Sources
   • Describe the sources of the underlying data.
   • Include details on the origin and credibility of these sources.
5. Data Processing
   • Detail any processing steps applied to create the final dataset, like spatial Kernel Density Estimation (KDE) for gun crime density.
   • Explain the rationale and methods used for processing.
6. Metadata
   • Automatically Generated:
     • Use SQL queries to extract structural metadata from PostgreSQL (e.g., table schemas, column data types).
   • Manually Added:
     • Update frequency.
     • Date of last update.
     • Data versioning information.
     • Dataset creator and contact information.
7. Glossary of Terms
   • Define specific terms, jargon, and acronyms used within the dataset and documentation.
8. Appendices
   • Include any additional relevant information.
   • This might contain detailed methodology, extended data examples, or technical notes.
9. Licensing and Usage Guidelines
   • State the licensing terms under which the dataset is released.
   • Provide instructions on how to appropriately cite the dataset.
10. Access and Format
    • Instructions on how to access the dataset.
    • Details of the file formats available.
11. Quality and Limitations
    • Discuss the quality and accuracy of the data.
    • Note any known limitations or biases in the dataset.
12. Update and Maintenance Information
    • Outline the process for updating the dataset.
    • Mention any maintenance schedules or plans.
13. Support and Feedback
    • Provide contact information for support.
    • Explain how users can give feedback or report issues.
14. Ethical Considerations
    • Address any ethical concerns, especially if the data includes sensitive information.

Considerations for Automatic Generation:

• Structural Metadata: Automate the extraction of metadata like table structure, column data types, and constraints using PostgreSQL system catalogs.
• Data Dictionary: Automatically generate a basic data dictionary, detailing columns, data types, and possibly some constraints or relationships.
• Version Control: Implement a system to automatically track and document changes or updates to the dataset.

Manual Input:

• Certain aspects, such as detailed descriptions, ethical considerations, usage guidelines, and data processing methodologies, will require manual documentation to provide context and clarity.

[nlebovits]

See also: https://github.com/CodeForPhilly/vacant-lots-proj/issues/80

[nlebovits]

Possible model for data license (from AllTransit):

How will you use the AllTransit data? Affordable Housing Planning/Advocacy Mapping Benchmarking or Indicators Regional Planning Inform Housing Policy Transportation Planning Inform Transportation Policy Other (please describe)

DATA LICENSE AGREEMENT + TERMS OF USE

This is a legal agreement between you, the end-user (“User”), and the Center for Neighborhood Technology (“CNT”). By downloading CNT’s data on housing and transportation affordability (hereinafter referred to as the “Data”), you agree to be bound by the terms and conditions of this Agreement and CNT’s privacy policy, located at http://www.cnt.org/about/privacy/. If you do not agree to be bound by these terms, do not download or use the Data.

NOW, THEREFORE, IN CONSIDERATION of the mutual promises and obligations contained herein, the parties agree as follows:

1. CNT hereby grants to the User a non-exclusive, revocable, limited, non-transferable, royalty-free license to use the Data solely for (1) research, scholarly or academic purposes, (2) the internal use of your business or organization, or (3) your own personal non-commercial use. User may not reproduce, sell, rent, lease, loan, distribute, sublicense or other transfer any Data, in whole or in part, to any other party. Notwithstanding the foregoing, User may incorporate limited portions of the Data in scholarly, research or academic publications or for purposes of news reporting, provided User acknowledges the source of the Data as set forth in Paragraph 2, and includes the following legend:

The Center for Neighborhood Technology bears no responsibility for the analyses or interpretations of the data presented here.

2. User agrees to cite CNT as the source of information whenever Data is used in reports, papers, publications, maps and/or other products. Citations shall be directly on tables, charts, and maps using the Data and shall include an express reference to CNT and the complete title of the Data. In instances where User makes further calculations using Data, User shall also cite that further calculations were made using source data provided by CNT.

3. All right, title, and interest in and to the Data, and other intellectual property and ownership rights embodied in the Data, including but not limited to source code and object code, are CNT’s sole and exclusive property and are protected from unauthorized use, copying and distribution by U.S. laws and regulations. User acknowledges that CNT owns all rights in and to the Data and that User does not acquire any rights in the Data by virtue of User’s use of the Data under this Agreement or otherwise. User agrees the rights granted by CNT in Paragraph 1 are subject to the terms of this Agreement and that the foregoing grant is expressly conditioned on User using the Data solely for non-commercial, non-profit and non-revenue based purposes. User further agrees not to challenge, directly or indirectly, CNT’s rights in the Data or assert any right, title or interest in or to the Data or assist any third-party in doing so.

4. DISCLAIMER OF WARRANTIES. CNT DOES NOT WARRANT THAT THE DATA IS ACCURATE OR COMPLETE. THE DATA IS PROVIDED "AS IS" TO THE FULLEST EXTENT PERMISSIBLE BY LAW, AND CNT AND ITS SUBSIDIARIES AND AFFILIATES DISCLAIM ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. CNT BEARS NO RESPONSIBILITY FOR THE ANALYSES OR INTERPRETATIONS OF THE DATA.

5. LIMITATIONS OF LIABILITY. USER ASSUMES ALL RISK ASSOCIATED WITH USE OF THE DATA AND AGREES THAT IN NO EVENT SHALL CNT BE LIABLE TO USER OR ANY THIRD PARTY FOR ANY DAMAGES WHATSOEVER INCLUDING WITHOUT LIMITATION AMOUNTS REPRESENTING DIRECT, INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL, EXEMPLARY, OR PUNITIVE DAMAGES, LOSS OF PROFITS, LOSS OF BUSINESS, HOWEVER CAUSED, ON ANY THEORY OF LIABILITY, INCLUDING BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE OR PROFESSIONAL ERRORS AND OMISSIONS) OR OTHERWISE, EVEN IF USER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN THE EVENT THAT APPLICABLE LAW DOES NOT PERMIT THE FOREGOING EXCLUSION OF LIABILITY AND DAMAGES, CNT’S LIABILITY SHALL BE LIMITED TO THE GREATER OF $100 OR THE LOWEST AMOUNT ALLOWED BY LAW.

6. INDEMNIFICATION. User, at its own expense, shall defend, indemnify and hold harmless, or have a third-party defend, indemnify or hold harmless, CNT, its subsidiaries and affiliates, and its and their directors, officers, employees, and agents, and their respective successors, heirs and assigns from and against any and all claims, allegations, demands, suits, and proceedings (collectively, “Claims”), and from and against any liability, judgments, awards, damages, settlements, fees and costs (including attorney’s fees) incurred in connection with any such Claim, brought or asserted by any third party in connection with this Agreement, the Data, or User’s use of the Data.

7. To assist CNT in the maintenance of the Data, the Licensee agrees to provide CNT, at the address shown below, with any information concerning errors or discrepancies found in the Data.

Center for Neighborhood Technology Attn: Linda Young Research Director 17 North State Street #1400 Chicago, IL 60602

8. This license will terminate (1) automatically without notice from CNT if you fail to comply with any of the provisions of this Agreement, or (2) upon written notice (by email or otherwise) from CNT. Upon termination of this Agreement, you agree to destroy all copies of any Data, in whole or in part and in any and all media, in your custody and control.

9. COLLECTION OF PERSONAL INFORMATION. CNT shares your concerns about maintaining the integrity and privacy of personal information collected on the Internet. CNT is committed to protecting your privacy, and CNT’s privacy policy, located at http://www.cnt.org/about/privacy/, is intended to explain CNT’s information collection and dissemination practices in connection with the Data and the CNT’s family of websites located at http://www.cnt.org. Please review the entire privacy policy and contact CNT (at info@cnt.org or anjuli@cnt.org using “privacy at cnt.org” in the subject line) if you have any questions or concerns.

10. This Agreement shall be governed by and in accordance with the laws of the State of Illinois without regard to conflict of laws principles. User agrees to submit to the jurisdiction of federal and state courts located in Cook County, Illinois in any dispute relating to or arising from this Agreement.