Pare down list of scraped committees?

[hancush]

By scraping all committees and subcommittees from the API, we add an additional 83 organizations to the OCD API. Let's have a look at this list and decide if there are any we can definitely omit.

• Committee of the Whole
• SELECT COMMITTEE ON REDISTRICTING
• Parks, Recreation, Cultural Affairs, and International Intergroup Relations
• Task Force on Affordable Houisng Preservation (HC: typo, correct spelling does not exist, needs correction)
• Unknown Body
• Select Committee on Police Performance & Community Relations
• Task Force on Mitchell-Lama Housing
• PENSIONS
• Legislative Documents Unit
• SUBCOMMITTEE ON PENSIONS (NYCC)
• INTERNATIONAL INTERGROUP RELATIONS & SPECIAL EVENTS
• MARKETS AND RETAIL BUSINESSES
• PRIVATE SECTOR PARTICIPATION & MUNICIPAL POLICY
• INSURANCE ISSUES
• Select Committee on Pest Control
• UNKNOWN COMMITTEE
• WATERFRONT DEVELOPMENT
• ASBESTOS
• SUBCOMMITTEE ON SITE SELECTION (NYCC)
• Committee on Lower Manhattan Redevelopment
• Task Force to Combat Gun Violence
• ~Subcommittee on Mental Health, Mental Retardation, Alcoholism & Drug Abuse Services~ (HC: correct version of manually added organization)
• PARENTAL INVOLVEMENT IN EDUCATION
• Subcommittee on Census 2000
• CITY CONTRACTS-86
• Subcommittee on Census
• Subcommittee on Crime and Substance Abuse
• Select Committee on Community Development
• Select Committee on Libraries
• YOUTH VIOLENCE
• Subcommittee on Private Uses of Public Space
• Select Committee on Charter Revision
• Subcommittee on Juvenile Justice
• Committee on Technology in Government
• Charter Reform
• Task Force on Infrastructure
• SELECT COMMITTEE
• Subcommittee on District Attorneys, Courts and Investigations
• HUMAN RIGHTS-86
• PHYSICALLY CHALLENGED
• SUBCOMMITTEE ON MEDICAL ALTERNATIVES (NYCC)
• SUBCOMMITTEE ON ZONING, PLANNING & LANDMARKS (NYCC)
• Subcommittee on Small Business, Retail and Emerging Technologies
• SUBCOMMITTEE ON CITY CONTRACTS (NYCC)
• MEDICAL TECHNOLOGY
• Select Committee on Charter Reform
• Select Committee on Technology in Government
• LEGISLATIVE COMMISSION ON THE HOMELESS
• PARKS CONCESSIONS
• Subcommittee on People with Disabilities
• Task Force on Hospital Closings
• HOMELESS
• Joint Committee
• VETERANS AFFAIRS
• Task Force on Technology in Government
• COMMITTEES (NYCC)
• MENTAL HEALTH
• Subcommittee on Immigration
• Subcommittee on Public Housing
• By the Committee on Rules, Privileges & Elections
• SELECT COMMITTEE ON IMMIGRATION (NYCC)
• Subcommittee on Abandonment, Foreclosure & Disinvestment
• Select Committee on Waterfronts
• Committee on Mental Health, Developmental Disability, Alcoholism, Substance Abuse amd Disability Services (HC: typo, correct spelling exists, needs deletion from Legistar API)
• Subcommittee on Federal Affairs
• Task Force on the Operations & Improvements of the Department of Buildings
• SUBCOMMITTEE ON WATERFRONT DEVELOPMENT (NYCC)
• SUBCOMMITTEE ON VETERANS AFFAIRS (NYCC)
• By the Committee on Standards and Ethics
• REVENUE ENHANCEMENT
• SAFETY IN PUBLIC HOUSING
• TECHNOLOGY IN GOVERNMENT
• Select Committee on Civil Rights
• Select Committee on Lower Manhattan Redevelopment
• Select Committee on Media Services and Consumer Rights
• Subcommittee on Revenue and Forecast
• MANUFACTURING AND SMALL BUSINESS
• Subcommittee on Substance Abuse
• SUBCOMMITTEE ON FRANCHISES, REVOCABLE CONSENTS & CONCESSIONS
• SUBCOMMITTEE ON REVENUE ENHANCEMENT (NYCC)
• Subcommittee on Permits, Dispositions and Concessions
• CITY PLANNING

[fgregg]

Is there a pattern for why some are terse and all caps? Are they earlier?

[hancush]

Here are the organizations with current memberships (not yet past end date):

                                                   name                                                    | count
-----------------------------------------------------------------------------------------------------------+-------
 New York City Council                                                                                     |    52
 Committee on Land Use                                                                                     |    42
 Committee on Education                                                                                    |    34
 Committee on Transportation                                                                               |    26
 Committee on Finance                                                                                      |    22
 Committee on Public Safety                                                                                |    22
 Committee on Housing and Buildings                                                                        |    22
 Committee on Rules, Privileges and Elections                                                              |    22
 Committee on Health                                                                                       |    18
 Committee on Cultural Affairs, Libraries and International Intergroup Relations                           |    18
 Committee on Economic Development                                                                         |    16
 Committee on Small Business                                                                               |    16
 Committee on General Welfare                                                                              |    16
 Committee on State and Federal Legislation                                                                |    14
 Committee on Governmental Operations                                                                      |    14
 Committee on Aging                                                                                        |    14
 Subcommittee on Landmarks, Public Siting and Maritime Uses                                                |    14
 Committee on Courts and Legal Services                                                                    |    14
 Task Force on Affordable Houisng Preservation                                                             |    14
 Committee on Higher Education                                                                             |    14
 Committee on Oversight and Investigations                                                                 |    14
 Committee on Youth Services                                                                               |    14
 Subcommittee on Zoning and Franchises                                                                     |    14
 Committee on Parks and Recreation                                                                         |    14
 Committee on Public Housing                                                                               |    14
 Committee on Recovery and Resiliency                                                                      |    14
 Committee on Mental Health, Developmental Disability, Alcoholism, Substance Abuse and Disability Services |    12
 Committee on Contracts                                                                                    |    12
 Committee on Civil Rights                                                                                 |    10
 Committee on Environmental Protection                                                                     |    10
 Committee on Technology                                                                                   |    10
 Subcommittee on Planning, Dispositions and Concessions                                                    |    10
 Committee on Immigration                                                                                  |    10
 Committee on Women's Issues                                                                               |    10
 Committee on Sanitation and Solid Waste Management                                                        |    10
 Committee on Civil Service and Labor                                                                      |    10
 Committee on Veterans                                                                                     |    10
 Committee on Fire and Criminal Justice Services                                                           |    10
 Committee on Consumer Affairs                                                                             |    10
 Committee on Juvenile Justice                                                                             |    10
 Committee on Waterfronts                                                                                  |    10
 Committee on Standards and Ethics                                                                         |    10
 Subcommittee on Libraries                                                                                 |     2
 Subcommittee on Senior Centers                                                                            |     2
 Subcommittee on Non-Public Schools                                                                        |     2
 Task Force on Mitchell-Lama Housing                                                                       |     1
 Subcommittee on Substance Abuse                                                                           |     1
(47 rows)

[hancush]

Here are the organizations where all memberships expired prior to 1996 (the oldest legislation in the API).

                             name                             | count
--------------------------------------------------------------+-------
 INTERNATIONAL INTERGROUP RELATIONS & SPECIAL EVENTS          |    12
 ASBESTOS                                                     |     8
 SUBCOMMITTEE ON SITE SELECTION (NYCC)                        |     3
 SUBCOMMITTEE ON REVENUE ENHANCEMENT (NYCC)                   |     3
 SUBCOMMITTEE ON ZONING, PLANNING & LANDMARKS (NYCC)          |     3
 SUBCOMMITTEE ON FRANCHISES, REVOCABLE CONSENTS & CONCESSIONS |     2
 SUBCOMMITTEE ON VETERANS AFFAIRS (NYCC)                      |     1
 PHYSICALLY CHALLENGED                                        |     1
 SELECT COMMITTEE ON IMMIGRATION (NYCC)                       |     1
 SUBCOMMITTEE ON WATERFRONT DEVELOPMENT (NYCC)                |     1
 SUBCOMMITTEE ON MEDICAL ALTERNATIVES (NYCC)                  |     1
(11 rows)

[hancush]

Playing in the Django shell trying to figure out which committees aren't referenced by posts, votes, etc.

Here's the script I came up with:

from opencivicdata.core.models import Organization

relations = ['actions', 'bills', 'children', 'votes', 'posts', 'memberships', 'memberships_on_behalf_of']

for org in Organization.objects.all():
    references = []
    for relation in relations:
        references.append(getattr(org, relation).all())
    if not any(references):
        print(org)

And the output:

WATERFRONT DEVELOPMENT
PARENTAL INVOLVEMENT IN EDUCATION
By the Committee on Rules, Privileges & Elections
MEDICAL TECHNOLOGY
UNKNOWN COMMITTEE
SELECT COMMITTEE ON REDISTRICTING
Joint Committee
PRIVATE SECTOR PARTICIPATION & MUNICIPAL POLICY
Unknown Body
CITY PLANNING
PENSIONS
By the Committee on Standards and Ethics
REVENUE ENHANCEMENT
PARKS CONCESSIONS
HOMELESS
HUMAN RIGHTS-86
SELECT COMMITTEE
CITY CONTRACTS-86
Legal and Government Affairs Division

[hancush]

Closing for now, we can always revisit if we decide to scrape less later.