search

cagov / data-infrastructure

CalData infrastructure
https://cagov.github.io/data-infrastructure
MIT License
5 stars 0 forks source link

Department list knowledge transfer #133

Closed ian-r-rose closed 1 year ago

ian-r-rose commented 1 year ago

I'm opening this issue as to track some running questions I have as I dive into CA department list data. Hoping to get @alannawil's thoughts on these, and we can fill in some answers for each

Q: Should the agency code be considered a unique identifier for each entity?

Q: These hierarchical layers "A", "B", "1", "2", and "3" are not very descriptive. Is there any actual terminology around these? Or legal meaning to them? Or are do they just indicate a local hierarchy? That is to say, does a "3" from one agency have anything in common with a "3" from another one?

Q: If an entity gets combined with another, or moved to be under the umbrella of a different agency, does it get a new code? Should we be attempting to track such changes?

Q: In the "Agency Profile" excel sheet, I see (at least) the following kinds of agency types:

Are these consistently used? Is there any mapping from these to "A", "B", "1", "2", "3"? Is establishing a taxonomy here a recipe for madness?

Q: Is the "Agency Profile" sheet available keyed by agency code?

Q: What does DOF USE ONLY mean? Is it distinct from DOF/SCO USE ONLY?

Q: What does DO NOT USE mean? Is it typically a defunct/deprecated entity, or can it mean something different?

Q: There are a number of "No subagency - DO NOT USE" entries. Fair enough, but those entries still have an agency code. Should that also be thrown away, or should entities underneath that level get that value for the subagency code?

alannawil commented 1 year ago

short answers for record keeping and can update - to discuss later @ian-r-rose

I'm opening this issue as to track some running questions I have as I dive into CA department list data. Hoping to get @alannawil's thoughts on these, and we can fill in some answers for each

Q: Should the agency code be considered a unique identifier for each entity? The unique identifier should be BU_code/DOF code/generically referred to as code. Agency code comes from SCO and annoyingly is not a unique identifier - can discuss.

Q: These hierarchical layers "A", "B", "1", "2", and "3" are not very descriptive. Is there any actual terminology around these? Or legal meaning to them? Or are do they just indicate a local hierarchy? That is to say, does a "3" from one agency have anything in common with a "3" from another one?

Yes there is - these are all predefined by DOF as such they are like the hierarchy of record - here is a link that describes the different levels, but that doesn't mean they are necessarily super sensible - https://dof.ca.gov/wp-content/uploads/Accounting/Policies_and_Procedures/Uniform_Codes_Manual/UCM_2-Organization_Codes-Introduction.pdf

Have talked to DOF about this and A, B, 1 are pretty good and consistent/comparable, but then it sort of falls apart in terms of comparability - the orgs are shoehorned to fit rather than deliberately allocated to a level. But it is still the "truth".

Q: If an entity gets combined with another, or moved to be under the umbrella of a different agency, does it get a new code? Should we be attempting to track such changes?

This is tricky ... the short answer is yes. However to deal w this is one of my big outstanding questions. Things do get moved in the coding and position in DOF. Previously stakeholders had flagged wanting complete org history - including code changes but I have been thinking about this and it gets messy fast. My increasing lean is just use DOF as given and have previous snapshots available but interested in your take. The most practical problem w this is around orgs that switch codes or collapse from a financial perspective, but are still operating as an entity. Unavoidable tension.

**Q: In the "Agency Profile" excel sheet, I see (at least) the following kinds of agency types:

Board Commission Council Authority Department Bureau Legislature Agency Office Program Univeristy Panel Are these consistently used? Is there any mapping from these to "A", "B", "1", "2", "3"? Is establishing a taxonomy here a recipe for madness?**

Nope not consistent and definitely recipe for madness-- these are user defined as is everything else in this list so we don't really know what's real/accurate or not. Can talk through more.

The agency type I/stakeholders were thinking as an attribute which can clarify is somewhat baked into hierarchy - if an agency is exec, constitutional or independent - which has been flagged by stakeholders but there is no real source on that.

Q: Is the "Agency Profile" sheet available keyed by agency code? Nope I wish - we can discuss why not- I have handmapped some of it because that list has nice alpha names and urls

Q: What does DOF USE ONLY mean? Is it distinct from DOF/SCO USE ONLY? DOF has previously said this is a "technical entity" - one that serves only finance purposes and is "empty" functionally. The ones w DOF/SCO also appear to be technical entities - I have on my running list to confirm that exact q of DOF/SCO use only - have previously asked SCO but they do not really know.

Q: What does DO NOT USE mean? Is it typically a defunct/deprecated entity, or can it mean something different?

Department of finance has previously said that this is deprecated. I have some doubts so want to straw man it for them to confirm this business rule.

Q: There are a number of "No subagency - DO NOT USE" entries. Fair enough, but those entries still have an agency code. Should that also be thrown away, or should entities underneath that level get that value for the subagency code? Ya I've thought this too - basically the B level is a waste of space, but I do lean towards not varying from DOF too much. But agree it does create confusion. Can discuss.

ian-r-rose commented 1 year ago

Thanks for the updates!

The unique identifier should be BU_code/DOF code/generically referred to as code. Agency code comes from SCO and annoyingly is not a unique identifier - can discuss.

What is a BU_code? When you say "Agency code comes from the SCO", does that mean the "A" level code?

Yes there is - these are all predefined by DOF as such they are like the hierarchy of record - here is a link that describes the different levels, but that doesn't mean they are necessarily super sensible - https://dof.ca.gov/wp-content/uploads/Accounting/Policies_and_Procedures/Uniform_Codes_Manual/UCM_2-Organization_Codes-Introduction.pdf

Thanks for the link, extremely helpful.

My increasing lean is just use DOF as given and have previous snapshots available but interested in your take. The most practical problem w this is around orgs that switch codes or collapse from a financial perspective, but are still operating as an entity.

Yeah, let's discuss. This is likely the most interesting part of the project from a data governance perspective. But I would hesitate to touch it unless we can get some kind of commitment from DOF /whoever to provide the data in a structured way (at least excel, preferably something better)

britt-allen commented 1 year ago

Organization codes link

Executive branch org chart link

Key Stakeholders

Agency profile comes from ca.gov, list is a little unreliable

BU = Business Unit, BU_code is a DOF code (4-digit), as close as UI as we'll get

ian-r-rose commented 1 year ago

Some useful links from a meeting with @alannawil and @britt-allen

Governer's org chart: https://www.gov.ca.gov/wp-content/uploads/2021/12/Exec-Branch-Org-Chart-1.14.22_fully-remediated.pdf?emrc=a3a7c5

DOF UCM: https://dof.ca.gov/accounting/accounting-policies-and-procedures/accounting-policies-and-procedures-uniform-codes-manual-organization-codes/

CSPS (California State Payroll System project) is doing a long project to update payroll system (FisCal) to be much more interoperable. Should we be doing this?

ian-r-rose commented 1 year ago

Ideas for carrots:

ian-r-rose commented 1 year ago

Technical entities

Here is a breakdown of counts of different kinds of technical entities (as defined by whether they are __ USE ONLY). The great majority are DOF USE ONLY, a few are DOF/SCO USE ONLY, and there are a couple of weird outliers corresponding to "State Allocation Board" and "Administrative Control Accounts".

restricted_use count
DOF/SCO USE ONLY 17
DOF USE ONLY 95
SCO/DGS USE ONLY 1
STO/SCO USE ONLY 1

Here is a table of all technical entities:

name agency_code subagency_code l1 l2 l3 parenthetical restricted_use
Office of Systems Integration 10 490 530 531     DOF/SCO USE ONLY
Payment of Interest on PMIA Loans 10 490 995       DOF USE ONLY
General Obligation Bonds 10 490 996       DOF USE ONLY
Mandated Local Costs, State 10 490 998       DOF USE ONLY
Miscellaneous Adjustments 10 490 999       DOF USE ONLY
Secretary for Business, Consumer Services, and Housing Agency 1000   1015       DOF USE ONLY
Secretary of State and Consumer Services 1000   1030       DOF USE ONLY
California African-American Museum 1000   1100 1105   Renum to 3105, 13/14 DOF USE ONLY
State Allocation Board 1000   1760 1820 1830   SCO/DGS USE ONLY
Payment of Interest on PMIA Loans 1000   1995       DOF USE ONLY
General Obligation Bonds 1000   1996       DOF USE ONLY
Mandated Local Costs, State 1000   1998       DOF USE ONLY
Miscellaneous Adjustments 1000   1999       DOF USE ONLY
Miscellaneous Adjustments - BT&H 1000   2810       DOF USE ONLY
Payment of Interest on PMIA Loans - BT&H 1000   2820       DOF USE ONLY
State Mandated Local Costs - BT&H 1000 2840       Renum from 2998 DOF USE ONLY
Secretary for Transportation Agency 2500 2020 2521       DOF USE ONLY
General Obligation Bonds - Transportation 2500 2020 2830     Renum from 2996 DOF USE ONLY
Statewide Distributed Costs 2500 2800       Renum from 2993 DOF USE ONLY
Miscellaneous Adjustments - T&C 2500   2925       DOF USE ONLY
Payment of Interest on PMIA Loans - T&C 2500   2930       DOF USE ONLY
General Obligation Bonds - T&C 2500   2935       DOF USE ONLY
State Mandated Local Costs - T&C 2500   2940       DOF USE ONLY
Secretary of the Natural Resources Agency 3000   3030       DOF USE ONLY
Payment of Interest on PMIA Loans – Resources 3000   3880       DOF USE ONLY
General Obligation Bonds – Resources 3000   3882       DOF USE ONLY
State Mandated Local Costs – Resources 3000   3884       DOF USE ONLY
Miscellaneous Adjustments – Resources 3000   3886       DOF USE ONLY
Secretary for Environmental Protection 3890   3895       DOF USE ONLY
Secretary for California Health & Human Services Agency 4000   4020       DOF USE ONLY
Set-Aside for Medi-Cal Long-Term Care Rates 4000   4268       DOF USE ONLY
Payment of Interest on PMIA Loans - H&HS 4000   5205       DOF USE ONLY
General Obligation Bonds - H&HS 4000   5206       DOF USE ONLY
Mandated Local Costs, State - H&HS 4000   5208       DOF USE ONLY
Miscellaneous Adjustments - H&HS 4000   5209       DOF USE ONLY
Secretary for Youth and Adult Corrections 5210   5230       DOF USE ONLY
Office of the Inspector General 5210   5235       DOF USE ONLY
Payment of Interest on PMIA Loans 5210   5995       DOF USE ONLY
General Obligation Bonds – Corrections and Rehab 5210   5996       DOF USE ONLY
Mandated Local Costs, State 5210   5998       DOF USE ONLY
Miscellaneous Adjustments 5210   5999       DOF USE ONLY
Office of the Secretary for Education, K-12 6000 6010 6050       DOF USE ONLY
Retirement Costs for Community Colleges 6000 6010 6305       DOF USE ONLY
Payment of Interest on PMIA Loans - K-12 6000 6010 6395       DOF USE ONLY
General Obligation Bonds - K-12 6000 6010 6396       DOF USE ONLY
Mandated Local Costs, State - K-12 6000 6010 6398       DOF USE ONLY
Miscellaneous Adjustments - K-12 6000 6010 6399       DOF USE ONLY
Secretary for Child Development and Education, Higher Education 6000 6013 6405       DOF USE ONLY
Payment of Interest on PMIA Loans - Hi Ed 6000 6013 6872       DOF USE ONLY
General Obligation Bonds - Hi Ed 6000 6013 6874       DOF USE ONLY
State Mandated Local Costs - Hi Ed 6000 6013 6876       DOF USE ONLY
Retirement Costs for Community Colleges - Hi Ed 6000 6013 6878       DOF USE ONLY
Miscellaneous Adjustments - Hi Ed 6000 6013 6879       DOF USE ONLY
Payment of Interest on PMIA Loans - Hi Ed 6000 6013 7995       DOF USE ONLY
General Obligation Bonds - Hi Ed 6000 6013 7996       DOF USE ONLY
Mandated Local Costs, State - Hi Ed 6000 6013 7998       DOF USE ONLY
Miscellaneous Adjustments - Hi Ed 6000 6013 7999       DOF USE ONLY
Higher Education Unallocated Salary Increase 6000 6025         DOF/SCO USE ONLY
Unallocated Salary Increase 6000 6025 7991       DOF USE ONLY
Secretary for Labor and Workforce Development 7000   7020       DOF USE ONLY
General Obligation Bonds – Labor and Workforce Development 7000   7399       DOF USE ONLY
Secretary for Government Operations Agency 7500   7511       DOF USE ONLY
Independent Citizens Redistricting Commission 8000 8010 8655       DOF USE ONLY
Pending New Organization 8000 8010 8990       DOF USE ONLY
State Mandated Local Costs 8000 8010 8994       DOF USE ONLY
Payment of Interest on PMIA Loans -General Government 8000 8010 8996       DOF USE ONLY
General Obligation Bonds - General Government 8000 8010 8998       DOF USE ONLY
Miscellaneous Adjustments - General Government 8000 8010 8999       DOF USE ONLY
Tax Relief/Local Government 8000 9010       Title change, 17/18  
Tax Relief 8000 9010 9100        
Sr Citizens Property Tax Assistance 8000 9010 9100 9110      
Sr Citizens Property Tax Deferral Program 8000 9010 9100 9120      
Sr Citizens Renters’ Tax Assistance 8000 9010 9100 9130      
Personal Property Tax Relief 8000 9010 9100 9140      
Homeowners Property Tax Relief 8000 9010 9100 9150      
Subventions for Open Space 8000 9010 9100 9160      
Payments to Local Government for Sales and Property Tax Revenue Loss 8000 9010 9100 9170      
Renters Tax Relief 8000 9010 9100 9180      
Substandard Housing 8000 9010 9100 9190      
Alternate Energy Tax Credit Refund 8000 9010 9100 9200      
Local Government Financing 8000 9010 9210        
Aid to Counties 8000 9010 9210 9220      
Aid to Education—K-12 8000 9010 9210 9230      
Aid to Community Colleges 8000 9010 9210 9240      
Aid to Cities 8000 9010 9210 9250      
Aid to Special Districts 8000 9010 9210 9260      
Local Agency Emergency Loan Program 8000 9010 9210 9270      
Local Agency Indebtedness Loan Program 8000 9010 9210 9280      
Trial Court Security – Court Construction 8000 9010 9285     New 14/15  
Trial Court Security – Judgeships 8000 9010 9286     New 17/18  
Rural Renaissance 8000 9010 9290        
Payment to Counties for Costs of Homicide Trials 8000 9010 9300        
Shared Revenue 8000 9010 9350        
Apportionment of Liquor License Fees 8000 9010 9350 9360      
Apportionment of Highway Properties Rental Receipts 8000 9010 9350 9370      
Apportionment of Off-Highway License Fees 8000 9010 9350 9380      
Apportionment of Federal Receipts for Flood Control Lands 8000 9010 9350 9390      
Apportionment of Federal Receipts for Forest Reserves 8000 9010 9350 9400      
Apportionment of Federal Receipts for Grazing Land 8000 9010 9350 9410      
Apportionment of Federal Potash Lease Rentals 8000 9010 9350 9420      
Apportionment of Mobilehome and Commercial Coach License Fees 8000 9010 9350 9425      
Apportionment of Motor Vehicle License Fees 8000 9010 9350 9430      
Apportionment of Cigarette Tax 8000 9010 9350 9440      
Apportionment of Highway Carriers’ Uniform Business Tax Fee 8000 9010 9350 9450      
Apportionment of Tideland Revenues 8000 9010 9350 9460      
Apportionment of Motor Vehicle Fuel Tax for County Roads 8000 9010 9350 9480      
Apportionment of Motor Vehicle Fuel Tax for City Streets 8000 9010 9350 9490      
Apportionment of Motor Vehicle Fuel Tax for County Roads and City Streets 8000 9010 9350 9500      
Apportionment of Motor Vehicle Fuel Tax to Counties and Cities for Streets and Highway Purposes 8000 9010 9350 9505      
Financial Aid to Local Agencies 8000 9010 9350 9510      
Apportionment of Local Agency Reimbursements 8000 9010 9350 9515      
Apportionment of Geothermal Resources Development 8000 9010 9350 9520      
Apportionments for Public Safety 8000 9010 9350 9530      
Apportionment of Traffic Congestion Relief Fund 8000 9010 9350 9533      
Apportionment of Local Transportation Funding 8000 9010 9350 9535      
Federal Revenue Sharing 8000 9010 9540        
Statewide Expenditures 8000 9040          
Payment of Interest on Pooled Money Investment Account Loans 8000 9040 9590        
General Obligation Bonds and Commercial Paper 8000 9040 9600        
Lease-Revenue Notes and Bonds 8000 9040 9610        
Tobacco Settlement Revenue Shortfall 8000 9040 9612        
Unenhanced Tobacco Bond Proceeds 8000 9040 9613        
Payment of Pension Obligation 8000 9040 9615        
Cash Management and Budgetary Loans 8000 9040 9620        
Interest Payments to the Federal Government 8000 9040 9625        
Health & Dental Benefits for Annuitants 8000 9040 9650        
Prefunding Health and Dental Benefits for Annuitants 8000 9040 9651        
Statewide Accounts Receivable Management Enhancements 8000 9040 9655     Abol. 17/18  
Budget Stabilization Account 8000 9040 9658        
Los Angeles County Medical Assistance Grant Program 8000 9040 9660        
Equity Claims and Settlements and Judgements 8000 9040 9670     Retitled 6/19  
Equity Claims by Victim Compensation Board and Department of General Services 8000 9040 9670 9671   Retitled 6/19  
Settlements and Judgments by Department of Justice 8000 9040 9670 9672      
San Francisco-Oakland Bay Bridge & I-880 Cypress Structure Disaster Fund 8000 9040 9673        
Construction & Repair of Local Streets and Roads 8000 9040 9675        
Refunds of Taxes, Licenses & Other Fees 8000 9040 9690        
Universal Telephone Service Program 8000 9040 9695        
Repayment of Loans Under Completed Programs 8000 9040 9700        
Working Capital Advances 8000 9040 9720        
State Clean Water Grants Revolving Fund 8000 9040 9720 9730      
Cooperative Personnel Services Revolving Fund 8000 9040 9720 9740      
County Formation Revolving Fund 8000 9040 9720 9750      
Augmentation for Mediterranean Fruit Fly 8000 9040 9790        
Augmentation for Employee Compensation 8000 9040 9800        
Reduction for Employee Compensation 8000 9040 9801       DOF/SCO USE ONLY
June to July Payroll Deferral 8000 9040 9802        
Contracts Impacted by Minimum Wage 8000 9040 9804        
Commission on Pay Equity 8000 9040 9805        
Payment of Specified Attorney Fees 8000 9040 9810        
Federal Levy of State Funds 8000 9040 9818        
Augmentation for Price Increases 8000 9040 9820        
Control/Replacement of Polychlorinated Biphenyls (PCBs) in State Facilities 8000 9040 9830     PCBs  
Augmentation for Contingencies or Emergency 8000 9040 9840        
Information Technology Equipment Management Program 8000 9040 9845        
Statewide Telecommunications System 8000 9040 9847        
Loans for Contingencies or Emergencies 8000 9040 9850        
Legislative Initiatives 8000 9040 9855        
Guarantee of Loan, State 8000 9040 9858        
Capital Outlay Planning and Studies Funding 8000 9040 9860        
Architecture Revolving Fund, Recovery of Uncharged Costs 8000 9040 9865        
Unallocated Cost-of-Living 8000 9040 9870        
Various Unallocated 8000 9040 9873        
General Fund Deficit Recovery Payments 8000 9040 9875        
Augmentation for Office of Administrative Law Services 8000 9040 9880        
Reserve for Liquidation of Encumbrances 8000 9040 9885        
Past Year Adjustments - Reserve for Encumbrances 8000 9040 9886        
Public School System Stabilization Account 8000 9040 9889       DOF/SCO USE ONLY
Reserve for Economic Uncertainties 8000 9040 9890        
Craig Brown v. US Dept of Health and Human Svs 8000 9040 9891       DOF/SCO USE ONLY
Supplemental Pension Payments 8000 9040 9892        
UC Retirement Adjustment 8000 9040 9893       DOF/SCO USE ONLY
Statewide Proposition 98 Reconciliation 8000 9040 9894       DOF USE ONLY
Petroleum Violation Escrow Account Program 8000 9040 9895        
Outer Continental Shelf Land Act. Section 8 8000 9040 9896     g  
Section 3.60 Rate Adjustments 8000 9040 9897       DOF USE ONLY
PERS General Fund Payment 8000 9040 9898       DOF/SCO USE ONLY
Information Technology Century Change and Alternative Procurements 8000 9040 9899        
Statewide General Administrative Expenditures 8000 9040 9900     Pro Rata  
Various Departments 8000 9040 9901       DOF USE ONLY
Alternative Procurements for Information Technology 8000 9040 9903        
Information Technology Century Change 8000 9040 9904        
Statewide Americans with Disabilities Act 8000 9040 9906       DOF/SCO USE ONLY
E-Government Projects 8000 9040 9907       DOF/SCO USE ONLY
Janitorial/Contract Services 8000 9040 9908       DOF/SCO USE ONLY
Health Insurance Portability & Accountability Act Compliance 8000 9040 9909       DOF/SCO USE ONLY
General Fund Credits from Federal Funds 8000 9040 9910     SWCAP  
Utilities Costs 8000 9040 9911       DOF/SCO USE ONLY
State Appropriations Limit 8000 9040 9912     Control Section 12  
Department of Justice Attorney Fees 8000 9040 9913       DOF/SCO USE ONLY
Postage Rate Increase 8000 9040 9914       DOF/SCO USE ONLY
Augmentation Authority per Control Section 31.70 8000 9040 9915       DOF USE ONLY
Department of Justice Legal Services 8000 9040 9917        
Child Development 8000 9040 9918        
Mandated Reductions 8000 9040 9920       DOF USE ONLY
Section 3.60(a) Rate Reductions 8000 9040 9930     a DOF USE ONLY
Section 3.60(b) Surplus Assets 8000 9040 9931     b DOF USE ONLY
PERS Payment Recovery 8000 9040 9934        
PERS Deferral 8000 9040 9935       DOF USE ONLY
PERS Surplus Asset Savings 8000 9040 9936       DOF USE ONLY
Federal Recoveries 8000 9040 9937       DOF USE ONLY
Antiterrorism Federal Reimbursements 8000 9040 9939       DOF USE ONLY
Estimated Unidentifiable Savings 8000 9040 9940       DOF USE ONLY
Revenue Bond Debt Refinancing 8000 9040 9941       DOF USE ONLY
Reorganizing & Downsizing State Government 8000 9040 9942       DOF USE ONLY
Control Section 14.95 8000 9040 9943       DOF USE ONLY
Unallocated Statewide Reductions 8000 9040 9945       DOF USE ONLY
Salary Reductions 8000 9040 9946       DOF USE ONLY
Past Years’ Disencumbrances 8000 9040 9947       DOF USE ONLY
Operating Expenses and Equipment Savings 8000 9040 9948       DOF USE ONLY
Local District Projects Savings 8000 9040 9949       DOF USE ONLY
Hiring Freeze Savings 8000 9040 9950       DOF USE ONLY
Alternate Retirement Program 8000 9040 9955        
Alternate Retirement Program 8000 9040 9956       DOF USE ONLY
Vacant Position Savings per Section 31.60 8000 9040 9960       DOF USE ONLY
Lease Revenue Debt Service 8000 9040 9966       DOF USE ONLY
California Fiscal Information System Use Only 8000 9040 9970        
Premium Offset for Employer-Paid Life Insurance 8000 9040 9980       DOF USE ONLY
Early Retirement Program 8000 9040 9985       DOF USE ONLY
Miscellaneous Control Accounts 8000 9040 9990       DOF/SCO USE ONLY
Administrative Control Accounts 8000 9040 9991       STO/SCO USE ONLY
SCO State Government Reporting 8000 9040 9992        
Adjustment to Reconcile to Controller 8000 9040 9996       DOF/SCO USE ONLY
Control Sections for Budget Enactment Process 8000 9040 9998       DOF USE ONLY
Major Revenue 8000 9999         DOF/SCO USE ONLY
Major Revenue 8000 9999 1       DOF USE ONLY
Major Policy Revenue 8000 9999 2       DOF USE ONLY
Prop 98 Exp Adjustment Related to Tax Reductions 8000 9999 3       DOF USE ONLY
alannawil commented 1 year ago

Nice!

Technical entities also should include 9000 coded entities - was a business rule they also have us. I think teeing up a couple examples for them of where this seem a little fishy will be helpful - for ex the AfAm history museum - this is not a technical entity despite the rule they gave us - it seems it is mismarked and should be just DO NOT USE as it is deprecated. Unless they are using old codes as technical entities which gets definitionally very annoying and confusing for users.

Setting up a google doc for email drafting here

I'm working on some documentation for this today to ease navigation to key docs and such, will flag shortly

Also found the DataSF list which we were initially trying to model after but is very much more orderly https://data.sfgov.org/City-Management-and-Ethics/Reference-Department-Code-List/j2hz-23ps

alannawil commented 1 year ago

@ian-r-rose just cc'd you on an email reply to Michele Hawkins - she is a very helpful SCO contact - more at the analyst level. Basically she sends that file to us every month for the last couple months - it was easy for them to pull and send via email - CSU entities are on a separate sheet. They can change the specifications if we end up needing. As we figure this out and get someone to maintain a crosswalk they could move from emailing that to use to a SFTP or maybe even something else.

ian-r-rose commented 1 year ago

Technical entities also should include 9000 coded entities - was a business rule they also have us. I think teeing up a couple examples for them of where this seem a little fishy will be helpful - for ex the AfAm history museum - this is not a technical entity despite the rule they gave us - it seems it is mismarked and should be just DO NOT USE as it is deprecated. Unless they are using old codes as technical entities which gets definitionally very annoying and confusing for users.

Thanks for the info, I'll update the tables to include 9000-level as an additional rule for selecting technical entities.

ian-r-rose commented 1 year ago

@ian-r-rose just cc'd you on an email reply to Michele Hawkins - she is a very helpful SCO contact - more at the analyst level. Basically she sends that file to us every month for the last couple months - it was easy for them to pull and send via email - CSU entities are on a separate sheet. They can change the specifications if we end up needing. As we figure this out and get someone to maintain a crosswalk they could move from emailing that to use to a SFTP or maybe even something else.

Great, thanks! Though I don't see that email, perhaps there was a typo in my address?

alannawil commented 1 year ago

@ian-r-rose just cc'd you on an email reply to Michele Hawkins - she is a very helpful SCO contact - more at the analyst level. Basically she sends that file to us every month for the last couple months - it was easy for them to pull and send via email - CSU entities are on a separate sheet. They can change the specifications if we end up needing. As we figure this out and get someone to maintain a crosswalk they could move from emailing that to use to a SFTP or maybe even something else.

Great, thanks! Though I don't see that email, perhaps there was a typo in my address?

lol yep my b!! Just actually added you thanks for the flag

ian-r-rose commented 1 year ago

Thanks! Also, I've gone ahead and updated the technical entities table above to include non-deprecated 9000-level entities.

ian-r-rose commented 1 year ago

I think most of my urgent questions for this issue have been answered, thanks @alannawil! I'm going to close this issue and follow-up work can be planned in new issues.

alannawil commented 1 year ago

Sounds good @ian-r-rose I am going to press close with comment and hope this stays closed 😝 I'll finish drafting email to DOF shortly. Cleaned up the g-drive and added a readme to help nav and also flag key resources.

And agreed - think as we move we should have one on integrating w SCO and also didn't touch on the integration of a couple other key attributes (alpha names, domains) which we can open separate issues on when the time is right