srt1
commented
6 years ago From: Stephen R Tibbets (CENSUS/CES FED) Sent: Friday, June 1, 2018 10:29 AM To: Andrew Foote (CENSUS/CES FED); Erika McEntarfer (CENSUS/CES FED); Joyce Key Hahn (CENSUS/CES FED); Warren, Lawrence Fujio (CENSUS/CES FED) Subject: Re: Comments on PSEO data scehema
In-line responses below:
From: Andrew Foote (CENSUS/CES FED) Sent: Friday, June 1, 2018 9:07 AM To: Stephen R Tibbets (CENSUS/CES FED); Erika McEntarfer (CENSUS/CES FED); Joyce Key Hahn (CENSUS/CES FED); Warren, Lawrence Fujio (CENSUS/CES FED) Subject: Re: Comments on PSEO data scehema
Stephen -
Thank you for your feedback on this. I think a lot of the comments on variables would be helped along by us having a schema.
For instance, the institution_id variable is sourced from the Department of Education (it is technically the Office of Post-Secondary Education ID (OPEID), and so it is a unique (national) identifier in the data. They don't have leading zeros in the reports I have, but we can add leading zeros to make it uniform.
It's good to know they are nationally unique, so that larger issue should not be one. If they do not use a leading zero, we probably should not - you never know, if we assume it is 6 digits, and they assign a number more than a million we're stuck. We should describe it as a numeric variable in the schema, rather than character - you can't really tell otherwise from a csv which would be appropriate.
As for the CIP codes, I did some poking to see if the ACS has a naming convention (because they have field of study), but it appears that they use a completely different coding scheme for field of study, which is not at all correlated with the CIP codes. Even on the NCES website, the 4 digit CIP codes are just referred to as CIP codes, so we could change that to variable to "cip" and it would be clear to people.
"CIP" is probably good enough, but I don't have a strong opinion on that one.
As for having full labels on output files, that's fine to relegate to schema - I put them up there initially because we didn't really have any infrastructure built.
I do think the csv's should not have labels. One option is to provide supplemental files in XLSX format as well, and include labels on those. They can be created directly from SAS (assuming we get 9.4, eventually - or we can do it manually for these early betas).
As for the file names, is one option having an additional variable that indicates the vintage the data was created on. For instance, the current data would have an 03262018 vintage tag, and new cells will have a different tag. We could have that on the internal version if that makes more sense.
One other small note - all the earnings data will be brought up to current year dollars when we update the tables. Right now it is in 2016 dollars, but it will need to be updated. One important thing to note is that the data will always (before pre-release) be in 2016 dollars, because the histogram is in 2016 dollars. Then everything will be transformed into current year dollars.
We typically don't include the "vintage" info in the release data itself, rather providing the info in an accompanying metadata file. Given the data is so small, we could add a variable tagging it directly. If we do roll up multiple cohorts into the same file, we can do a little thinking about how best to tag them. If records from this initial release later get cast into future dollars, those numbers should be tagged differently than this first release, but potentially we would want them to be tagged differently from future cohorts. We can do some brainstorming about that.
You are right about earnings - we should round to the whole dollar.
Easy enough
And we can add a separate flag for suppressed, rather than my solution, which isn't really one. I think each cell should have a status flag (we release all variables or none).
You've just combined two separate concepts into a single variable, in a way. Sometimes better to break it out - not a big deal.
Andrew Foote, Economist, LEHD Center for Economic Studies U.S. Census Bureau Office 301.763.7469
Cell 202.384.4784
Room 5K058E
Andrew.Foote@census.gov
census.gov From: Stephen R Tibbets (CENSUS/CES FED) Sent: Thursday, May 31, 2018 4:48:51 PM To: Andrew Foote (CENSUS/CES FED); Erika McEntarfer (CENSUS/CES FED); Joyce Key Hahn (CENSUS/CES FED); Warren, Lawrence Fujio (CENSUS/CES FED) Subject: Comments on PSEO data scehema
All,
I did have some thoughts on things we might consider adjusting in the next PSEO release:
- Current file names are graduate_earnings_utsys.csv and graduate_earnings_all.csv.
- the "utsys" file is not named in a way that makes it obvious what the next release (in two or three years) would be. If we are just replacing the exact file with an expanded one containing the whole universe (old plus new cohort), it should be vintaged so we can distinguish between the old and new file - even though the "old" data will be exactly the same as the previous release.
- Will the "old" data be at least brought into current dollars for comparability?
- the "all" file is presumably a rollup of all of the other files. Since adding a new state (CO) will change this file, the old one and the new one should be in separate releases - similar to note above.
- There should be a metadata file describing the contents of the release (see https://lehd.ces.census.gov/data/j2j/R2017Q4/j2j/ak/version_j2j.txt)
- There should be some kind of directory structure under PSEO, possibly with a "Current" if we want to point to the latest one of multiple (e.g., old and new "all" or "utsys" files). We should ask Heath what he would recommend on that.
- I don't like having full labels contained in the output files - I'd prefer the labels were relegated to a data schema. I assume the thinking was that these files are small, so it doesn't matter. But even so, we should have a formal schema for the characteristic variables.
- Institution_ID
- Where does the "institution_id" come from? Is it a state assigned variable?
- I notice they have different lengths - do they not use leading zeroes?
- If is state assigned, can the identifiers crash across state systems? If so, we might want to pad it with something so that they are nationally unique (a la the SEIN).
- degcip_4dig is a very quirky variable name to me. Is there Census standard on what variable name is used for the CIP code?
- Why do we provide earnings precision to the nearest penny? Seems a little overboard to me - all QWI and J2J earnings measures are rounded to the nearest dollar.
- I might add a separate flag for "released" vs. "suppressed or not avail" - which is more-or-less what I assume the cellcount=-1 is. In QWI/J2J each variable has a "status" flag attached. We may not need that here exactly, if the percentiles are all released together, but I would have a flag indicating when things are missing.
Our current full QWI/J2J data schema is here:
https://lehd.ces.census.gov/data/schema/V4.2.0/lehd_public_use_schema.html
Given how simple this file is, it should not be a big deal to formally document, but we should do it. Please let me know if you have any thoughts on my suggestions.
Thanks, Stephen
larsvilhuber
andrewfoote
jodyhoonstarr
We want to include formal documentation of the PSEO outputs in the data schema. Note, I have made some comments to Andrew about the first release, suggesting some adjustments for future releases. Not sure what schema number we want to call this, or if we want a separate schema. It's a pretty simple release, though the future iterations are a little different to QWI/J2J.
See:
https://lehd.ces.census.gov/data/pseo_beta.html
https://lehd.ces.census.gov/data/pseo/