biocaddie / prototype_issues

Used to report and track bioCADDIE prototype issues
3 stars 5 forks source link

Ingest from DATS problems #256

Closed altergc closed 6 years ago

altergc commented 7 years ago

Inconsistencies between ICPSR's DATS and DataMed.docx Sanda found a number of problems with the records that were ingested from ICPSR DATS. In the attached file she compares one record in DataMED to the way that it appears in the JSON file.

yul129 commented 7 years ago

By comparing transformed ICPSR JSON file (attached at the bottom of this email) with the ICPSR website for ICPSR 3 (https://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3?searchSource=revise&q=icpsr+3) 1.The description, license and creators are all consistent 2.The Grant No. information is under "grant" in DataMed, not under "acknowledges" 3.The availability field need to be re-mapped, Burak and i will work on it today.

{ "dataset": { "availability": "available", "types": [ "other" ], "aggregation": "collection of dataset", "privacy": "not applicable", "refinement": "curated", "dateReleased": "19840706T000000+0000", "creators": [ "Inter-university Consortium for Political and Social Research" ], "ID": "3", "title": " Historical, Demographic, Economic, and Social Data: The United States, 1790-1970", "keywords": [ "census data", "counties", "demographic characteristics", "economic conditions", "eighteenth century", "families", "historical data", "marriage", "national identity", "nineteenth century", "occupations", "political change", "population", "population size", "religion", "states (USA)", "twentieth century" ], "description": "Detailed county and state-level ecological or descriptive data for the United States for the years 1790 to 1970 are contained in this collection. These data files contain extensive information about the social and political character of the United States, including a breakdown of population by state, race, nationality, number of families, size of the family, births, deaths, marriages, occupation, religion, and general economic conditions. Though not complete over the full time span of this study, statistics are available on such diverse subjects as total numbers of newspapers and periodicals, total capital invested in manufacturing, total numbers of educational institutions, total number of churches, taxation by state, and land surface area in square miles." }, "datasetDistributions": [ { "format": "DOCUMENT", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/dldocs2?study=3&path=ICPSR" }, { "format": "SAS", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=sas&path=ICPSR" }, { "format": "SPSS", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=spss&path=ICPSR" }, { "format": "STATA", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=stata&path=ICPSR" }, { "format": "ASCII", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=ascii&path=ICPSR" }, { "format": "DELIMITED", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=delimited&path=ICPSR" } ], "dataRepository": { "ID": "SCR:003194", "name": "Inter-university Consortium for Political and Social Research", "abbreviation": "ICPSR", "homePage": "http://www.icpsr.umich.edu/" }, "organization": { "ID": "SCR:011668", "name": "University of Michigan", "abbreviation": "UMich", "homePage": "http://www.umich.edu/" }, "dataStandard": { "version": "2.5", "name": "ddi:codebook:2_5 http://www.ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/codebook.xsd" }, "citations": [ { "ID": [ "icpsr:990", "issn:0161-5440" ], "title": "A Methodology for Revising Estimates: Female Market Participation in the United States Before 1940", "authors": [ "Abel, Marjorie", "Folbre, Nancy" ] }, { "ID": [ "icpsr:92981" ], "title": "Cultural Diversity and Economic Growth: Evidence from the US During the Ages of Mass Migration", "authors": [ "Ager, Philipp, Brückner, Markus" ] }, { "ID": [ "icpsr:1318", "issn:0070-3370" ], "title": "The human ecology of tornadoes", "authors": [ "Aguirre, B.E.", "Saenz, Rogelino", "Edmiston, John", "Yang, Nan", "Agramonte, Elsa", "Stuart, Dietra L." ] }, { "ID": [ "icpsr:1465", "issn:0022-3433" ], "title": "Military Conversion Policies in the USA: 1940s and 1990s", "authors": [ "Alexander, J. Davidson" ] }, { "ID": [ "icpsr:1508", "issn:0145-5532" ], "title": "From the Populist era to the New Deal: A study of partisan realignment in Washington State, 1889-1950", "authors": [ "Allen, Howard W.", "Austin, Erik W." ] }, { "ID": [ "icpsr:68045" ], "title": "Race, Class, and the Death Penalty: Capital Punishment in American History", "authors": [ "Allen, Howard W.", "Clubb, Jerome M." ] }, { "ID": [ "icpsr:92400", "issn:0022-0507" ], "title": "Time on the ladder: Career mobility in agriculture, 1890-1938", "authors": [ "Alston, Lee J.", "Ferrie, Joseph P." ] }, { "ID": [ "icpsr:92733", "issn:1545-2921" ], "title": "Electoral behavior of US counties: A panel data approach", "authors": [ "Anatolyev, Stanislav" ] }, { "ID": [ "icpsr:86433", "issn:0145-5532" ], "title": "Did railroads induce or follow economic growth? Urbanization and population growth in the American Midwest, 1850-1860", "authors": [ "Atack, Jeremy", "Bateman, Fred", "Haines, Michael", "Margo, Robert A." ] }, { "ID": [ "icpsr:2436", "issn:0730-8884" ], "title": "The Deskilling Controversy", "authors": [ "Attewell, Paul" ] }, { "ID": [ "icpsr:2706", "issn:0038-0407" ], "title": "Schooling all the masses: Reconsidering the origins of American schooling in the postbellum era", "authors": [ "Baker, David P." ] }, { "ID": [ "icpsr:40038" ], "title": "The Culture of Honor and Southern Homicide", "authors": [ "Baller, Robert", "Zevenbergen, Matthew P.", "Messner, Steven F." ] }, { "ID": [ "icpsr:2800", "issn:0021-9347" ], "title": "A critical inquiry of enslaved African females and the antebellum hospital experience", "authors": [ "Bankole, Katherine Olukemi" ] }, { "ID": [ "icpsr:3212", "issn:0022-0027" ], "title": "Trends in American Major War and Peace", "authors": [ "Beer, Francis A." ] }, { "ID": [ "icpsr:92396", "issn:0166-0462" ], "title": "Population growth in U.S. counties, 1840-1990", "authors": [ "Beeson, Patricia E.", "DeJong, David N.", "Troesken, Werner" ] }, { "ID": [ "icpsr:92451", "issn:1432-9840" ], "title": "Historical land-cover conversion (1665-1820) in the Choptank Watershed, eastern United States", "authors": [ "Benitez, Jorge A.", "Fisher, Thomas R." ] }, { "ID": [ "icpsr:120303" ], "title": "Essays on State Dependence in the Government Spending Multiplier", "authors": [ "Biolsi, Christopher" ] }, { "ID": [ "icpsr:3881", "issn:0145-5532" ], "title": "Group Enmity and Accord: The Commercial Press in Three American Cities", "authors": [ "Blau, Judith R." ] }, { "ID": [ "icpsr:93031", "issn:0049-089X" ], "title": "The expansion of religious affiliation: An explanation of the growth of church participation in the United States 1850-1930", "authors": [ "Blau, Judith R.", "Land, Kenneth C.", "Redding, Kent" ] }, { "ID": [ "icpsr:3884", "issn:0731-1214" ], "title": "Spatial processes and the duality of church and faith: a Simmelian perspective on U.S. denominational growth, 1900-1930", "authors": [ "Blau, Judith R.", "Redding, Kent", "Davis, Walter R.", "Land, Kenneth C." ] }, { "ID": [ "icpsr:3885", "issn:0884-8971" ], "title": "Ethnocultural cleavages and the growth of church membership in the United States, 1860-1930", "authors": [ "Blau, Judith R.", "Redding, Kent", "Land, Kenneth C." ] }, { "ID": [ "icpsr:112530", "issn:1945-7790" ], "title": "Malaria eradication in the Americas: A retrospective analysis of childhood exposure", "authors": [ "Bleakley, Hoyt" ] }, { "ID": [ "icpsr:87451", "issn:0034-6535" ], "title": "Chronic disease burden and the interaction of education, fertility, and growth", "authors": [ "Bleakley, Hoyt", "Lange, Fabian" ] }, { "ID": [ "icpsr:4276", "issn:0022-0515" ], "title": "The Economics of Immigration", "authors": [ "Borjas, George J." ] }, { "ID": [ "icpsr:40582", "issn:1071-4839" ], "title": "The drug war is the new Jim Crow", "authors": [ "Boyd, Graham" ] }, { "ID": [ "icpsr:33600" ], "title": "To Be 'Read By the Whole People': Press, Party, and Public Sphere in the United States, 1790-1840", "authors": [ "Brooke, John L." ] }, { "ID": [ "icpsr:92403", "issn:0002-8282" ], "title": "Fundamentals, panics, and bank distress during the Depression", "authors": [ "Calomiris, Charles W.", "Mason, Joseph R." ] }, { "ID": [ "icpsr:5760", "issn:0022-2445" ], "title": "Determinants of Household Headship", "authors": [ "Carliner, Geoffrey" ] }, { "ID": [ "icpsr:5896", "issn:0161-5440" ], "title": "Fixing the Facts: Editing of the 1880 U.S. Census of Occupations with Implications for Long-Term Labor-Force Trends and the Sociology of Official Statistics", "authors": [ "Carter, Susan B.", "Sutch, Richard" ] }, { "ID": [ "icpsr:112613" ], "title": "Did suburbanization cause residential segregation? Evidence from U.S. metropolitan areas", "authors": [ "Chatterjee, Boishampayan" ] }, { "ID": [ "icpsr:6663" ], "title": "Demographic and compositional change", "authors": [ "Clubb, Jerome M.", "Austin, Erik W.", "Traugott, Michael W." ] }, { "ID": [ "icpsr:6928", "issn:0003-1224" ], "title": "Formal Organization and the Fate of Social Movements: Craft Association and Class Alliance in the Knights of Labor", "authors": [ "Conell, Carol", "Voss, Kim" ] }, { "ID": [ "icpsr:7212", "issn:0022-3433" ], "title": "A Causal Analysis of the Defense Expenditures of the USA and the USSR", "authors": [ "Correa, Hector", "Kim, Ji-Won" ] }, { "ID": [ "icpsr:7231" ], "title": "Unequal at Birth: A Long-Term Comparison of Income and Birth Weight", "authors": [ "Costa, Dora L." ] }, { "ID": [ "icpsr:7271", "issn:0022-0027" ], "title": "War and American Democracy: Electoral Costs of the Last Five Wars", "authors": [ "Cotton, Timothy Y.C." ] }, { "ID": [ "icpsr:7383", "issn:0895-5638" ], "title": "Transportation improvements and land values in the antebellum United States: A hedonic approach", "authors": [ "Craig, Lee A.", "Palmquist, Raymond B.", "Weiss, Thomas" ] }, { "ID": [ "icpsr:92735" ], "title": "Did Big Government's Largesse Help the Locals? The Implications of WWII Spending for Local Economic Activity, 1939-1958", "authors": [ "Cullen, Joseph", "Fishback, Price V." ] }, { "ID": [ "icpsr:8001" ], "title": "Frontier America, 1800-1840: A Comparative Demographic Analysis of the Frontier Process", "authors": [ "Davis, James E." ] }, { "ID": [ "icpsr:8027", "issn:0192-513X" ], "title": "Childless and Single-Childed Women in Early Twentieth-Century America", "authors": [ "Davis, Nancy J." ] }, { "ID": [ "icpsr:8182", "issn:0037-7732" ], "title": "Mass Communication and Social Change", "authors": [ "DeFleur, Melvin L." ] }, { "ID": [ "icpsr:119578" ], "title": "Party Formation in the United States", "authors": [ "DeWitt, Darin Dion" ] }, { "ID": [ "icpsr:92974" ], "title": "Why Does Financial Development Matter? The United States from 1900 to 1940", "authors": [ "Dehejia, Rajeev", "Lleras-Muney, Adriana" ] }, { "ID": [ "icpsr:9290", "issn:0278-5927" ], "title": "Transplanted to the Rose City: The Creation of East European Jewish Community in Portland, Oregon", "authors": [ "Eisenberg, Ellen" ] }, { "ID": [ "icpsr:10371", "issn:0034-6535" ], "title": "Narrowing the Black-White gap in child literacy in 1910: The roles of school inputs and family inputs", "authors": [ "Fishback, Price V.", "Baskin, John H." ] }, { "ID": [ "icpsr:92454" ], "title": "The Welfare of Children During the Great Depression", "authors": [ "Fishback, Price V.", "Haines, Michael R.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92395", "issn:0014-4983" ], "title": "The impact of the New Deal on black and white infant mortality in the South", "authors": [ "Fishback, Price V.", "Haines, Michael R.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92452", "issn:0022-0507" ], "title": "Did New Deal grant programs stimulate local economies? A study of federal grants and retail sales during the Great Depression", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:93578" ], "title": "Do Federal Programs Affect Internal Migration? The Impact of New Deal Expenditures on Mobility During the Great Depression", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92975" ], "title": "The Impact of New Deal Expenditures on Local Economic Activity: An Examination of Retail Sales, 1929-1939", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92731" ], "title": "The Origins of Modern Housing Finance: The Impact of Federal Housing Programs During the Great Depression", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:10372", "issn:0033-5533" ], "title": "Did Workers by for the Passage of Workers' Compensation Laws", "authors": [ "Fishback, Price V.", "Kantor, Shawn Everett" ] }, { "ID": [ "icpsr:10373", "issn:0022-0507" ], "title": "The durable experiment: State insurance of workers' compensation risk in the early twentieth century", "authors": [ "Fishback, Price V.", "Kantor, Shawn Everett" ] }, { "ID": [ "icpsr:92399", "issn:0014-4983" ], "title": "Can the New Deal's three Rs be rehabilitated? A program-by-program, county-by-county analysis", "authors": [ "Fishback, Price V.", "Kantor, Shawn", "Wallis, John Joseph" ] }, { "ID": [ "icpsr:10507", "issn:0038-4038" ], "title": "Electoral incentives, public policy, and the New Deal realignment", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:92394", "issn:0014-4983" ], "title": "Population, land, economic conditions, and the allocation of New Deal spending", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:10508", "issn:0095-2583" ], "title": "The value of the vote: A model and test of the effects of turnout on distributive policy", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:92976", "issn:0022-3808" ], "title": "Voter influence and big policy change: The positive political economy of the New Deal", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:10955", "issn:0037-7732" ], "title": "Risk and Recreancy: Weber, the Division of Labor, and the Rationality of Risk Perceptions", "authors": [ "Freudenburg, William R." ] }, { "ID": [ "icpsr:11227", "issn:0161-5440" ], "title": "Metropolitan Classification for Census Years before World War II", "authors": [ "Gardner, Todd" ] }, { "ID": [ "icpsr:11485", "issn:0145-5532" ], "title": "Populism, Race, and Political Interest in Virginia", "authors": [ "Gerteis, Joseph" ] }, { "ID": [ "icpsr:86772", "issn:0022-0507" ], "title": "The uneven rise of American public schools to 1850", "authors": [ "Go, Sun", "Lindert, Peter" ] }, { "ID": [ "icpsr:12030", "issn:0020-8833" ], "title": "Kondratieff Wave as War Cycles", "authors": [ "Goldstein, Joshua S." ] }, { "ID": [ "icpsr:12214", "issn:0037-7732" ], "title": "The Changing Structure of Property Crime in an Affluent Society", "authors": [ "Gould, Leroy C." ] }, { "ID": [ "icpsr:35884", "issn:0022-1953" ], "title": "From local to national political cultures: Social capital and civic organization in the Great Plains", "authors": [ "Gutmann, Myron P.", "Pullum, Sara M." ] }, { "ID": [ "icpsr:12994" ], "title": "Fertility and Occupation: Population Patterns in Industrialization", "authors": [ "Haines, Michael" ] }, { "ID": [ "icpsr:93556" ], "title": "Spatial aspects of the American fertility transition in the Nineteenth Century", "authors": [ "Haines, Michael R.", "Hacker, J. David" ] }, { "ID": [ "icpsr:92453" ], "title": "The Puzzle of the Antebellum Fertility Decline in the United States: New Evidence and Reconsideration", "authors": [ "Haines, Michael R.", "Hacker, J. David" ] }, { "ID": [ "icpsr:13190", "issn:0003-1224" ], "title": "Revival Religion and Antislavery Politics", "authors": [ "Hammond, John L." ] }, { "ID": [ "icpsr:13331", "issn:0002-1482" ], "title": "Beyond the marginality thesis: the acquisition and loss of land by African Americans in Georgia, 1880-1930", "authors": [ "Hargis, Peggy G." ] }, { "ID": [ "icpsr:13332", "issn:0022-1953" ], "title": "The 'low-country advantage' for African-Americans in Georgia, 1880-1930", "authors": [ "Hargis, Peggy G.", "Horan, Patrick M." ] }, { "ID": [ "icpsr:13564", "issn:0020-8183" ], "title": "Openings in the Wall: Transnational Migrants, Labor Unions, and U.S. Immigration Policy", "authors": [ "Haus, Leah" ] }, { "ID": [ "icpsr:14016", "issn:0161-5440" ], "title": "The Census, Estimation Biases, and Female Labor-Force Participation Rates in 1880 Colorado", "authors": [ "Herr, Elizabeth" ] }, { "ID": [ "icpsr:92734", "issn:0362-9805" ], "title": "Third parties, elections, and roll-call votes: The populist party and the late nineteenth-century U.S. congress", "authors": [ "Hirano, Shigeo" ] }, { "ID": [ "icpsr:96012", "issn:0145-5532" ], "title": "Resisting redemption: The Republican vote in Georgia in 1876", "authors": [ "Hogan, Richard" ] }, { "ID": [ "icpsr:14702", "issn:0022-2445" ], "title": "The Provider Role: Its Meaning and Measurement", "authors": [ "Hood, Jane C." ] }, { "ID": [ "icpsr:15551", "issn:0360-0572" ], "title": "Gender Inequality and Higher Education", "authors": [ "Jacobs, Jerry A." ] }, { "ID": [ "icpsr:15649", "issn:0898-588X" ], "title": "Prelude to progressivism: Party decay, populism, and the doctrine of 'free and unrestricted competition' in American antitrust policy, 1890-1897", "authors": [ "James, Scott C." ] }, { "ID": [ "icpsr:92971" ], "title": "Striking the Roots of Crime: The Impact of the New Deal on Criminal Activity", "authors": [ "Johnson, Ryan S.", "Kantor, Shawn", "Fishback, Price V." ] }, { "ID": [ "icpsr:16242" ], "title": "Education and Social Change in Nineteenth-Century Massachusetts", "authors": [ "Kaestle, Carl F.", "Vinovskis, Maris A." ] }, { "ID": [ "icpsr:16485", "issn:0037-7732" ], "title": "The Theory of Ecological Expansion: An Empirical Test", "authors": [ "Kasarda, John D." ] }, { "ID": [ "icpsr:16653", "issn:0037-7732" ], "title": "The Family Wage System in Pennsylvania's Anthracite Region: 1850-1900", "authors": [ "Keil, Thomas J.", "Usui, Wayne M." ] }, { "ID": [ "icpsr:16795", "issn:0037-7732" ], "title": "Unemployment and Protest in the United States, 1890-1940: A Methodological Critique and Research Note", "authors": [ "Kerbo, Harold R.", "Shaffer, Richard A." ] }, { "ID": [ "icpsr:17170", "issn:0022-2445" ], "title": "Marital Complaints, Demographic Characteristics, and Symptoms of Mental Distress in Divorce", "authors": [ "Kitson, Gay C.", "Sussman, Marvin B." ] }, { "ID": [ "icpsr:17264" ], "title": "The Third Electoral System, 1853-1892: Parties, Voters, and Political Cultures", "authors": [ "Kleppner, Paul" ] }, { "ID": [ "icpsr:17712", "issn:0070-3370" ], "title": "The elderly who live alone in the United States: Historical perspectives on household change", "authors": [ "Kramarow, Ellen A." ] }, { "ID": [ "icpsr:18097", "issn:0081-1750" ], "title": "On the large-sample estimation of regression models with spatial- or network-effects terms: A two-stage least squares approach", "authors": [ "Land, Kenneth C.", "Deane, Glenn" ] }, { "ID": [ "icpsr:18098", "issn:0003-1224" ], "title": "Religious Pluralism and Church Membership: A Spatial Diffusion Model", "authors": [ "Land, Kenneth C.", "Deane, Glenn", "Blau, Judith R." ] }, { "ID": [ "icpsr:18111", "issn:0070-3370" ], "title": "Generation, ethnicity, and marriage: Historical patterns in the northern United States", "authors": [ "Landale, Nancy S.", "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:18459", "issn:0014-4983" ], "title": "Farm value and retirement of farm owners in early-twentieth-century America", "authors": [ "Lee, Chulhee" ] }, { "ID": [ "icpsr:18817", "issn:0021-8723" ], "title": "Draft Evasion in the North during the Civil War, 1863-1865", "authors": [ "Levine, Peter" ] }, { "ID": [ "icpsr:19111" ], "title": "Fertility and Scarcity in America", "authors": [ "Lindert, Peter" ] }, { "ID": [ "icpsr:19350", "issn:0020-8183" ], "title": "Dividend Government and U.S. Trade Policy: Theory and Evidence", "authors": [ "Lohmann, Susanne", "O'Halloran, Sharyn" ] }, { "ID": [ "icpsr:19560", "issn:0022-3433" ], "title": "Empire by Invitation? The United States and Western Europe, 1945-1952", "authors": [ "Lundestad, Geir" ] }, { "ID": [ "icpsr:20175", "issn:0145-5532" ], "title": "Immigrant Support for the American Socialist Party, 1912 and 1920", "authors": [ "Marks, Gary", "Burbank, Matthew" ] }, { "ID": [ "icpsr:20443", "issn:0038-4941" ], "title": "The Changing Geographic Structure of Black and White Segregation in the United States", "authors": [ "Massey, Douglas S.", "Hajnal, Zoltan L." ] }, { "ID": [ "icpsr:20593", "issn:0038-4941" ], "title": "Demographic and Economic Determinants of United States Income Inequality", "authors": [ "Maxwell, Nan L." ] }, { "ID": [ "icpsr:20706", "issn:0730-8884" ], "title": "'Government by Injunction': The U. S. Judiciary and Strike Action in the Late 19th and Early 20th Centuries", "authors": [ "McCammon, Holly J." ] }, { "ID": [ "icpsr:20860", "issn:0037-7732" ], "title": "Fiscal Politics and the Police: Detroit, 1928-76", "authors": [ "McDowall, David", "Loftin, Colin" ] }, { "ID": [ "icpsr:20991", "issn:0020-8833" ], "title": "A Liberal Trade Order? The Long-Run Pattern of Imports to the Advanced Capitalist States", "authors": [ "McKeown, Timothy J." ] }, { "ID": [ "icpsr:40039" ], "title": "The Legacy of Lynching and Southern Homicide: A Spatial Analysis", "authors": [ "Messner, Steven F.", "Baller, Robert D.", "Zevenbergen, Matthew P." ] }, { "ID": [ "icpsr:40041", "issn:0003-1224" ], "title": "The legacy of lynching and southern homicide", "authors": [ "Messner, Steven F.", "Baller, Robert D.", "Zevenbergen, Matthew P." ] }, { "ID": [ "icpsr:21524", "issn:0161-5440" ], "title": "Estimating Interregional Migration in the United Staes form Sample Data: A Procedural Note", "authors": [ "Miller, Ann R." ] }, { "ID": [ "icpsr:21840", "issn:0022-2445" ], "title": "Urbanization and the Malleable Household: An Examination of Boarding and Lodging in American Families", "authors": [ "Modell, John", "Hareven, Tamara K." ] }, { "ID": [ "icpsr:22989", "issn:1040-0656" ], "title": "'We do not want our girls to marry foreigners': Gender, race, and American citizenship", "authors": [ "Nicolosi, Ann Marie" ] }, { "ID": [ "icpsr:40356", "issn:1040-0656" ], "title": "We Do Not Want Our Girls to Marry Foreigners: Gender, Race, and American Citizenship", "authors": [ "Nicolosi, Ann Marie" ] }, { "ID": [ "icpsr:23474", "issn:0037-7732" ], "title": "Causes in the Shifts in Occupational Segregation of the Foreign-Born: Evidence from American Cities, 1870-1880", "authors": [ "Olzak, Susan D." ] }, { "ID": [ "icpsr:92973", "issn:0009-8078" ], "title": "Measures of war: A quantitative examination of the Civil War's destructiveness in the Confederacy", "authors": [ "Paskoff, Paul F." ] }, { "ID": [ "icpsr:24940", "issn:0277-9536" ], "title": "Childhood conditions that predict survival to advanced ages among African-Americans", "authors": [ "Preston, Samuel H.", "Hill, Mark E.", "Drevenstedt, Greg L." ] }, { "ID": [ "icpsr:24953", "issn:0037-7732" ], "title": "Between Depression and Prosperity? Changes in the Community Context of Historical African American Migration", "authors": [ "Price-Spratlen, Townsand" ] }, { "ID": [ "icpsr:68051", "issn:0145-5532" ], "title": "Urban destination selection among African Americans during the 1950s Great Migration", "authors": [ "Price-Spratlen, Townsand" ] }, { "ID": [ "icpsr:50112", "issn:0166-0462" ], "title": "Moving to nice weather", "authors": [ "Rappaport, Jordan" ] }, { "ID": [ "icpsr:36346", "issn:0161-2387" ], "title": "U.S. Urban Decline and Growth, 1950 to 2000", "authors": [ "Rappaport, Jordan" ] }, { "ID": [ "icpsr:25348", "issn:0003-1224" ], "title": "Failed Populism: Movement-Party Disjuncture in North Carolina, 1890 to 1900", "authors": [ "Redding, Kent" ] }, { "ID": [ "icpsr:25349", "issn:0161-5440" ], "title": "Estimating levels and modeling determinants of black and white voter turnout in the South, 1880 to 1912", "authors": [ "Redding, Kent", "James, David R." ] }, { "ID": [ "icpsr:26432", "issn:0730-8884" ], "title": "'Provided Only That Their Voice Was Strong': Insurgency and Organization of American Labor from NRA to Taft-Hartley", "authors": [ "Rubin, Beth A.", "Griffin, Larry J.", "Wallace, Michael" ] }, { "ID": [ "icpsr:26466", "issn:0003-1224" ], "title": "The origins of African-American family structure", "authors": [ "Ruggles, Steven" ] }, { "ID": [ "icpsr:26926" ], "title": "Racial and Ethnic Groups", "authors": [ "Schaefer, Richard T." ] }, { "ID": [ "icpsr:27007" ], "title": "Educational documentation to accompany the great American history machine", "authors": [ "Schlatterer, Helen Marie Fell" ] }, { "ID": [ "icpsr:111679", "issn:2154-4727" ], "title": "The literate South: Reading before Emancipation", "authors": [ "Schweiger, Beth Barton" ] }, { "ID": [ "icpsr:27502", "issn:0145-5532" ], "title": "'New political history': Some statistical questions raised", "authors": [ "Shade, William G." ] }, { "ID": [ "icpsr:27884", "issn:0002-1482" ], "title": "An assessment of the frontier's influence on voter turnout", "authors": [ "Shortridge, Ray Myles" ] }, { "ID": [ "icpsr:93580" ], "title": "The New Deal and the Diffusion of Tractors in the 1930s", "authors": [ "Sorensen, Todd", "Fishback, Price", "Kantor, Shawn" ] }, { "ID": [ "icpsr:28727", "issn:0037-7732" ], "title": "Populism and black lynching in Georgia, 1890-1900", "authors": [ "Soule, Sarah A." ] }, { "ID": [ "icpsr:28779", "issn:0145-5532" ], "title": "Aid to Sharecroppers: How Agrarian Class Structure and Tenant-Farmer Politics Influenced Federal Relief in the South, 1933-1935", "authors": [ "Southworth, Caleb" ] }, { "ID": [ "icpsr:28950", "issn:0192-513X" ], "title": "Divorce and Suicide: A Time Series Analysis, 1933-1970", "authors": [ "Stack, Steven" ] }, { "ID": [ "icpsr:28948", "issn:0022-2445" ], "title": "The Effect of Domestic/Religious Individualism on Suicide, 1954-1978", "authors": [ "Stack, Steven" ] }, { "ID": [ "icpsr:29057", "issn:0161-5440" ], "title": "Census Manuscript Schedules Matched with Property Tax Lists: A Source of Information on Long-Term rends in Wealth Inequality", "authors": [ "Steckel, Richard H." ] }, { "ID": [ "icpsr:29148", "issn:1078-0874" ], "title": "Urban Underrepresentation in the U.S. Senate", "authors": [ "Stephens, G. Ross" ] }, { "ID": [ "icpsr:92455", "issn:0022-0507" ], "title": "The geography of wage discrimination in the pre-civil rights South", "authors": [ "Sundstrom, William A." ] }, { "ID": [ "icpsr:92458", "issn:0022-0507" ], "title": "Important or impotent? Taking another look at the 1920 California Alien Land Law", "authors": [ "Suzuki, Masao" ] }, { "ID": [ "icpsr:29700", "issn:0022-2445" ], "title": "Broken Homes: Stable Risk, Changing Reasons, Changing Forms", "authors": [ "Sweetser, Dorrian Apple" ] }, { "ID": [ "icpsr:92457" ], "title": "From Home to Hospital: The Evolution of Childbirth in the United States, 1927-1940", "authors": [ "Thomasson, Melissa A." ] }, { "ID": [ "icpsr:30093", "issn:0022-0027" ], "title": "War, Inflation, and the Kondratieff Long Wave", "authors": [ "Thompson, William R.", "Zuk, L. Gary" ] }, { "ID": [ "icpsr:30290", "issn:0037-7732" ], "title": "African Americans and immigrants in northern cities: The effects of relative group size on occupational standing in 1920", "authors": [ "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:30288", "issn:0037-7732" ], "title": "Educational Selection in the Migration of Southern Blacks, 1880-1990", "authors": [ "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:30285", "issn:0003-1224" ], "title": "The spatial diffusion of fertility: A cross-sectional analysis of counties in the American South, 1940", "authors": [ "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:30291", "issn:0003-1224" ], "title": "Racial Violence and Black Migration in the American South, 1910 to 1930", "authors": [ "Tolnay, Stewart E.", "Beck, E.M." ] }, { "ID": [ "icpsr:30292", "issn:0037-7732" ], "title": "Black lynchings: The power threat hypothesis revisited", "authors": [ "Tolnay, Stewart E.", "Beck, E.M.", "Massey, James L." ] }, { "ID": [ "icpsr:30295", "issn:0037-7732" ], "title": "'Narrow and Filthy Alleys of the City'?: The Residential Settlement Patterns of Black Southern Migrants to the North", "authors": [ "Tolnay, Stewart E.", "Crowder, Kyle D.", "Adelman, Robert M." ] }, { "ID": [ "icpsr:30296", "issn:0002-9602" ], "title": "Vicarious Violence: Spatial Effects on Southern Lynchings, 1980-1919", "authors": [ "Tolnay, Stewart E.", "Deane, Glenn", "Beck, E.M." ] }, { "ID": [ "icpsr:30297", "issn:0070-3370" ], "title": "The Persistence of high fertility in the American South on the eve of the baby boom", "authors": [ "Tolnay, Stewart E.", "Glynn, Patricia J." ] }, { "ID": [ "icpsr:30620", "issn:0037-7732" ], "title": "Increasing Scale and Spatial Differentiation: New Tests of Two Theories from Shevky and Bell", "authors": [ "Udry, J. Richard" ] }, { "ID": [ "icpsr:30914", "issn:0020-8833" ], "title": "Democratic Convergence and Free Trade", "authors": [ "Verdier, Daniel" ] }, { "ID": [ "icpsr:31224", "issn:0038-0407" ], "title": "Schooling or Working? Public Education, Racial Politics, and the Organization of Production in 1910", "authors": [ "Walters, Pamela Barnhouse", "McCammon, Holly J.", "James, David R." ] }, { "ID": [ "icpsr:31225", "issn:0002-9602" ], "title": "The Family Economy, Work, and Educational Participation in the United States, 1890-1940", "authors": [ "Walters, Pamela Barnhouse", "O'Connell, Philip J." ] }, { "ID": [ "icpsr:31226", "issn:0003-1224" ], "title": "Educational expansion and economic output in the United States, 1890-1969: A production function analysis", "authors": [ "Walters, Pamela Barnhouse", "Robinson, Richard" ] }, { "ID": [ "icpsr:31767", "issn:0038-0407" ], "title": "Tug-of-war: Political mobilization and access to schooling in the southern racial state", "authors": [ "Werum, Regina E." ] }, { "ID": [ "icpsr:92732", "issn:0167-5923" ], "title": "Sending or receiving stations? The dual influence of railroads in early 20th-century Great Plains settlement", "authors": [ "White, Katherine J. Curtis" ] }, { "ID": [ "icpsr:92978" ], "title": "Temporal and Spatial Variation in 20th Century U.S. Great Plains Population Change", "authors": [ "White, Katherine J. Curtis" ] }, { "ID": [ "icpsr:92977" ], "title": "Growth Curve Modeling: An Alternative Approach to Estimating Patterns of Population Change", "authors": [ "White, Katherine J. Curtis", "Herting, Jerald R." ] }, { "ID": [ "icpsr:32058", "issn:0022-2445" ], "title": "The Decline in Men's Labor Force Participation and Income and the Changing Structure of Family Economic Support", "authors": [ "Wilkie, Jane Riblett" ] }, { "ID": [ "icpsr:40398", "issn:0009-8078" ], "title": "The 1835 anti-Abolition meetings in the South: A new look at the controversy over the Abolition Postal Campaign", "authors": [ "Wyly-Jones, Susan" ] }, { "ID": [ "icpsr:92459", "issn:0034-6446" ], "title": "Black enterprise and the legacy of slavery", "authors": [ "Zajonc, Tristan" ] } ], "grant": { "identifiers": [ "GS-1435, GS-1435A1, GS-1231, GS-2473, and APR75-01320" ], "funders": [ "National Science Foundation. Research Applied to National Needs" ] }, "access": { "landingPage": "http://doi.org/10.3886/ICPSR00003.v1" }, "license": { "copyright": "ICPSR metadata records are licensed under a Creative Commons Attribution-Noncommercial 3.0 United States License." } }


From: George Alter [notifications@github.com] Sent: Tuesday, March 28, 2017 4:21 PM To: biocaddie/prototype_issues Cc: Subscribed Subject: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Inconsistencies between ICPSR's DATS and DataMed.docxhttps://github.com/biocaddie/prototype_issues/files/877339/Inconsistencies.between.ICPSR.s.DATS.and.DataMed.docx Sanda found a number of problems with the records that were ingested from ICPSR DATS. In the attached file she compares one record in DataMED to the way that it appears in the JSON file.

- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/biocaddie/prototype_issues/issues/256, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALPhs82TIQdRALkFulUjOcw5NoFOy9TWks5rqZYOgaJpZM4MsUGY.

yul129 commented 7 years ago
  1. the license information are under datsetDistribution, which is also consistent with the ICPSR website.

    From: Yueling Li Sent: Wednesday, March 29, 2017 3:28 PM To: biocaddie/prototype_issues; biocaddie/prototype_issues Cc: Subscribed Subject: RE: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

By comparing transformed ICPSR JSON file (attached at the bottom of this email) with the ICPSR website for ICPSR 3 (https://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3?searchSource=revise&q=icpsr+3) 1.The description, license and creators are all consistent 2.The Grant No. information is under "grant" in DataMed, not under "acknowledges" 3.The availability field need to be re-mapped, Burak and i will work on it today.

{ "dataset": { "availability": "available", "types": [ "other" ], "aggregation": "collection of dataset", "privacy": "not applicable", "refinement": "curated", "dateReleased": "19840706T000000+0000", "creators": [ "Inter-university Consortium for Political and Social Research" ], "ID": "3", "title": " Historical, Demographic, Economic, and Social Data: The United States, 1790-1970", "keywords": [ "census data", "counties", "demographic characteristics", "economic conditions", "eighteenth century", "families", "historical data", "marriage", "national identity", "nineteenth century", "occupations", "political change", "population", "population size", "religion", "states (USA)", "twentieth century" ], "description": "Detailed county and state-level ecological or descriptive data for the United States for the years 1790 to 1970 are contained in this collection. These data files contain extensive information about the social and political character of the United States, including a breakdown of population by state, race, nationality, number of families, size of the family, births, deaths, marriages, occupation, religion, and general economic conditions. Though not complete over the full time span of this study, statistics are available on such diverse subjects as total numbers of newspapers and periodicals, total capital invested in manufacturing, total numbers of educational institutions, total number of churches, taxation by state, and land surface area in square miles." }, "datasetDistributions": [ { "format": "DOCUMENT", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/dldocs2?study=3&path=ICPSR" }, { "format": "SAS", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=sas&path=ICPSR" }, { "format": "SPSS", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=spss&path=ICPSR" }, { "format": "STATA", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=stata&path=ICPSR" }, { "format": "ASCII", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=ascii&path=ICPSR" }, { "format": "DELIMITED", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=delimited&path=ICPSR" } ], "dataRepository": { "ID": "SCR:003194", "name": "Inter-university Consortium for Political and Social Research", "abbreviation": "ICPSR", "homePage": "http://www.icpsr.umich.edu/" }, "organization": { "ID": "SCR:011668", "name": "University of Michigan", "abbreviation": "UMich", "homePage": "http://www.umich.edu/" }, "dataStandard": { "version": "2.5", "name": "ddi:codebook:2_5 http://www.ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/codebook.xsd" }, "citations": [ { "ID": [ "icpsr:990", "issn:0161-5440" ], "title": "A Methodology for Revising Estimates: Female Market Participation in the United States Before 1940", "authors": [ "Abel, Marjorie", "Folbre, Nancy" ] }, { "ID": [ "icpsr:92981" ], "title": "Cultural Diversity and Economic Growth: Evidence from the US During the Ages of Mass Migration", "authors": [ "Ager, Philipp, Brückner, Markus" ] }, { "ID": [ "icpsr:1318", "issn:0070-3370" ], "title": "The human ecology of tornadoes", "authors": [ "Aguirre, B.E.", "Saenz, Rogelino", "Edmiston, John", "Yang, Nan", "Agramonte, Elsa", "Stuart, Dietra L." ] }, { "ID": [ "icpsr:1465", "issn:0022-3433" ], "title": "Military Conversion Policies in the USA: 1940s and 1990s", "authors": [ "Alexander, J. Davidson" ] }, { "ID": [ "icpsr:1508", "issn:0145-5532" ], "title": "From the Populist era to the New Deal: A study of partisan realignment in Washington State, 1889-1950", "authors": [ "Allen, Howard W.", "Austin, Erik W." ] }, { "ID": [ "icpsr:68045" ], "title": "Race, Class, and the Death Penalty: Capital Punishment in American History", "authors": [ "Allen, Howard W.", "Clubb, Jerome M." ] }, { "ID": [ "icpsr:92400", "issn:0022-0507" ], "title": "Time on the ladder: Career mobility in agriculture, 1890-1938", "authors": [ "Alston, Lee J.", "Ferrie, Joseph P." ] }, { "ID": [ "icpsr:92733", "issn:1545-2921" ], "title": "Electoral behavior of US counties: A panel data approach", "authors": [ "Anatolyev, Stanislav" ] }, { "ID": [ "icpsr:86433", "issn:0145-5532" ], "title": "Did railroads induce or follow economic growth? Urbanization and population growth in the American Midwest, 1850-1860", "authors": [ "Atack, Jeremy", "Bateman, Fred", "Haines, Michael", "Margo, Robert A." ] }, { "ID": [ "icpsr:2436", "issn:0730-8884" ], "title": "The Deskilling Controversy", "authors": [ "Attewell, Paul" ] }, { "ID": [ "icpsr:2706", "issn:0038-0407" ], "title": "Schooling all the masses: Reconsidering the origins of American schooling in the postbellum era", "authors": [ "Baker, David P." ] }, { "ID": [ "icpsr:40038" ], "title": "The Culture of Honor and Southern Homicide", "authors": [ "Baller, Robert", "Zevenbergen, Matthew P.", "Messner, Steven F." ] }, { "ID": [ "icpsr:2800", "issn:0021-9347" ], "title": "A critical inquiry of enslaved African females and the antebellum hospital experience", "authors": [ "Bankole, Katherine Olukemi" ] }, { "ID": [ "icpsr:3212", "issn:0022-0027" ], "title": "Trends in American Major War and Peace", "authors": [ "Beer, Francis A." ] }, { "ID": [ "icpsr:92396", "issn:0166-0462" ], "title": "Population growth in U.S. counties, 1840-1990", "authors": [ "Beeson, Patricia E.", "DeJong, David N.", "Troesken, Werner" ] }, { "ID": [ "icpsr:92451", "issn:1432-9840" ], "title": "Historical land-cover conversion (1665-1820) in the Choptank Watershed, eastern United States", "authors": [ "Benitez, Jorge A.", "Fisher, Thomas R." ] }, { "ID": [ "icpsr:120303" ], "title": "Essays on State Dependence in the Government Spending Multiplier", "authors": [ "Biolsi, Christopher" ] }, { "ID": [ "icpsr:3881", "issn:0145-5532" ], "title": "Group Enmity and Accord: The Commercial Press in Three American Cities", "authors": [ "Blau, Judith R." ] }, { "ID": [ "icpsr:93031", "issn:0049-089X" ], "title": "The expansion of religious affiliation: An explanation of the growth of church participation in the United States 1850-1930", "authors": [ "Blau, Judith R.", "Land, Kenneth C.", "Redding, Kent" ] }, { "ID": [ "icpsr:3884", "issn:0731-1214" ], "title": "Spatial processes and the duality of church and faith: a Simmelian perspective on U.S. denominational growth, 1900-1930", "authors": [ "Blau, Judith R.", "Redding, Kent", "Davis, Walter R.", "Land, Kenneth C." ] }, { "ID": [ "icpsr:3885", "issn:0884-8971" ], "title": "Ethnocultural cleavages and the growth of church membership in the United States, 1860-1930", "authors": [ "Blau, Judith R.", "Redding, Kent", "Land, Kenneth C." ] }, { "ID": [ "icpsr:112530", "issn:1945-7790" ], "title": "Malaria eradication in the Americas: A retrospective analysis of childhood exposure", "authors": [ "Bleakley, Hoyt" ] }, { "ID": [ "icpsr:87451", "issn:0034-6535" ], "title": "Chronic disease burden and the interaction of education, fertility, and growth", "authors": [ "Bleakley, Hoyt", "Lange, Fabian" ] }, { "ID": [ "icpsr:4276", "issn:0022-0515" ], "title": "The Economics of Immigration", "authors": [ "Borjas, George J." ] }, { "ID": [ "icpsr:40582", "issn:1071-4839" ], "title": "The drug war is the new Jim Crow", "authors": [ "Boyd, Graham" ] }, { "ID": [ "icpsr:33600" ], "title": "To Be 'Read By the Whole People': Press, Party, and Public Sphere in the United States, 1790-1840", "authors": [ "Brooke, John L." ] }, { "ID": [ "icpsr:92403", "issn:0002-8282" ], "title": "Fundamentals, panics, and bank distress during the Depression", "authors": [ "Calomiris, Charles W.", "Mason, Joseph R." ] }, { "ID": [ "icpsr:5760", "issn:0022-2445" ], "title": "Determinants of Household Headship", "authors": [ "Carliner, Geoffrey" ] }, { "ID": [ "icpsr:5896", "issn:0161-5440" ], "title": "Fixing the Facts: Editing of the 1880 U.S. Census of Occupations with Implications for Long-Term Labor-Force Trends and the Sociology of Official Statistics", "authors": [ "Carter, Susan B.", "Sutch, Richard" ] }, { "ID": [ "icpsr:112613" ], "title": "Did suburbanization cause residential segregation? Evidence from U.S. metropolitan areas", "authors": [ "Chatterjee, Boishampayan" ] }, { "ID": [ "icpsr:6663" ], "title": "Demographic and compositional change", "authors": [ "Clubb, Jerome M.", "Austin, Erik W.", "Traugott, Michael W." ] }, { "ID": [ "icpsr:6928", "issn:0003-1224" ], "title": "Formal Organization and the Fate of Social Movements: Craft Association and Class Alliance in the Knights of Labor", "authors": [ "Conell, Carol", "Voss, Kim" ] }, { "ID": [ "icpsr:7212", "issn:0022-3433" ], "title": "A Causal Analysis of the Defense Expenditures of the USA and the USSR", "authors": [ "Correa, Hector", "Kim, Ji-Won" ] }, { "ID": [ "icpsr:7231" ], "title": "Unequal at Birth: A Long-Term Comparison of Income and Birth Weight", "authors": [ "Costa, Dora L." ] }, { "ID": [ "icpsr:7271", "issn:0022-0027" ], "title": "War and American Democracy: Electoral Costs of the Last Five Wars", "authors": [ "Cotton, Timothy Y.C." ] }, { "ID": [ "icpsr:7383", "issn:0895-5638" ], "title": "Transportation improvements and land values in the antebellum United States: A hedonic approach", "authors": [ "Craig, Lee A.", "Palmquist, Raymond B.", "Weiss, Thomas" ] }, { "ID": [ "icpsr:92735" ], "title": "Did Big Government's Largesse Help the Locals? The Implications of WWII Spending for Local Economic Activity, 1939-1958", "authors": [ "Cullen, Joseph", "Fishback, Price V." ] }, { "ID": [ "icpsr:8001" ], "title": "Frontier America, 1800-1840: A Comparative Demographic Analysis of the Frontier Process", "authors": [ "Davis, James E." ] }, { "ID": [ "icpsr:8027", "issn:0192-513X" ], "title": "Childless and Single-Childed Women in Early Twentieth-Century America", "authors": [ "Davis, Nancy J." ] }, { "ID": [ "icpsr:8182", "issn:0037-7732" ], "title": "Mass Communication and Social Change", "authors": [ "DeFleur, Melvin L." ] }, { "ID": [ "icpsr:119578" ], "title": "Party Formation in the United States", "authors": [ "DeWitt, Darin Dion" ] }, { "ID": [ "icpsr:92974" ], "title": "Why Does Financial Development Matter? The United States from 1900 to 1940", "authors": [ "Dehejia, Rajeev", "Lleras-Muney, Adriana" ] }, { "ID": [ "icpsr:9290", "issn:0278-5927" ], "title": "Transplanted to the Rose City: The Creation of East European Jewish Community in Portland, Oregon", "authors": [ "Eisenberg, Ellen" ] }, { "ID": [ "icpsr:10371", "issn:0034-6535" ], "title": "Narrowing the Black-White gap in child literacy in 1910: The roles of school inputs and family inputs", "authors": [ "Fishback, Price V.", "Baskin, John H." ] }, { "ID": [ "icpsr:92454" ], "title": "The Welfare of Children During the Great Depression", "authors": [ "Fishback, Price V.", "Haines, Michael R.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92395", "issn:0014-4983" ], "title": "The impact of the New Deal on black and white infant mortality in the South", "authors": [ "Fishback, Price V.", "Haines, Michael R.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92452", "issn:0022-0507" ], "title": "Did New Deal grant programs stimulate local economies? A study of federal grants and retail sales during the Great Depression", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:93578" ], "title": "Do Federal Programs Affect Internal Migration? The Impact of New Deal Expenditures on Mobility During the Great Depression", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92975" ], "title": "The Impact of New Deal Expenditures on Local Economic Activity: An Examination of Retail Sales, 1929-1939", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92731" ], "title": "The Origins of Modern Housing Finance: The Impact of Federal Housing Programs During the Great Depression", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:10372", "issn:0033-5533" ], "title": "Did Workers by for the Passage of Workers' Compensation Laws", "authors": [ "Fishback, Price V.", "Kantor, Shawn Everett" ] }, { "ID": [ "icpsr:10373", "issn:0022-0507" ], "title": "The durable experiment: State insurance of workers' compensation risk in the early twentieth century", "authors": [ "Fishback, Price V.", "Kantor, Shawn Everett" ] }, { "ID": [ "icpsr:92399", "issn:0014-4983" ], "title": "Can the New Deal's three Rs be rehabilitated? A program-by-program, county-by-county analysis", "authors": [ "Fishback, Price V.", "Kantor, Shawn", "Wallis, John Joseph" ] }, { "ID": [ "icpsr:10507", "issn:0038-4038" ], "title": "Electoral incentives, public policy, and the New Deal realignment", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:92394", "issn:0014-4983" ], "title": "Population, land, economic conditions, and the allocation of New Deal spending", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:10508", "issn:0095-2583" ], "title": "The value of the vote: A model and test of the effects of turnout on distributive policy", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:92976", "issn:0022-3808" ], "title": "Voter influence and big policy change: The positive political economy of the New Deal", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:10955", "issn:0037-7732" ], "title": "Risk and Recreancy: Weber, the Division of Labor, and the Rationality of Risk Perceptions", "authors": [ "Freudenburg, William R." ] }, { "ID": [ "icpsr:11227", "issn:0161-5440" ], "title": "Metropolitan Classification for Census Years before World War II", "authors": [ "Gardner, Todd" ] }, { "ID": [ "icpsr:11485", "issn:0145-5532" ], "title": "Populism, Race, and Political Interest in Virginia", "authors": [ "Gerteis, Joseph" ] }, { "ID": [ "icpsr:86772", "issn:0022-0507" ], "title": "The uneven rise of American public schools to 1850", "authors": [ "Go, Sun", "Lindert, Peter" ] }, { "ID": [ "icpsr:12030", "issn:0020-8833" ], "title": "Kondratieff Wave as War Cycles", "authors": [ "Goldstein, Joshua S." ] }, { "ID": [ "icpsr:12214", "issn:0037-7732" ], "title": "The Changing Structure of Property Crime in an Affluent Society", "authors": [ "Gould, Leroy C." ] }, { "ID": [ "icpsr:35884", "issn:0022-1953" ], "title": "From local to national political cultures: Social capital and civic organization in the Great Plains", "authors": [ "Gutmann, Myron P.", "Pullum, Sara M." ] }, { "ID": [ "icpsr:12994" ], "title": "Fertility and Occupation: Population Patterns in Industrialization", "authors": [ "Haines, Michael" ] }, { "ID": [ "icpsr:93556" ], "title": "Spatial aspects of the American fertility transition in the Nineteenth Century", "authors": [ "Haines, Michael R.", "Hacker, J. David" ] }, { "ID": [ "icpsr:92453" ], "title": "The Puzzle of the Antebellum Fertility Decline in the United States: New Evidence and Reconsideration", "authors": [ "Haines, Michael R.", "Hacker, J. David" ] }, { "ID": [ "icpsr:13190", "issn:0003-1224" ], "title": "Revival Religion and Antislavery Politics", "authors": [ "Hammond, John L." ] }, { "ID": [ "icpsr:13331", "issn:0002-1482" ], "title": "Beyond the marginality thesis: the acquisition and loss of land by African Americans in Georgia, 1880-1930", "authors": [ "Hargis, Peggy G." ] }, { "ID": [ "icpsr:13332", "issn:0022-1953" ], "title": "The 'low-country advantage' for African-Americans in Georgia, 1880-1930", "authors": [ "Hargis, Peggy G.", "Horan, Patrick M." ] }, { "ID": [ "icpsr:13564", "issn:0020-8183" ], "title": "Openings in the Wall: Transnational Migrants, Labor Unions, and U.S. Immigration Policy", "authors": [ "Haus, Leah" ] }, { "ID": [ "icpsr:14016", "issn:0161-5440" ], "title": "The Census, Estimation Biases, and Female Labor-Force Participation Rates in 1880 Colorado", "authors": [ "Herr, Elizabeth" ] }, { "ID": [ "icpsr:92734", "issn:0362-9805" ], "title": "Third parties, elections, and roll-call votes: The populist party and the late nineteenth-century U.S. congress", "authors": [ "Hirano, Shigeo" ] }, { "ID": [ "icpsr:96012", "issn:0145-5532" ], "title": "Resisting redemption: The Republican vote in Georgia in 1876", "authors": [ "Hogan, Richard" ] }, { "ID": [ "icpsr:14702", "issn:0022-2445" ], "title": "The Provider Role: Its Meaning and Measurement", "authors": [ "Hood, Jane C." ] }, { "ID": [ "icpsr:15551", "issn:0360-0572" ], "title": "Gender Inequality and Higher Education", "authors": [ "Jacobs, Jerry A." ] }, { "ID": [ "icpsr:15649", "issn:0898-588X" ], "title": "Prelude to progressivism: Party decay, populism, and the doctrine of 'free and unrestricted competition' in American antitrust policy, 1890-1897", "authors": [ "James, Scott C." ] }, { "ID": [ "icpsr:92971" ], "title": "Striking the Roots of Crime: The Impact of the New Deal on Criminal Activity", "authors": [ "Johnson, Ryan S.", "Kantor, Shawn", "Fishback, Price V." ] }, { "ID": [ "icpsr:16242" ], "title": "Education and Social Change in Nineteenth-Century Massachusetts", "authors": [ "Kaestle, Carl F.", "Vinovskis, Maris A." ] }, { "ID": [ "icpsr:16485", "issn:0037-7732" ], "title": "The Theory of Ecological Expansion: An Empirical Test", "authors": [ "Kasarda, John D." ] }, { "ID": [ "icpsr:16653", "issn:0037-7732" ], "title": "The Family Wage System in Pennsylvania's Anthracite Region: 1850-1900", "authors": [ "Keil, Thomas J.", "Usui, Wayne M." ] }, { "ID": [ "icpsr:16795", "issn:0037-7732" ], "title": "Unemployment and Protest in the United States, 1890-1940: A Methodological Critique and Research Note", "authors": [ "Kerbo, Harold R.", "Shaffer, Richard A." ] }, { "ID": [ "icpsr:17170", "issn:0022-2445" ], "title": "Marital Complaints, Demographic Characteristics, and Symptoms of Mental Distress in Divorce", "authors": [ "Kitson, Gay C.", "Sussman, Marvin B." ] }, { "ID": [ "icpsr:17264" ], "title": "The Third Electoral System, 1853-1892: Parties, Voters, and Political Cultures", "authors": [ "Kleppner, Paul" ] }, { "ID": [ "icpsr:17712", "issn:0070-3370" ], "title": "The elderly who live alone in the United States: Historical perspectives on household change", "authors": [ "Kramarow, Ellen A." ] }, { "ID": [ "icpsr:18097", "issn:0081-1750" ], "title": "On the large-sample estimation of regression models with spatial- or network-effects terms: A two-stage least squares approach", "authors": [ "Land, Kenneth C.", "Deane, Glenn" ] }, { "ID": [ "icpsr:18098", "issn:0003-1224" ], "title": "Religious Pluralism and Church Membership: A Spatial Diffusion Model", "authors": [ "Land, Kenneth C.", "Deane, Glenn", "Blau, Judith R." ] }, { "ID": [ "icpsr:18111", "issn:0070-3370" ], "title": "Generation, ethnicity, and marriage: Historical patterns in the northern United States", "authors": [ "Landale, Nancy S.", "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:18459", "issn:0014-4983" ], "title": "Farm value and retirement of farm owners in early-twentieth-century America", "authors": [ "Lee, Chulhee" ] }, { "ID": [ "icpsr:18817", "issn:0021-8723" ], "title": "Draft Evasion in the North during the Civil War, 1863-1865", "authors": [ "Levine, Peter" ] }, { "ID": [ "icpsr:19111" ], "title": "Fertility and Scarcity in America", "authors": [ "Lindert, Peter" ] }, { "ID": [ "icpsr:19350", "issn:0020-8183" ], "title": "Dividend Government and U.S. Trade Policy: Theory and Evidence", "authors": [ "Lohmann, Susanne", "O'Halloran, Sharyn" ] }, { "ID": [ "icpsr:19560", "issn:0022-3433" ], "title": "Empire by Invitation? The United States and Western Europe, 1945-1952", "authors": [ "Lundestad, Geir" ] }, { "ID": [ "icpsr:20175", "issn:0145-5532" ], "title": "Immigrant Support for the American Socialist Party, 1912 and 1920", "authors": [ "Marks, Gary", "Burbank, Matthew" ] }, { "ID": [ "icpsr:20443", "issn:0038-4941" ], "title": "The Changing Geographic Structure of Black and White Segregation in the United States", "authors": [ "Massey, Douglas S.", "Hajnal, Zoltan L." ] }, { "ID": [ "icpsr:20593", "issn:0038-4941" ], "title": "Demographic and Economic Determinants of United States Income Inequality", "authors": [ "Maxwell, Nan L." ] }, { "ID": [ "icpsr:20706", "issn:0730-8884" ], "title": "'Government by Injunction': The U. S. Judiciary and Strike Action in the Late 19th and Early 20th Centuries", "authors": [ "McCammon, Holly J." ] }, { "ID": [ "icpsr:20860", "issn:0037-7732" ], "title": "Fiscal Politics and the Police: Detroit, 1928-76", "authors": [ "McDowall, David", "Loftin, Colin" ] }, { "ID": [ "icpsr:20991", "issn:0020-8833" ], "title": "A Liberal Trade Order? The Long-Run Pattern of Imports to the Advanced Capitalist States", "authors": [ "McKeown, Timothy J." ] }, { "ID": [ "icpsr:40039" ], "title": "The Legacy of Lynching and Southern Homicide: A Spatial Analysis", "authors": [ "Messner, Steven F.", "Baller, Robert D.", "Zevenbergen, Matthew P." ] }, { "ID": [ "icpsr:40041", "issn:0003-1224" ], "title": "The legacy of lynching and southern homicide", "authors": [ "Messner, Steven F.", "Baller, Robert D.", "Zevenbergen, Matthew P." ] }, { "ID": [ "icpsr:21524", "issn:0161-5440" ], "title": "Estimating Interregional Migration in the United Staes form Sample Data: A Procedural Note", "authors": [ "Miller, Ann R." ] }, { "ID": [ "icpsr:21840", "issn:0022-2445" ], "title": "Urbanization and the Malleable Household: An Examination of Boarding and Lodging in American Families", "authors": [ "Modell, John", "Hareven, Tamara K." ] }, { "ID": [ "icpsr:22989", "issn:1040-0656" ], "title": "'We do not want our girls to marry foreigners': Gender, race, and American citizenship", "authors": [ "Nicolosi, Ann Marie" ] }, { "ID": [ "icpsr:40356", "issn:1040-0656" ], "title": "We Do Not Want Our Girls to Marry Foreigners: Gender, Race, and American Citizenship", "authors": [ "Nicolosi, Ann Marie" ] }, { "ID": [ "icpsr:23474", "issn:0037-7732" ], "title": "Causes in the Shifts in Occupational Segregation of the Foreign-Born: Evidence from American Cities, 1870-1880", "authors": [ "Olzak, Susan D." ] }, { "ID": [ "icpsr:92973", "issn:0009-8078" ], "title": "Measures of war: A quantitative examination of the Civil War's destructiveness in the Confederacy", "authors": [ "Paskoff, Paul F." ] }, { "ID": [ "icpsr:24940", "issn:0277-9536" ], "title": "Childhood conditions that predict survival to advanced ages among African-Americans", "authors": [ "Preston, Samuel H.", "Hill, Mark E.", "Drevenstedt, Greg L." ] }, { "ID": [ "icpsr:24953", "issn:0037-7732" ], "title": "Between Depression and Prosperity? Changes in the Community Context of Historical African American Migration", "authors": [ "Price-Spratlen, Townsand" ] }, { "ID": [ "icpsr:68051", "issn:0145-5532" ], "title": "Urban destination selection among African Americans during the 1950s Great Migration", "authors": [ "Price-Spratlen, Townsand" ] }, { "ID": [ "icpsr:50112", "issn:0166-0462" ], "title": "Moving to nice weather", "authors": [ "Rappaport, Jordan" ] }, { "ID": [ "icpsr:36346", "issn:0161-2387" ], "title": "U.S. Urban Decline and Growth, 1950 to 2000", "authors": [ "Rappaport, Jordan" ] }, { "ID": [ "icpsr:25348", "issn:0003-1224" ], "title": "Failed Populism: Movement-Party Disjuncture in North Carolina, 1890 to 1900", "authors": [ "Redding, Kent" ] }, { "ID": [ "icpsr:25349", "issn:0161-5440" ], "title": "Estimating levels and modeling determinants of black and white voter turnout in the South, 1880 to 1912", "authors": [ "Redding, Kent", "James, David R." ] }, { "ID": [ "icpsr:26432", "issn:0730-8884" ], "title": "'Provided Only That Their Voice Was Strong': Insurgency and Organization of American Labor from NRA to Taft-Hartley", "authors": [ "Rubin, Beth A.", "Griffin, Larry J.", "Wallace, Michael" ] }, { "ID": [ "icpsr:26466", "issn:0003-1224" ], "title": "The origins of African-American family structure", "authors": [ "Ruggles, Steven" ] }, { "ID": [ "icpsr:26926" ], "title": "Racial and Ethnic Groups", "authors": [ "Schaefer, Richard T." ] }, { "ID": [ "icpsr:27007" ], "title": "Educational documentation to accompany the great American history machine", "authors": [ "Schlatterer, Helen Marie Fell" ] }, { "ID": [ "icpsr:111679", "issn:2154-4727" ], "title": "The literate South: Reading before Emancipation", "authors": [ "Schweiger, Beth Barton" ] }, { "ID": [ "icpsr:27502", "issn:0145-5532" ], "title": "'New political history': Some statistical questions raised", "authors": [ "Shade, William G." ] }, { "ID": [ "icpsr:27884", "issn:0002-1482" ], "title": "An assessment of the frontier's influence on voter turnout", "authors": [ "Shortridge, Ray Myles" ] }, { "ID": [ "icpsr:93580" ], "title": "The New Deal and the Diffusion of Tractors in the 1930s", "authors": [ "Sorensen, Todd", "Fishback, Price", "Kantor, Shawn" ] }, { "ID": [ "icpsr:28727", "issn:0037-7732" ], "title": "Populism and black lynching in Georgia, 1890-1900", "authors": [ "Soule, Sarah A." ] }, { "ID": [ "icpsr:28779", "issn:0145-5532" ], "title": "Aid to Sharecroppers: How Agrarian Class Structure and Tenant-Farmer Politics Influenced Federal Relief in the South, 1933-1935", "authors": [ "Southworth, Caleb" ] }, { "ID": [ "icpsr:28950", "issn:0192-513X" ], "title": "Divorce and Suicide: A Time Series Analysis, 1933-1970", "authors": [ "Stack, Steven" ] }, { "ID": [ "icpsr:28948", "issn:0022-2445" ], "title": "The Effect of Domestic/Religious Individualism on Suicide, 1954-1978", "authors": [ "Stack, Steven" ] }, { "ID": [ "icpsr:29057", "issn:0161-5440" ], "title": "Census Manuscript Schedules Matched with Property Tax Lists: A Source of Information on Long-Term rends in Wealth Inequality", "authors": [ "Steckel, Richard H." ] }, { "ID": [ "icpsr:29148", "issn:1078-0874" ], "title": "Urban Underrepresentation in the U.S. Senate", "authors": [ "Stephens, G. Ross" ] }, { "ID": [ "icpsr:92455", "issn:0022-0507" ], "title": "The geography of wage discrimination in the pre-civil rights South", "authors": [ "Sundstrom, William A." ] }, { "ID": [ "icpsr:92458", "issn:0022-0507" ], "title": "Important or impotent? Taking another look at the 1920 California Alien Land Law", "authors": [ "Suzuki, Masao" ] }, { "ID": [ "icpsr:29700", "issn:0022-2445" ], "title": "Broken Homes: Stable Risk, Changing Reasons, Changing Forms", "authors": [ "Sweetser, Dorrian Apple" ] }, { "ID": [ "icpsr:92457" ], "title": "From Home to Hospital: The Evolution of Childbirth in the United States, 1927-1940", "authors": [ "Thomasson, Melissa A." ] }, { "ID": [ "icpsr:30093", "issn:0022-0027" ], "title": "War, Inflation, and the Kondratieff Long Wave", "authors": [ "Thompson, William R.", "Zuk, L. Gary" ] }, { "ID": [ "icpsr:30290", "issn:0037-7732" ], "title": "African Americans and immigrants in northern cities: The effects of relative group size on occupational standing in 1920", "authors": [ "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:30288", "issn:0037-7732" ], "title": "Educational Selection in the Migration of Southern Blacks, 1880-1990", "authors": [ "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:30285", "issn:0003-1224" ], "title": "The spatial diffusion of fertility: A cross-sectional analysis of counties in the American South, 1940", "authors": [ "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:30291", "issn:0003-1224" ], "title": "Racial Violence and Black Migration in the American South, 1910 to 1930", "authors": [ "Tolnay, Stewart E.", "Beck, E.M." ] }, { "ID": [ "icpsr:30292", "issn:0037-7732" ], "title": "Black lynchings: The power threat hypothesis revisited", "authors": [ "Tolnay, Stewart E.", "Beck, E.M.", "Massey, James L." ] }, { "ID": [ "icpsr:30295", "issn:0037-7732" ], "title": "'Narrow and Filthy Alleys of the City'?: The Residential Settlement Patterns of Black Southern Migrants to the North", "authors": [ "Tolnay, Stewart E.", "Crowder, Kyle D.", "Adelman, Robert M." ] }, { "ID": [ "icpsr:30296", "issn:0002-9602" ], "title": "Vicarious Violence: Spatial Effects on Southern Lynchings, 1980-1919", "authors": [ "Tolnay, Stewart E.", "Deane, Glenn", "Beck, E.M." ] }, { "ID": [ "icpsr:30297", "issn:0070-3370" ], "title": "The Persistence of high fertility in the American South on the eve of the baby boom", "authors": [ "Tolnay, Stewart E.", "Glynn, Patricia J." ] }, { "ID": [ "icpsr:30620", "issn:0037-7732" ], "title": "Increasing Scale and Spatial Differentiation: New Tests of Two Theories from Shevky and Bell", "authors": [ "Udry, J. Richard" ] }, { "ID": [ "icpsr:30914", "issn:0020-8833" ], "title": "Democratic Convergence and Free Trade", "authors": [ "Verdier, Daniel" ] }, { "ID": [ "icpsr:31224", "issn:0038-0407" ], "title": "Schooling or Working? Public Education, Racial Politics, and the Organization of Production in 1910", "authors": [ "Walters, Pamela Barnhouse", "McCammon, Holly J.", "James, David R." ] }, { "ID": [ "icpsr:31225", "issn:0002-9602" ], "title": "The Family Economy, Work, and Educational Participation in the United States, 1890-1940", "authors": [ "Walters, Pamela Barnhouse", "O'Connell, Philip J." ] }, { "ID": [ "icpsr:31226", "issn:0003-1224" ], "title": "Educational expansion and economic output in the United States, 1890-1969: A production function analysis", "authors": [ "Walters, Pamela Barnhouse", "Robinson, Richard" ] }, { "ID": [ "icpsr:31767", "issn:0038-0407" ], "title": "Tug-of-war: Political mobilization and access to schooling in the southern racial state", "authors": [ "Werum, Regina E." ] }, { "ID": [ "icpsr:92732", "issn:0167-5923" ], "title": "Sending or receiving stations? The dual influence of railroads in early 20th-century Great Plains settlement", "authors": [ "White, Katherine J. Curtis" ] }, { "ID": [ "icpsr:92978" ], "title": "Temporal and Spatial Variation in 20th Century U.S. Great Plains Population Change", "authors": [ "White, Katherine J. Curtis" ] }, { "ID": [ "icpsr:92977" ], "title": "Growth Curve Modeling: An Alternative Approach to Estimating Patterns of Population Change", "authors": [ "White, Katherine J. Curtis", "Herting, Jerald R." ] }, { "ID": [ "icpsr:32058", "issn:0022-2445" ], "title": "The Decline in Men's Labor Force Participation and Income and the Changing Structure of Family Economic Support", "authors": [ "Wilkie, Jane Riblett" ] }, { "ID": [ "icpsr:40398", "issn:0009-8078" ], "title": "The 1835 anti-Abolition meetings in the South: A new look at the controversy over the Abolition Postal Campaign", "authors": [ "Wyly-Jones, Susan" ] }, { "ID": [ "icpsr:92459", "issn:0034-6446" ], "title": "Black enterprise and the legacy of slavery", "authors": [ "Zajonc, Tristan" ] } ], "grant": { "identifiers": [ "GS-1435, GS-1435A1, GS-1231, GS-2473, and APR75-01320" ], "funders": [ "National Science Foundation. Research Applied to National Needs" ] }, "access": { "landingPage": "http://doi.org/10.3886/ICPSR00003.v1" }, "license": { "copyright": "ICPSR metadata records are licensed under a Creative Commons Attribution-Noncommercial 3.0 United States License." } }


From: George Alter [notifications@github.com] Sent: Tuesday, March 28, 2017 4:21 PM To: biocaddie/prototype_issues Cc: Subscribed Subject: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Inconsistencies between ICPSR's DATS and DataMed.docxhttps://github.com/biocaddie/prototype_issues/files/877339/Inconsistencies.between.ICPSR.s.DATS.and.DataMed.docx Sanda found a number of problems with the records that were ingested from ICPSR DATS. In the attached file she compares one record in DataMED to the way that it appears in the JSON file.

- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/biocaddie/prototype_issues/issues/256, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALPhs82TIQdRALkFulUjOcw5NoFOy9TWks5rqZYOgaJpZM4MsUGY.

aegururaj commented 7 years ago

I thought that the mapping was done by ICPSR and we have them exporting their metadata to directly for ingestion?

From: yul129 notifications@github.com Reply-To: biocaddie/prototype_issues reply@reply.github.com Date: Wednesday, March 29, 2017 at 5:30 PM To: biocaddie/prototype_issues prototype_issues@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

  1. the license information are under datsetDistribution, which is also consistent with the ICPSR website.

    From: Yueling Li Sent: Wednesday, March 29, 2017 3:28 PM To: biocaddie/prototype_issues; biocaddie/prototype_issues Cc: Subscribed Subject: RE: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

By comparing transformed ICPSR JSON file (attached at the bottom of this email) with the ICPSR website for ICPSR 3 (https://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3?searchSource=revise&q=icpsr+3) 1.The description, license and creators are all consistent 2.The Grant No. information is under "grant" in DataMed, not under "acknowledges" 3.The availability field need to be re-mapped, Burak and i will work on it today.

{ "dataset": { "availability": "available", "types": [ "other" ], "aggregation": "collection of dataset", "privacy": "not applicable", "refinement": "curated", "dateReleased": "19840706T000000+0000", "creators": [ "Inter-university Consortium for Political and Social Research" ], "ID": "3", "title": " Historical, Demographic, Economic, and Social Data: The United States, 1790-1970", "keywords": [ "census data", "counties", "demographic characteristics", "economic conditions", "eighteenth century", "families", "historical data", "marriage", "national identity", "nineteenth century", "occupations", "political change", "population", "population size", "religion", "states (USA)", "twentieth century" ], "description": "Detailed county and state-level ecological or descriptive data for the United States for the years 1790 to 1970 are contained in this collection. These data files contain extensive information about the social and political character of the United States, including a breakdown of population by state, race, nationality, number of families, size of the family, births, deaths, marriages, occupation, religion, and general economic conditions. Though not complete over the full time span of this study, statistics are available on such diverse subjects as total numbers of newspapers and periodicals, total capital invested in manufacturing, total numbers of educational institutions, total number of churches, taxation by state, and land surface area in square miles." }, "datasetDistributions": [ { "format": "DOCUMENT", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/dldocs2?study=3&path=ICPSR" }, { "format": "SAS", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=sas&path=ICPSR" }, { "format": "SPSS", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=spss&path=ICPSR" }, { "format": "STATA", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=stata&path=ICPSR" }, { "format": "ASCII", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=ascii&path=ICPSR" }, { "format": "DELIMITED", "storedIn": "Inter-university Consortium for Political and Social Research", "accessType": "download", "authorization": "registration", "authentication": "simpleLogin", "primary": "true", "license": "Terms of use are available at http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3/terms AVAILABLE. This study is freely available to ICPSR member institutions.", "accessURL": "http://www.icpsr.umich.edu/cgi-bin/bob/terms2?study=3&bundle=delimited&path=ICPSR" } ], "dataRepository": { "ID": "SCR:003194", "name": "Inter-university Consortium for Political and Social Research", "abbreviation": "ICPSR", "homePage": "http://www.icpsr.umich.edu/" }, "organization": { "ID": "SCR:011668", "name": "University of Michigan", "abbreviation": "UMich", "homePage": "http://www.umich.edu/" }, "dataStandard": { "version": "2.5", "name": "ddi:codebook:2_5 http://www.ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/codebook.xsd" }, "citations": [ { "ID": [ "icpsr:990", "issn:0161-5440" ], "title": "A Methodology for Revising Estimates: Female Market Participation in the United States Before 1940", "authors": [ "Abel, Marjorie", "Folbre, Nancy" ] }, { "ID": [ "icpsr:92981" ], "title": "Cultural Diversity and Economic Growth: Evidence from the US During the Ages of Mass Migration", "authors": [ "Ager, Philipp, Brückner, Markus" ] }, { "ID": [ "icpsr:1318", "issn:0070-3370" ], "title": "The human ecology of tornadoes", "authors": [ "Aguirre, B.E.", "Saenz, Rogelino", "Edmiston, John", "Yang, Nan", "Agramonte, Elsa", "Stuart, Dietra L." ] }, { "ID": [ "icpsr:1465", "issn:0022-3433" ], "title": "Military Conversion Policies in the USA: 1940s and 1990s", "authors": [ "Alexander, J. Davidson" ] }, { "ID": [ "icpsr:1508", "issn:0145-5532" ], "title": "From the Populist era to the New Deal: A study of partisan realignment in Washington State, 1889-1950", "authors": [ "Allen, Howard W.", "Austin, Erik W." ] }, { "ID": [ "icpsr:68045" ], "title": "Race, Class, and the Death Penalty: Capital Punishment in American History", "authors": [ "Allen, Howard W.", "Clubb, Jerome M." ] }, { "ID": [ "icpsr:92400", "issn:0022-0507" ], "title": "Time on the ladder: Career mobility in agriculture, 1890-1938", "authors": [ "Alston, Lee J.", "Ferrie, Joseph P." ] }, { "ID": [ "icpsr:92733", "issn:1545-2921" ], "title": "Electoral behavior of US counties: A panel data approach", "authors": [ "Anatolyev, Stanislav" ] }, { "ID": [ "icpsr:86433", "issn:0145-5532" ], "title": "Did railroads induce or follow economic growth? Urbanization and population growth in the American Midwest, 1850-1860", "authors": [ "Atack, Jeremy", "Bateman, Fred", "Haines, Michael", "Margo, Robert A." ] }, { "ID": [ "icpsr:2436", "issn:0730-8884" ], "title": "The Deskilling Controversy", "authors": [ "Attewell, Paul" ] }, { "ID": [ "icpsr:2706", "issn:0038-0407" ], "title": "Schooling all the masses: Reconsidering the origins of American schooling in the postbellum era", "authors": [ "Baker, David P." ] }, { "ID": [ "icpsr:40038" ], "title": "The Culture of Honor and Southern Homicide", "authors": [ "Baller, Robert", "Zevenbergen, Matthew P.", "Messner, Steven F." ] }, { "ID": [ "icpsr:2800", "issn:0021-9347" ], "title": "A critical inquiry of enslaved African females and the antebellum hospital experience", "authors": [ "Bankole, Katherine Olukemi" ] }, { "ID": [ "icpsr:3212", "issn:0022-0027" ], "title": "Trends in American Major War and Peace", "authors": [ "Beer, Francis A." ] }, { "ID": [ "icpsr:92396", "issn:0166-0462" ], "title": "Population growth in U.S. counties, 1840-1990", "authors": [ "Beeson, Patricia E.", "DeJong, David N.", "Troesken, Werner" ] }, { "ID": [ "icpsr:92451", "issn:1432-9840" ], "title": "Historical land-cover conversion (1665-1820) in the Choptank Watershed, eastern United States", "authors": [ "Benitez, Jorge A.", "Fisher, Thomas R." ] }, { "ID": [ "icpsr:120303" ], "title": "Essays on State Dependence in the Government Spending Multiplier", "authors": [ "Biolsi, Christopher" ] }, { "ID": [ "icpsr:3881", "issn:0145-5532" ], "title": "Group Enmity and Accord: The Commercial Press in Three American Cities", "authors": [ "Blau, Judith R." ] }, { "ID": [ "icpsr:93031", "issn:0049-089X" ], "title": "The expansion of religious affiliation: An explanation of the growth of church participation in the United States 1850-1930", "authors": [ "Blau, Judith R.", "Land, Kenneth C.", "Redding, Kent" ] }, { "ID": [ "icpsr:3884", "issn:0731-1214" ], "title": "Spatial processes and the duality of church and faith: a Simmelian perspective on U.S. denominational growth, 1900-1930", "authors": [ "Blau, Judith R.", "Redding, Kent", "Davis, Walter R.", "Land, Kenneth C." ] }, { "ID": [ "icpsr:3885", "issn:0884-8971" ], "title": "Ethnocultural cleavages and the growth of church membership in the United States, 1860-1930", "authors": [ "Blau, Judith R.", "Redding, Kent", "Land, Kenneth C." ] }, { "ID": [ "icpsr:112530", "issn:1945-7790" ], "title": "Malaria eradication in the Americas: A retrospective analysis of childhood exposure", "authors": [ "Bleakley, Hoyt" ] }, { "ID": [ "icpsr:87451", "issn:0034-6535" ], "title": "Chronic disease burden and the interaction of education, fertility, and growth", "authors": [ "Bleakley, Hoyt", "Lange, Fabian" ] }, { "ID": [ "icpsr:4276", "issn:0022-0515" ], "title": "The Economics of Immigration", "authors": [ "Borjas, George J." ] }, { "ID": [ "icpsr:40582", "issn:1071-4839" ], "title": "The drug war is the new Jim Crow", "authors": [ "Boyd, Graham" ] }, { "ID": [ "icpsr:33600" ], "title": "To Be 'Read By the Whole People': Press, Party, and Public Sphere in the United States, 1790-1840", "authors": [ "Brooke, John L." ] }, { "ID": [ "icpsr:92403", "issn:0002-8282" ], "title": "Fundamentals, panics, and bank distress during the Depression", "authors": [ "Calomiris, Charles W.", "Mason, Joseph R." ] }, { "ID": [ "icpsr:5760", "issn:0022-2445" ], "title": "Determinants of Household Headship", "authors": [ "Carliner, Geoffrey" ] }, { "ID": [ "icpsr:5896", "issn:0161-5440" ], "title": "Fixing the Facts: Editing of the 1880 U.S. Census of Occupations with Implications for Long-Term Labor-Force Trends and the Sociology of Official Statistics", "authors": [ "Carter, Susan B.", "Sutch, Richard" ] }, { "ID": [ "icpsr:112613" ], "title": "Did suburbanization cause residential segregation? Evidence from U.S. metropolitan areas", "authors": [ "Chatterjee, Boishampayan" ] }, { "ID": [ "icpsr:6663" ], "title": "Demographic and compositional change", "authors": [ "Clubb, Jerome M.", "Austin, Erik W.", "Traugott, Michael W." ] }, { "ID": [ "icpsr:6928", "issn:0003-1224" ], "title": "Formal Organization and the Fate of Social Movements: Craft Association and Class Alliance in the Knights of Labor", "authors": [ "Conell, Carol", "Voss, Kim" ] }, { "ID": [ "icpsr:7212", "issn:0022-3433" ], "title": "A Causal Analysis of the Defense Expenditures of the USA and the USSR", "authors": [ "Correa, Hector", "Kim, Ji-Won" ] }, { "ID": [ "icpsr:7231" ], "title": "Unequal at Birth: A Long-Term Comparison of Income and Birth Weight", "authors": [ "Costa, Dora L." ] }, { "ID": [ "icpsr:7271", "issn:0022-0027" ], "title": "War and American Democracy: Electoral Costs of the Last Five Wars", "authors": [ "Cotton, Timothy Y.C." ] }, { "ID": [ "icpsr:7383", "issn:0895-5638" ], "title": "Transportation improvements and land values in the antebellum United States: A hedonic approach", "authors": [ "Craig, Lee A.", "Palmquist, Raymond B.", "Weiss, Thomas" ] }, { "ID": [ "icpsr:92735" ], "title": "Did Big Government's Largesse Help the Locals? The Implications of WWII Spending for Local Economic Activity, 1939-1958", "authors": [ "Cullen, Joseph", "Fishback, Price V." ] }, { "ID": [ "icpsr:8001" ], "title": "Frontier America, 1800-1840: A Comparative Demographic Analysis of the Frontier Process", "authors": [ "Davis, James E." ] }, { "ID": [ "icpsr:8027", "issn:0192-513X" ], "title": "Childless and Single-Childed Women in Early Twentieth-Century America", "authors": [ "Davis, Nancy J." ] }, { "ID": [ "icpsr:8182", "issn:0037-7732" ], "title": "Mass Communication and Social Change", "authors": [ "DeFleur, Melvin L." ] }, { "ID": [ "icpsr:119578" ], "title": "Party Formation in the United States", "authors": [ "DeWitt, Darin Dion" ] }, { "ID": [ "icpsr:92974" ], "title": "Why Does Financial Development Matter? The United States from 1900 to 1940", "authors": [ "Dehejia, Rajeev", "Lleras-Muney, Adriana" ] }, { "ID": [ "icpsr:9290", "issn:0278-5927" ], "title": "Transplanted to the Rose City: The Creation of East European Jewish Community in Portland, Oregon", "authors": [ "Eisenberg, Ellen" ] }, { "ID": [ "icpsr:10371", "issn:0034-6535" ], "title": "Narrowing the Black-White gap in child literacy in 1910: The roles of school inputs and family inputs", "authors": [ "Fishback, Price V.", "Baskin, John H." ] }, { "ID": [ "icpsr:92454" ], "title": "The Welfare of Children During the Great Depression", "authors": [ "Fishback, Price V.", "Haines, Michael R.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92395", "issn:0014-4983" ], "title": "The impact of the New Deal on black and white infant mortality in the South", "authors": [ "Fishback, Price V.", "Haines, Michael R.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92452", "issn:0022-0507" ], "title": "Did New Deal grant programs stimulate local economies? A study of federal grants and retail sales during the Great Depression", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:93578" ], "title": "Do Federal Programs Affect Internal Migration? The Impact of New Deal Expenditures on Mobility During the Great Depression", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92975" ], "title": "The Impact of New Deal Expenditures on Local Economic Activity: An Examination of Retail Sales, 1929-1939", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:92731" ], "title": "The Origins of Modern Housing Finance: The Impact of Federal Housing Programs During the Great Depression", "authors": [ "Fishback, Price V.", "Horrace, William C.", "Kantor, Shawn" ] }, { "ID": [ "icpsr:10372", "issn:0033-5533" ], "title": "Did Workers by for the Passage of Workers' Compensation Laws", "authors": [ "Fishback, Price V.", "Kantor, Shawn Everett" ] }, { "ID": [ "icpsr:10373", "issn:0022-0507" ], "title": "The durable experiment: State insurance of workers' compensation risk in the early twentieth century", "authors": [ "Fishback, Price V.", "Kantor, Shawn Everett" ] }, { "ID": [ "icpsr:92399", "issn:0014-4983" ], "title": "Can the New Deal's three Rs be rehabilitated? A program-by-program, county-by-county analysis", "authors": [ "Fishback, Price V.", "Kantor, Shawn", "Wallis, John Joseph" ] }, { "ID": [ "icpsr:10507", "issn:0038-4038" ], "title": "Electoral incentives, public policy, and the New Deal realignment", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:92394", "issn:0014-4983" ], "title": "Population, land, economic conditions, and the allocation of New Deal spending", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:10508", "issn:0095-2583" ], "title": "The value of the vote: A model and test of the effects of turnout on distributive policy", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:92976", "issn:0022-3808" ], "title": "Voter influence and big policy change: The positive political economy of the New Deal", "authors": [ "Fleck, Robert K." ] }, { "ID": [ "icpsr:10955", "issn:0037-7732" ], "title": "Risk and Recreancy: Weber, the Division of Labor, and the Rationality of Risk Perceptions", "authors": [ "Freudenburg, William R." ] }, { "ID": [ "icpsr:11227", "issn:0161-5440" ], "title": "Metropolitan Classification for Census Years before World War II", "authors": [ "Gardner, Todd" ] }, { "ID": [ "icpsr:11485", "issn:0145-5532" ], "title": "Populism, Race, and Political Interest in Virginia", "authors": [ "Gerteis, Joseph" ] }, { "ID": [ "icpsr:86772", "issn:0022-0507" ], "title": "The uneven rise of American public schools to 1850", "authors": [ "Go, Sun", "Lindert, Peter" ] }, { "ID": [ "icpsr:12030", "issn:0020-8833" ], "title": "Kondratieff Wave as War Cycles", "authors": [ "Goldstein, Joshua S." ] }, { "ID": [ "icpsr:12214", "issn:0037-7732" ], "title": "The Changing Structure of Property Crime in an Affluent Society", "authors": [ "Gould, Leroy C." ] }, { "ID": [ "icpsr:35884", "issn:0022-1953" ], "title": "From local to national political cultures: Social capital and civic organization in the Great Plains", "authors": [ "Gutmann, Myron P.", "Pullum, Sara M." ] }, { "ID": [ "icpsr:12994" ], "title": "Fertility and Occupation: Population Patterns in Industrialization", "authors": [ "Haines, Michael" ] }, { "ID": [ "icpsr:93556" ], "title": "Spatial aspects of the American fertility transition in the Nineteenth Century", "authors": [ "Haines, Michael R.", "Hacker, J. David" ] }, { "ID": [ "icpsr:92453" ], "title": "The Puzzle of the Antebellum Fertility Decline in the United States: New Evidence and Reconsideration", "authors": [ "Haines, Michael R.", "Hacker, J. David" ] }, { "ID": [ "icpsr:13190", "issn:0003-1224" ], "title": "Revival Religion and Antislavery Politics", "authors": [ "Hammond, John L." ] }, { "ID": [ "icpsr:13331", "issn:0002-1482" ], "title": "Beyond the marginality thesis: the acquisition and loss of land by African Americans in Georgia, 1880-1930", "authors": [ "Hargis, Peggy G." ] }, { "ID": [ "icpsr:13332", "issn:0022-1953" ], "title": "The 'low-country advantage' for African-Americans in Georgia, 1880-1930", "authors": [ "Hargis, Peggy G.", "Horan, Patrick M." ] }, { "ID": [ "icpsr:13564", "issn:0020-8183" ], "title": "Openings in the Wall: Transnational Migrants, Labor Unions, and U.S. Immigration Policy", "authors": [ "Haus, Leah" ] }, { "ID": [ "icpsr:14016", "issn:0161-5440" ], "title": "The Census, Estimation Biases, and Female Labor-Force Participation Rates in 1880 Colorado", "authors": [ "Herr, Elizabeth" ] }, { "ID": [ "icpsr:92734", "issn:0362-9805" ], "title": "Third parties, elections, and roll-call votes: The populist party and the late nineteenth-century U.S. congress", "authors": [ "Hirano, Shigeo" ] }, { "ID": [ "icpsr:96012", "issn:0145-5532" ], "title": "Resisting redemption: The Republican vote in Georgia in 1876", "authors": [ "Hogan, Richard" ] }, { "ID": [ "icpsr:14702", "issn:0022-2445" ], "title": "The Provider Role: Its Meaning and Measurement", "authors": [ "Hood, Jane C." ] }, { "ID": [ "icpsr:15551", "issn:0360-0572" ], "title": "Gender Inequality and Higher Education", "authors": [ "Jacobs, Jerry A." ] }, { "ID": [ "icpsr:15649", "issn:0898-588X" ], "title": "Prelude to progressivism: Party decay, populism, and the doctrine of 'free and unrestricted competition' in American antitrust policy, 1890-1897", "authors": [ "James, Scott C." ] }, { "ID": [ "icpsr:92971" ], "title": "Striking the Roots of Crime: The Impact of the New Deal on Criminal Activity", "authors": [ "Johnson, Ryan S.", "Kantor, Shawn", "Fishback, Price V." ] }, { "ID": [ "icpsr:16242" ], "title": "Education and Social Change in Nineteenth-Century Massachusetts", "authors": [ "Kaestle, Carl F.", "Vinovskis, Maris A." ] }, { "ID": [ "icpsr:16485", "issn:0037-7732" ], "title": "The Theory of Ecological Expansion: An Empirical Test", "authors": [ "Kasarda, John D." ] }, { "ID": [ "icpsr:16653", "issn:0037-7732" ], "title": "The Family Wage System in Pennsylvania's Anthracite Region: 1850-1900", "authors": [ "Keil, Thomas J.", "Usui, Wayne M." ] }, { "ID": [ "icpsr:16795", "issn:0037-7732" ], "title": "Unemployment and Protest in the United States, 1890-1940: A Methodological Critique and Research Note", "authors": [ "Kerbo, Harold R.", "Shaffer, Richard A." ] }, { "ID": [ "icpsr:17170", "issn:0022-2445" ], "title": "Marital Complaints, Demographic Characteristics, and Symptoms of Mental Distress in Divorce", "authors": [ "Kitson, Gay C.", "Sussman, Marvin B." ] }, { "ID": [ "icpsr:17264" ], "title": "The Third Electoral System, 1853-1892: Parties, Voters, and Political Cultures", "authors": [ "Kleppner, Paul" ] }, { "ID": [ "icpsr:17712", "issn:0070-3370" ], "title": "The elderly who live alone in the United States: Historical perspectives on household change", "authors": [ "Kramarow, Ellen A." ] }, { "ID": [ "icpsr:18097", "issn:0081-1750" ], "title": "On the large-sample estimation of regression models with spatial- or network-effects terms: A two-stage least squares approach", "authors": [ "Land, Kenneth C.", "Deane, Glenn" ] }, { "ID": [ "icpsr:18098", "issn:0003-1224" ], "title": "Religious Pluralism and Church Membership: A Spatial Diffusion Model", "authors": [ "Land, Kenneth C.", "Deane, Glenn", "Blau, Judith R." ] }, { "ID": [ "icpsr:18111", "issn:0070-3370" ], "title": "Generation, ethnicity, and marriage: Historical patterns in the northern United States", "authors": [ "Landale, Nancy S.", "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:18459", "issn:0014-4983" ], "title": "Farm value and retirement of farm owners in early-twentieth-century America", "authors": [ "Lee, Chulhee" ] }, { "ID": [ "icpsr:18817", "issn:0021-8723" ], "title": "Draft Evasion in the North during the Civil War, 1863-1865", "authors": [ "Levine, Peter" ] }, { "ID": [ "icpsr:19111" ], "title": "Fertility and Scarcity in America", "authors": [ "Lindert, Peter" ] }, { "ID": [ "icpsr:19350", "issn:0020-8183" ], "title": "Dividend Government and U.S. Trade Policy: Theory and Evidence", "authors": [ "Lohmann, Susanne", "O'Halloran, Sharyn" ] }, { "ID": [ "icpsr:19560", "issn:0022-3433" ], "title": "Empire by Invitation? The United States and Western Europe, 1945-1952", "authors": [ "Lundestad, Geir" ] }, { "ID": [ "icpsr:20175", "issn:0145-5532" ], "title": "Immigrant Support for the American Socialist Party, 1912 and 1920", "authors": [ "Marks, Gary", "Burbank, Matthew" ] }, { "ID": [ "icpsr:20443", "issn:0038-4941" ], "title": "The Changing Geographic Structure of Black and White Segregation in the United States", "authors": [ "Massey, Douglas S.", "Hajnal, Zoltan L." ] }, { "ID": [ "icpsr:20593", "issn:0038-4941" ], "title": "Demographic and Economic Determinants of United States Income Inequality", "authors": [ "Maxwell, Nan L." ] }, { "ID": [ "icpsr:20706", "issn:0730-8884" ], "title": "'Government by Injunction': The U. S. Judiciary and Strike Action in the Late 19th and Early 20th Centuries", "authors": [ "McCammon, Holly J." ] }, { "ID": [ "icpsr:20860", "issn:0037-7732" ], "title": "Fiscal Politics and the Police: Detroit, 1928-76", "authors": [ "McDowall, David", "Loftin, Colin" ] }, { "ID": [ "icpsr:20991", "issn:0020-8833" ], "title": "A Liberal Trade Order? The Long-Run Pattern of Imports to the Advanced Capitalist States", "authors": [ "McKeown, Timothy J." ] }, { "ID": [ "icpsr:40039" ], "title": "The Legacy of Lynching and Southern Homicide: A Spatial Analysis", "authors": [ "Messner, Steven F.", "Baller, Robert D.", "Zevenbergen, Matthew P." ] }, { "ID": [ "icpsr:40041", "issn:0003-1224" ], "title": "The legacy of lynching and southern homicide", "authors": [ "Messner, Steven F.", "Baller, Robert D.", "Zevenbergen, Matthew P." ] }, { "ID": [ "icpsr:21524", "issn:0161-5440" ], "title": "Estimating Interregional Migration in the United Staes form Sample Data: A Procedural Note", "authors": [ "Miller, Ann R." ] }, { "ID": [ "icpsr:21840", "issn:0022-2445" ], "title": "Urbanization and the Malleable Household: An Examination of Boarding and Lodging in American Families", "authors": [ "Modell, John", "Hareven, Tamara K." ] }, { "ID": [ "icpsr:22989", "issn:1040-0656" ], "title": "'We do not want our girls to marry foreigners': Gender, race, and American citizenship", "authors": [ "Nicolosi, Ann Marie" ] }, { "ID": [ "icpsr:40356", "issn:1040-0656" ], "title": "We Do Not Want Our Girls to Marry Foreigners: Gender, Race, and American Citizenship", "authors": [ "Nicolosi, Ann Marie" ] }, { "ID": [ "icpsr:23474", "issn:0037-7732" ], "title": "Causes in the Shifts in Occupational Segregation of the Foreign-Born: Evidence from American Cities, 1870-1880", "authors": [ "Olzak, Susan D." ] }, { "ID": [ "icpsr:92973", "issn:0009-8078" ], "title": "Measures of war: A quantitative examination of the Civil War's destructiveness in the Confederacy", "authors": [ "Paskoff, Paul F." ] }, { "ID": [ "icpsr:24940", "issn:0277-9536" ], "title": "Childhood conditions that predict survival to advanced ages among African-Americans", "authors": [ "Preston, Samuel H.", "Hill, Mark E.", "Drevenstedt, Greg L." ] }, { "ID": [ "icpsr:24953", "issn:0037-7732" ], "title": "Between Depression and Prosperity? Changes in the Community Context of Historical African American Migration", "authors": [ "Price-Spratlen, Townsand" ] }, { "ID": [ "icpsr:68051", "issn:0145-5532" ], "title": "Urban destination selection among African Americans during the 1950s Great Migration", "authors": [ "Price-Spratlen, Townsand" ] }, { "ID": [ "icpsr:50112", "issn:0166-0462" ], "title": "Moving to nice weather", "authors": [ "Rappaport, Jordan" ] }, { "ID": [ "icpsr:36346", "issn:0161-2387" ], "title": "U.S. Urban Decline and Growth, 1950 to 2000", "authors": [ "Rappaport, Jordan" ] }, { "ID": [ "icpsr:25348", "issn:0003-1224" ], "title": "Failed Populism: Movement-Party Disjuncture in North Carolina, 1890 to 1900", "authors": [ "Redding, Kent" ] }, { "ID": [ "icpsr:25349", "issn:0161-5440" ], "title": "Estimating levels and modeling determinants of black and white voter turnout in the South, 1880 to 1912", "authors": [ "Redding, Kent", "James, David R." ] }, { "ID": [ "icpsr:26432", "issn:0730-8884" ], "title": "'Provided Only That Their Voice Was Strong': Insurgency and Organization of American Labor from NRA to Taft-Hartley", "authors": [ "Rubin, Beth A.", "Griffin, Larry J.", "Wallace, Michael" ] }, { "ID": [ "icpsr:26466", "issn:0003-1224" ], "title": "The origins of African-American family structure", "authors": [ "Ruggles, Steven" ] }, { "ID": [ "icpsr:26926" ], "title": "Racial and Ethnic Groups", "authors": [ "Schaefer, Richard T." ] }, { "ID": [ "icpsr:27007" ], "title": "Educational documentation to accompany the great American history machine", "authors": [ "Schlatterer, Helen Marie Fell" ] }, { "ID": [ "icpsr:111679", "issn:2154-4727" ], "title": "The literate South: Reading before Emancipation", "authors": [ "Schweiger, Beth Barton" ] }, { "ID": [ "icpsr:27502", "issn:0145-5532" ], "title": "'New political history': Some statistical questions raised", "authors": [ "Shade, William G." ] }, { "ID": [ "icpsr:27884", "issn:0002-1482" ], "title": "An assessment of the frontier's influence on voter turnout", "authors": [ "Shortridge, Ray Myles" ] }, { "ID": [ "icpsr:93580" ], "title": "The New Deal and the Diffusion of Tractors in the 1930s", "authors": [ "Sorensen, Todd", "Fishback, Price", "Kantor, Shawn" ] }, { "ID": [ "icpsr:28727", "issn:0037-7732" ], "title": "Populism and black lynching in Georgia, 1890-1900", "authors": [ "Soule, Sarah A." ] }, { "ID": [ "icpsr:28779", "issn:0145-5532" ], "title": "Aid to Sharecroppers: How Agrarian Class Structure and Tenant-Farmer Politics Influenced Federal Relief in the South, 1933-1935", "authors": [ "Southworth, Caleb" ] }, { "ID": [ "icpsr:28950", "issn:0192-513X" ], "title": "Divorce and Suicide: A Time Series Analysis, 1933-1970", "authors": [ "Stack, Steven" ] }, { "ID": [ "icpsr:28948", "issn:0022-2445" ], "title": "The Effect of Domestic/Religious Individualism on Suicide, 1954-1978", "authors": [ "Stack, Steven" ] }, { "ID": [ "icpsr:29057", "issn:0161-5440" ], "title": "Census Manuscript Schedules Matched with Property Tax Lists: A Source of Information on Long-Term rends in Wealth Inequality", "authors": [ "Steckel, Richard H." ] }, { "ID": [ "icpsr:29148", "issn:1078-0874" ], "title": "Urban Underrepresentation in the U.S. Senate", "authors": [ "Stephens, G. Ross" ] }, { "ID": [ "icpsr:92455", "issn:0022-0507" ], "title": "The geography of wage discrimination in the pre-civil rights South", "authors": [ "Sundstrom, William A." ] }, { "ID": [ "icpsr:92458", "issn:0022-0507" ], "title": "Important or impotent? Taking another look at the 1920 California Alien Land Law", "authors": [ "Suzuki, Masao" ] }, { "ID": [ "icpsr:29700", "issn:0022-2445" ], "title": "Broken Homes: Stable Risk, Changing Reasons, Changing Forms", "authors": [ "Sweetser, Dorrian Apple" ] }, { "ID": [ "icpsr:92457" ], "title": "From Home to Hospital: The Evolution of Childbirth in the United States, 1927-1940", "authors": [ "Thomasson, Melissa A." ] }, { "ID": [ "icpsr:30093", "issn:0022-0027" ], "title": "War, Inflation, and the Kondratieff Long Wave", "authors": [ "Thompson, William R.", "Zuk, L. Gary" ] }, { "ID": [ "icpsr:30290", "issn:0037-7732" ], "title": "African Americans and immigrants in northern cities: The effects of relative group size on occupational standing in 1920", "authors": [ "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:30288", "issn:0037-7732" ], "title": "Educational Selection in the Migration of Southern Blacks, 1880-1990", "authors": [ "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:30285", "issn:0003-1224" ], "title": "The spatial diffusion of fertility: A cross-sectional analysis of counties in the American South, 1940", "authors": [ "Tolnay, Stewart E." ] }, { "ID": [ "icpsr:30291", "issn:0003-1224" ], "title": "Racial Violence and Black Migration in the American South, 1910 to 1930", "authors": [ "Tolnay, Stewart E.", "Beck, E.M." ] }, { "ID": [ "icpsr:30292", "issn:0037-7732" ], "title": "Black lynchings: The power threat hypothesis revisited", "authors": [ "Tolnay, Stewart E.", "Beck, E.M.", "Massey, James L." ] }, { "ID": [ "icpsr:30295", "issn:0037-7732" ], "title": "'Narrow and Filthy Alleys of the City'?: The Residential Settlement Patterns of Black Southern Migrants to the North", "authors": [ "Tolnay, Stewart E.", "Crowder, Kyle D.", "Adelman, Robert M." ] }, { "ID": [ "icpsr:30296", "issn:0002-9602" ], "title": "Vicarious Violence: Spatial Effects on Southern Lynchings, 1980-1919", "authors": [ "Tolnay, Stewart E.", "Deane, Glenn", "Beck, E.M." ] }, { "ID": [ "icpsr:30297", "issn:0070-3370" ], "title": "The Persistence of high fertility in the American South on the eve of the baby boom", "authors": [ "Tolnay, Stewart E.", "Glynn, Patricia J." ] }, { "ID": [ "icpsr:30620", "issn:0037-7732" ], "title": "Increasing Scale and Spatial Differentiation: New Tests of Two Theories from Shevky and Bell", "authors": [ "Udry, J. Richard" ] }, { "ID": [ "icpsr:30914", "issn:0020-8833" ], "title": "Democratic Convergence and Free Trade", "authors": [ "Verdier, Daniel" ] }, { "ID": [ "icpsr:31224", "issn:0038-0407" ], "title": "Schooling or Working? Public Education, Racial Politics, and the Organization of Production in 1910", "authors": [ "Walters, Pamela Barnhouse", "McCammon, Holly J.", "James, David R." ] }, { "ID": [ "icpsr:31225", "issn:0002-9602" ], "title": "The Family Economy, Work, and Educational Participation in the United States, 1890-1940", "authors": [ "Walters, Pamela Barnhouse", "O'Connell, Philip J." ] }, { "ID": [ "icpsr:31226", "issn:0003-1224" ], "title": "Educational expansion and economic output in the United States, 1890-1969: A production function analysis", "authors": [ "Walters, Pamela Barnhouse", "Robinson, Richard" ] }, { "ID": [ "icpsr:31767", "issn:0038-0407" ], "title": "Tug-of-war: Political mobilization and access to schooling in the southern racial state", "authors": [ "Werum, Regina E." ] }, { "ID": [ "icpsr:92732", "issn:0167-5923" ], "title": "Sending or receiving stations? The dual influence of railroads in early 20th-century Great Plains settlement", "authors": [ "White, Katherine J. Curtis" ] }, { "ID": [ "icpsr:92978" ], "title": "Temporal and Spatial Variation in 20th Century U.S. Great Plains Population Change", "authors": [ "White, Katherine J. Curtis" ] }, { "ID": [ "icpsr:92977" ], "title": "Growth Curve Modeling: An Alternative Approach to Estimating Patterns of Population Change", "authors": [ "White, Katherine J. Curtis", "Herting, Jerald R." ] }, { "ID": [ "icpsr:32058", "issn:0022-2445" ], "title": "The Decline in Men's Labor Force Participation and Income and the Changing Structure of Family Economic Support", "authors": [ "Wilkie, Jane Riblett" ] }, { "ID": [ "icpsr:40398", "issn:0009-8078" ], "title": "The 1835 anti-Abolition meetings in the South: A new look at the controversy over the Abolition Postal Campaign", "authors": [ "Wyly-Jones, Susan" ] }, { "ID": [ "icpsr:92459", "issn:0034-6446" ], "title": "Black enterprise and the legacy of slavery", "authors": [ "Zajonc, Tristan" ] } ], "grant": { "identifiers": [ "GS-1435, GS-1435A1, GS-1231, GS-2473, and APR75-01320" ], "funders": [ "National Science Foundation. Research Applied to National Needs" ] }, "access": { "landingPage": "http://doi.org/10.3886/ICPSR00003.v1" }, "license": { "copyright": "ICPSR metadata records are licensed under a Creative Commons Attribution-Noncommercial 3.0 United States License." } }


From: George Alter [notifications@github.com] Sent: Tuesday, March 28, 2017 4:21 PM To: biocaddie/prototype_issues Cc: Subscribed Subject: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Inconsistencies between ICPSR's DATS and DataMed.docxhttps://github.com/biocaddie/prototype_issues/files/877339/Inconsistencies.between.ICPSR.s.DATS.and.DataMed.docx Sanda found a number of problems with the records that were ingested from ICPSR DATS. In the attached file she compares one record in DataMED to the way that it appears in the JSON file.

- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/biocaddie/prototype_issues/issues/256, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALPhs82TIQdRALkFulUjOcw5NoFOy9TWks5rqZYOgaJpZM4MsUGY.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_biocaddie_prototype-5Fissues_issues_256-23issuecomment-2D290246184&d=DQMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=d5w4WGsMSBN5okxuESTpnEMKGxfgm8DFlqtMmXNdF8c&m=q8TYyKC_SJ8g4T-8GcA6mjnF1MrW4Hl1go4Cf5YKIEc&s=aC3tud2pGd1Soc65Ei2lIZVVapvee0CtlIOniUnMZwg&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALO1FmOrREgldymLA5FIWGSGBlI4n28Qks5rqtuAgaJpZM4MsUGY&d=DQMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=d5w4WGsMSBN5okxuESTpnEMKGxfgm8DFlqtMmXNdF8c&m=q8TYyKC_SJ8g4T-8GcA6mjnF1MrW4Hl1go4Cf5YKIEc&s=-4GmVVZ8HL9tK3RjdyuSWbyv0F294N1-0CNQtBXWXO8&e=.

altergc commented 7 years ago

I don't know what the "curated JSON" refers to. Sanda's note compares this JSON generated by the ICPSR website https://www.icpsr.umich.edu/icpsrweb/neutral/dats/studies/22626

to this page in DataMED https://datamed.biocaddie.org/display-item.php?repository=0025&id=580ea0355152c648c68d003e&query=india%20human%20development

Also, would it be possible to add Sanda (sandai@umich.edu ) to this Github thread? Thanks. George

sandaionescu commented 7 years ago

Hi. The inconsistencies I pointed out are not between the ICPSR-generated JSON and the ICPSR website, but between the ICPSR-generated JSON and the DataMed website. Please review my comparison table (attached below). I am using ICPSR study no 22626 and in the table the differences are shown by field, side by side, copied and pasted from the two sources mentioned above. On the DataMed site, the study creators are not captured, the description is not displayed, there are two grant numbers instead of one (the second number is invalid), and the keywords have
tags separating them instead of being displayed in a list. Sanda.

Inconsistencies between ICPSR's DATS and DataMed.docx

yul129 commented 7 years ago

Hi Sanda,

Thank you for the clarification, I will take a look today.

Yueling


From: sandaionescu [notifications@github.com] Sent: Thursday, March 30, 2017 8:59 AM To: biocaddie/prototype_issues Cc: Yueling Li; Comment Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Hi. The inconsistencies I pointed out are not between the ICPSR-generated JSON and the ICPSR website, but between the ICPSR-generated JSON and the DataMed website. Please review my comparison table (attached below). I am using ICPSR study no 22626 and in the table the differences are shown by field, side by side, copied and pasted from the two sources mentioned above. On the DataMed site, the study creators are not captured, the description is not displayed, there are two grant numbers instead of one (the second number is invalid), and the keywords have tags separating them instead of being displayed in a list. Sanda.

Inconsistencies between ICPSR's DATS and DataMed.docxhttps://github.com/biocaddie/prototype_issues/files/883028/Inconsistencies.between.ICPSR.s.DATS.and.DataMed.docx

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/biocaddie/prototype_issues/issues/256#issuecomment-290457266, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALPhswWzeyeTNN47XFg4SsYYy3mtvGHoks5rq9FqgaJpZM4MsUGY.

yul129 commented 7 years ago

Hi Sanda,

We have switched the data ingesting point from OAI to ICPSR-generated JSON pages, and modified the transformation file correspondingly. Now the DataMed.org should match and reflect the JSON file generated by ICPSR.

Please let me know if there are any inconsistencies. Yueling


From: sandaionescu [notifications@github.com] Sent: Thursday, March 30, 2017 8:59 AM To: biocaddie/prototype_issues Cc: Yueling Li; Comment Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Hi. The inconsistencies I pointed out are not between the ICPSR-generated JSON and the ICPSR website, but between the ICPSR-generated JSON and the DataMed website. Please review my comparison table (attached below). I am using ICPSR study no 22626 and in the table the differences are shown by field, side by side, copied and pasted from the two sources mentioned above. On the DataMed site, the study creators are not captured, the description is not displayed, there are two grant numbers instead of one (the second number is invalid), and the keywords have tags separating them instead of being displayed in a list. Sanda.

Inconsistencies between ICPSR's DATS and DataMed.docxhttps://github.com/biocaddie/prototype_issues/files/883028/Inconsistencies.between.ICPSR.s.DATS.and.DataMed.docx

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/biocaddie/prototype_issues/issues/256#issuecomment-290457266, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALPhswWzeyeTNN47XFg4SsYYy3mtvGHoks5rq9FqgaJpZM4MsUGY.

altergc commented 6 years ago

The rendering of DATS JSON from ICPSR still has errors. Sanda prepared this review:

Review of the ICPSR metadata on the DataMed website October 2017

  1. Many of the records on DataMed do not display the "creator/creators" field (i.e., author or authors of the study) Ex: Detroit Area Study, 1968 (ICPSR 7324) Australian National Political Attitudes, 1967 (ICPSR 7282) Political Change in Britain, 1966 (ICPSR 7234) -But others do, for example ANES 1970 Time Series Study (ICPSR 7298) Something is definitely not right with the way the "creator" field is not parsed.

  2. I have not found related publications for any of the studies. They are not listed in the main record on DataMed, nor have I found any links to a separate page that would list them.

  3. In the ICPSR .json, the study description includes some html tags for formatting. These are generally not seen in the DataMed description (which is what we want) but there are some exceptions. For example in the description for Detroit Area Study, 1959 (ICPSR 7323) a <p> tag appears at the beginning of the description:

"<p>This study of 767 adults in the Detroit metropolitan area provides information on their religious beliefs and practices, as well as their feelings about various forms of taxation such as sales tax, income tax, and property tax. The collection was a combination of two separate studies..."

A question for the DataMed team would be if they wanted the html formatting tags removed from the .json. Matthew could do that when he updates to the most recent version of DATS.

jgrethe commented 6 years ago

Hi Yueling, Can you check on the above...

@aegururaj : for potential UI issues - who should be added to this ticket?

yul129 commented 6 years ago

Yes, i am working this with Burak now.


From: Jeffrey S. Grethe, Ph.D. [notifications@github.com] Sent: Wednesday, October 04, 2017 11:53 AM To: biocaddie/prototype_issues Cc: Yueling Li; Assign Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Hi Yueling, Can you check on the above...

@aegururajhttps://github.com/aegururaj : for potential UI issues - who should be added to this ticket?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/biocaddie/prototype_issues/issues/256#issuecomment-334255044, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALPhsxouQy1SQobQWmegM4JBsUgV0sqYks5so9RCgaJpZM4MsUGY.

bozyurt commented 6 years ago

ICSPR is reindexed with JSON api and available at

http://biocaddie.scicrunch.io/icpsr_20171004/dataset

Burak


From: yul129 [notifications@github.com] Sent: Wednesday, October 04, 2017 12:05 PM To: biocaddie/prototype_issues Cc: Subscribed Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Yes, i am working this with Burak now.


From: Jeffrey S. Grethe, Ph.D. [notifications@github.com] Sent: Wednesday, October 04, 2017 11:53 AM To: biocaddie/prototype_issues Cc: Yueling Li; Assign Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Hi Yueling, Can you check on the above...

@aegururajhttps://github.com/aegururaj : for potential UI issues - who should be added to this ticket?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/biocaddie/prototype_issues/issues/256#issuecomment-334255044, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALPhsxouQy1SQobQWmegM4JBsUgV0sqYks5so9RCgaJpZM4MsUGY.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/biocaddie/prototype_issues/issues/256#issuecomment-334258218, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHLczM6WZjRHNvj0nGdeaJFe1WcZT7LKks5so9bvgaJpZM4MsUGY.

sandaionescu commented 6 years ago

When will it be available on the public site? That's when I need to check for accuracy. Thanks, Sanda.

aegururaj commented 6 years ago

Sanda, we’ll let you know as soon as the UI gets updated.

Thanks

Anu

From: sandaionescu notifications@github.com Reply-To: biocaddie/prototype_issues reply@reply.github.com Date: Thursday, October 5, 2017 at 12:25 PM To: biocaddie/prototype_issues prototype_issues@noreply.github.com Cc: "Gururaj, Anupama" Anupama.E.Gururaj@uth.tmc.edu, Mention mention@noreply.github.com Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

When will it be available on the public site? That's when I need to check for accuracy. Thanks, Sanda.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_biocaddie_prototype-5Fissues_issues_256-23issuecomment-2D334518512&d=DwMCaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=d5w4WGsMSBN5okxuESTpnEMKGxfgm8DFlqtMmXNdF8c&m=8McFVlollBdAteoghHk42WDJ-L6FnA9YEj9O2xLCxoo&s=7l12VeRh1bOU7vRc_u5k_trNLa-NBsXJgaHgd-3USqU&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALO1FiCshzZIeReoYE5rUwXKsX4uj4wIks5spQL6gaJpZM4MsUGY&d=DwMCaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=d5w4WGsMSBN5okxuESTpnEMKGxfgm8DFlqtMmXNdF8c&m=8McFVlollBdAteoghHk42WDJ-L6FnA9YEj9O2xLCxoo&s=L50eBmFQtzKRdE24w9Azo-_FNcQRhQZJfBajgHndJqk&e=.

JingAn2017 commented 6 years ago

I am still waiting for Duo enrollment email to connect to the datamed server. As soon as I get the account activated, I will update the UI. Thank you.

JingAn2017 commented 6 years ago

@sandaionescu the development server UI is updated, please have a look at. http://datamedbeta.biocaddie.org/

sandaionescu commented 6 years ago

Hi, I do not have access to the development server (need authentication) so please do let me know when this becomes live on the public site. Thank you, Sanda.

aegururaj commented 6 years ago

Sanda,

I sent you the login information to the dev server. Please check on the dev server. Once we get the green signal from you, we’ll update the production server.

Thanks

Anu

From: sandaionescu notifications@github.com Reply-To: biocaddie/prototype_issues reply@reply.github.com Date: Thursday, October 12, 2017 at 2:47 PM To: biocaddie/prototype_issues prototype_issues@noreply.github.com Cc: "Gururaj, Anupama" Anupama.E.Gururaj@uth.tmc.edu, Mention mention@noreply.github.com Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Hi, I do not have access to the development server (need authentication) so please do let me know when this becomes live on the public site. Thank you, Sanda.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_biocaddie_prototype-5Fissues_issues_256-23issuecomment-2D336230472&d=DwMCaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=d5w4WGsMSBN5okxuESTpnEMKGxfgm8DFlqtMmXNdF8c&m=SDtQC8IWFWBQmv3y1GQn81wQNVKWC3ftoqzsUnX_e0g&s=7gzeA85W8ejjQ-UWhdZdBMC8NknK_KLZH_yPoCnz2-I&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALO1Fo5j18br0z9UBOlT63Q3wNEKEUNAks5srl68gaJpZM4MsUGY&d=DwMCaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=d5w4WGsMSBN5okxuESTpnEMKGxfgm8DFlqtMmXNdF8c&m=SDtQC8IWFWBQmv3y1GQn81wQNVKWC3ftoqzsUnX_e0g&s=vsNiS5P6pDj5f2ZjvL9c9VpmG8GhQ0Ntfzcp2lSmNL0&e=.

sandaionescu commented 6 years ago

Thanks, Anu. I'll try to check as soon as I can. Sanda.

Sanda Ionescu ICPSR University of Michigan Ann Arbor, MI Phone: 734-615-2932

On Thu, Oct 12, 2017 at 5:17 PM, aegururaj notifications@github.com wrote:

Sanda,

I sent you the login information to the dev server. Please check on the dev server. Once we get the green signal from you, we’ll update the production server.

Thanks

Anu

From: sandaionescu notifications@github.com Reply-To: biocaddie/prototype_issues reply@reply.github.com Date: Thursday, October 12, 2017 at 2:47 PM To: biocaddie/prototype_issues prototype_issues@noreply.github.com Cc: "Gururaj, Anupama" Anupama.E.Gururaj@uth.tmc.edu, Mention < mention@noreply.github.com> Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Hi, I do not have access to the development server (need authentication) so please do let me know when this becomes live on the public site. Thank you, Sanda.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense. proofpoint.com/v2/url?u=https-3A__github.com_biocaddie_ prototype-5Fissues_issues_256-23issuecomment-2D336230472&d= DwMCaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r= d5w4WGsMSBN5okxuESTpnEMKGxfgm8DFlqtMmXNdF8c&m= SDtQC8IWFWBQmv3y1GQn81wQNVKWC3ftoqzsUnX_e0g&s=7gzeA85W8ejjQ- UWhdZdBMC8NknK_KLZH_yPoCnz2-I&e=, or mute the threadhttps://urldefense. proofpoint.com/v2/url?u=https-3A__github.com_notifications_ unsubscribe-2Dauth_ALO1Fo5j18br0z9UBOlT63Q3wNEKEUNAks5srl68gaJpZM4MsUGY&d= DwMCaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r= d5w4WGsMSBN5okxuESTpnEMKGxfgm8DFlqtMmXNdF8c&m= SDtQC8IWFWBQmv3y1GQn81wQNVKWC3ftoqzsUnX_e0g&s= vsNiS5P6pDj5f2ZjvL9c9VpmG8GhQ0Ntfzcp2lSmNL0&e=.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/biocaddie/prototype_issues/issues/256#issuecomment-336273544, or mute the thread https://github.com/notifications/unsubscribe-auth/AT_EpjLLhW-OI_LY1ZILBboG6PUK_-usks5sroHygaJpZM4MsUGY .

sandaionescu commented 6 years ago

I have checked the ICPSR harvested metadata on the development server.

  1. Some studies are still missing the "creator" field. Ex.: Israeli Election Study, 1977 , ICPSR 2989; Reporting Sexual Assault to the Police in Honolulu, Hawaii, 1987-1992, ICPSR 3051; Chinese Household Income Project, 1995, ICPSR 3012. This seems to be the case for those studies that have individual authors, rather than organizations.
  2. I still do not see any related publications. These are entered in the ICPSR .json files but I am not sure if a specific decision has been made (or not) on how to display them. In many cases there are dozens or even hundreds of related publications associated with a specific study, and I believe it would make more sense to enable a link to these from the DataMed study page, rather than display them on the same page. At ICPSR we display only a few, with a link to the complete list. Sanda.

yul129 commented 6 years ago

@sandaionescu Below are my answers/solutions:

  1. the reason for the missing creators field: creator.name have been tagged inconsistently in ICPSR JSON files. E.g. it is creator.name in ICPSR 1, but creator.fullName in ICPSR 7282. I have wrote in both situations into transformation file now, and hopeful would catch both fields.

  2. similarly, the schema used to define citations in ICPSR JSON files are also inconsistent. E.g. citation.identifiers exists in ICPSR 1, but does not exist in ICPSR 7282.. The transformation file will fetch the citation id infromation whenever it is available.

I could not find any publication information in the ICPSR JSON files, only the citations......

altergc commented 6 years ago

Hi Yueling, Thanks for finding these issues in the DATS JSON at ICPSR. Since the Oxford team is working on a new version of DATS, we will address these things when we move to the next version of DATS. Thanks, George

bozyurt commented 6 years ago

ICPSR is reindexed.

http://biocaddie.scicrunch.io/icpsr_20171016/dataset

Burak

On 10/16/2017 12:22 PM, yul129 wrote:

@sandaionescu https://github.com/sandaionescu Below are my answers/solutions:

1.

the reason for the missing creators field: creator.name have been
tagged inconsistently in ICPSR JSON files. E.g. it is creator.name
in ICPSR 1
<https://www.icpsr.umich.edu/icpsrweb/neutral/dats/studies/1>, but
creator.fullName in ICPSR 7282
<https://www.icpsr.umich.edu/icpsrweb/neutral/dats/studies/7282>.
I have wrote in both situations into transformation file now, and
hopeful would catch both fields.

2.

similarly, the schema used to define citations in ICPSR JSON files
are also inconsistent. E.g. citation.identifiers exists in ICPSR 1
<https://www.icpsr.umich.edu/icpsrweb/neutral/dats/studies/1>, but
does not exist in ICPSR 7282.
<https://www.icpsr.umich.edu/icpsrweb/neutral/dats/studies/7282>.
The transformation file will fetch the citation id infromation
whenever it is available.

I could not find any publication information in the ICPSR JSON files, only the citations......

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/biocaddie/prototype_issues/issues/256#issuecomment-337005544, or mute the thread https://github.com/notifications/unsubscribe-auth/AHLczN10sH4MsHkc6zrCayP4Ira543CZks5ss60MgaJpZM4MsUGY.

-- I. Burak Ozyurt PhD Project Scientist University of California, San Diego 9500 Gilman Drive, M/C 0608 La Jolla, CA 92093-0608

altergc commented 6 years ago

Matthew Richardson sent the following explanations for the comments sent by Yueling:


Here are the two examples:

http://www.icpsr.umich.edu/icpsrweb/neutral/dats/studies/1 http://www.icpsr.umich.edu/icpsrweb/neutral/dats/studies/7282

1) Creator.name is used for organizations; creator.fullName is used for people. That's not inconsistent; that is the spec.

https://github.com/biocaddie/DATS/blob/master/dataset_schema.json

creators is an array of one of either

http://biocaddie.github.io/dats/person_schema.json http://biocaddie.github.io/dats/organization_schema.json

The person schema expresses names as firstName, lastName, middleInitial...and fullName, for when you don't have the name parts broken out.

The organization schema uses name.

2) For citations, identifiers are provided when we have them. We do not have DOIs for all the citations in the bibliography. For example, conference papers almost never have DOIs; reports seldom do; journal articles usually have DOIs.

I don't know what is meant by "I could not find any publication information in the ICPSR JSON files, only the citations...." What is the difference between publications and citations in this context?

yul129 commented 6 years ago

Thank you for the clarification, Matthew.

  1. -Both creator.name and creator.fullName will be mapped to creator.name in DATS model.
  2. -I have mapped both identifiers and alternateIdentifiers to the model whenever they are available.
  3. -Below are the definitions of publication and citation listed in DATS: The primaryPublications refers to the primary publication(s) associated with the dataset, usually describing how the dataset was produced. Citations refers to the publication(s) that cite this dataset.

From: George Alter [notifications@github.com] Sent: Tuesday, October 17, 2017 5:15 AM To: biocaddie/prototype_issues Cc: Yueling Li; Assign Subject: Re: [biocaddie/prototype_issues] Ingest from DATS problems (#256)

Matthew Richardson sent the following explanations for the comments sent by Yueling:


Here are the two examples:

http://www.icpsr.umich.edu/icpsrweb/neutral/dats/studies/1 http://www.icpsr.umich.edu/icpsrweb/neutral/dats/studies/7282

  1. Creator.name is used for organizations; creator.fullName is used for people. That's not inconsistent; that is the spec.

https://github.com/biocaddie/DATS/blob/master/dataset_schema.json

creators is an array of one of either

http://biocaddie.github.io/dats/person_schema.json http://biocaddie.github.io/dats/organization_schema.json

The person schema expresses names as firstName, lastName, middleInitial...and fullName, for when you don't have the name parts broken out.

The organization schema uses name.

  1. For citations, identifiers are provided when we have them. We do not have DOIs for all the citations in the bibliography. For example, conference papers almost never have DOIs; reports seldom do; journal articles usually have DOIs.

I don't know what is meant by "I could not find any publication information in the ICPSR JSON files, only the citations...." What is the difference between publications and citations in this context?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/biocaddie/prototype_issues/issues/256#issuecomment-337212273, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALPhsytQybiydE70lRJyk3P_Brjmy7K5ks5stJp1gaJpZM4MsUGY.

JingAn2017 commented 6 years ago

@sandaionescu The development server UI is updated, please check. http://datamedbeta.biocaddie.org/

Thank you.

yul129 commented 6 years ago
  1. The "creators" filed is fixed, and now it returns both person's name and name of organization.
  2. The citations information are still missing on the development server: http://datamedbeta.biocaddie.org/display-item.php?repository=0025&id=59d533545152c65187649952 however, it exists on elastic search. @JingAn2017 screen shot 2017-10-17 at 3 14 19 pm
JingAn2017 commented 6 years ago

@sandaionescu @yul129 I checked the code of displaying and found there is a field of "citation" which already exists, will we use this name to keep unified?

Thank you.

yul129 commented 6 years ago

@JingAn2017 Yes, please map "citations" to "citation". Thanks.

yul129 commented 6 years ago

@JingAn2017 I updated the field name in transfermation file, and burak will re-index it today.

JingAn2017 commented 6 years ago

@yul129 Sure, then I will map to "citation". Thank you.

JingAn2017 commented 6 years ago

@sandaionescu The development server UI is updated, please check. http://datamedbeta.biocaddie.org/

Thank you.

yul129 commented 6 years ago

The citation section and creators are up in dev, just tested it.

JingAn2017 commented 6 years ago

The production server is updated.