GSA / project-open-data-dashboard

Project Open Data Dashboard
http://labs.data.gov/dashboard/
Other
138 stars 119 forks source link

Add crawl for DAP script installation #55

Open jlberryhill opened 10 years ago

jlberryhill commented 10 years ago

"Use and Impact" tab (from #54), could you please add a crawl that checks each agency's /data/ page for DAP script installation. This is how a lot of agencies are identifying their top data users.

Once metadata v1.1 is supported in January, I think it would also be good to check each agency's accessURL for DAP as well, since this will let them get more detailed analytics on their landing pages.

paulOMB commented 9 years ago

@jlberryhill and I are starting to go through Dashboard evaluations. Would it be possible to get the DAP crawl up for this milestone, or at least a manual Yes/No box for DAP on the /data/ page to help agencies identify data users?

rebeccawilliams commented 8 years ago

A manual Yes/No box was added, but automation still needs to be added.

rebeccawilliams commented 8 years ago

@JJediny Look for a consistent way to parse DAP script on /data pages.

JJediny commented 8 years ago

DAP provided a list of agencies with the script functioning on all /data endpoints, as of 12/30/15... It might make it more practical to do a manual crosswalk to see if its worth the effort to automate? @rebeccawilliams @philipashlock

Page Pageviews
bls.gov/data/ 177,537
sdo.gsfc.nasa.gov/data/ 41,944
fas.usda.gov/data 26,022
census.gov/population/international/data/ 13,617
census.gov/popest/data/ 13,023
census.gov/econ/census/data/ 12,502
ladsweb.nascom.nasa.gov/data/ 10,959
census.gov/hhes/www/poverty/data/ 9,580
modis.gsfc.nasa.gov/data/ 7,823
www2.census.gov/programs-surveys/acs/summary_file/2014/data/ 5,074
collegescorecard.ed.gov/data/ 4,628
census.gov/population/race/data/ 3,973
cdc.gov/diabetes/data/ 3,922
census.gov/population/metro/data/ 3,407
water.usgs.gov/data/ 2,295
cdc.gov/breastfeeding/data/ 2,200
bis.doc.gov/data 1,920
census.gov/hhes/www/income/data/ 1,860
data.nasa.gov/data 1,841
census.gov/population/projections/data/ 1,781
nist.gov/pml/data/ 1,604
census.gov/population/age/data/ 1,598
cdc.gov/cancer/dcpc/data/ 1,591
beta.fec.gov/data/ 1,115
developer.epa.gov/category/data/ 915
eauth.va.gov/wssweb/wss-common-webparts/mvc/updatepersonalcontactinfo/data 900
census.gov/population/foreign/data/ 884
census.gov/cps/data/ 813
census.gov/population/hispanic/data/ 738
dhs.gov/topic/data 685
census.gov/hhes/socdemo/education/data/ 659
marinecadastre.gov/data/ 636
glerl.noaa.gov/data/ 549
cdc.gov/physicalactivity/data/ 525
cdc.gov/healthyyouth/data/ 454
cdc.gov/niosh/data/ 368
notalone.gov/data/ 357
stats.bls.gov/data/ 346
www2.census.gov/programs-surveys/acs/summary_file/2013/data/ 326
census.gov/hhes/families/data/ 315
cdc.gov/aging/data/ 300
cnpp.usda.gov/data 300
smap.jpl.nasa.gov/data/ 283
gapanalysis.usgs.gov/gaplandcover/data/ 278
wa.water.usgs.gov/data/ 243
esrl.noaa.gov/gmd/dv/data/ 207
census.gov/hhes/veterans/data/ 202
census.gov/hhes/samesex/data/ 149
disa.mil/network-services/data 140
fermi.gsfc.nasa.gov/ssc/data/ 137
census.gov/population/ancestry/data/ 136
census.gov/housing/nychvs/data/ 123
ca.water.usgs.gov/data/ 120
nifa.usda.gov/data 108
census.gov/hhes/socdemo/marriage/data/ 99
sohowww.nascom.nasa.gov/data/ 99
water.usgs.gov/watuse/data/ 96
census.gov/hhes/commuting/data/ 86
gapanalysis.usgs.gov/padus/data/ 79
cdc.gov/drugoverdose/data/ 75
census.gov/hhes/www/hlthins/data/ 75
gapanalysis.usgs.gov/species/data/ 75
census.gov/did/www/saipe/data/ 72
commerce.gov/categories/data 72
census.gov/people/io/data/ 70
math.nist.gov/matrixmarket/data/ 70
census.gov/hhes/migration/data/ 69
useiti.doi.gov/data/ 66
ngdc.noaa.gov/mgg/global/relief/etopo1/data/ 61
nist.gov/data/ 59
cbp.gov/trade/priority-issues/adcvd/data 58
gapanalysis.usgs.gov/data/ 56
census.gov/hhes/fertility/data/ 54
gps.gov/systems/gps/modernization/sa/data/ 52
sefsc.noaa.gov/data/ 42
census.gov/population/intmigration/data/ 39
census.gov/research/data/ 39
ssa.gov/data/ 39
fas.usda.gov/data/ 38
socialsecurity.gov/open/data/ 36
sealevel.jpl.nasa.gov/data/ 34
coast.noaa.gov/htdata/lidar1_z/geoid12a/data/ 31
www3.epa.gov/epawaste/inforesources/data/ 31
transportation.gov/data 29
eddataexpress.ed.gov/data-elements.cfm/tool/data/ 28
census.gov/people/disability/data/ 26
census.gov/people/eeotabulation/data/ 25
census.gov/did/www/schooldistricts/data/ 24
usaid.gov/data 24
cdc.gov/foodsafety/fdoss/data/ 20
nomads.ncdc.noaa.gov/data/ 20
va.gov/data/ 20
census.gov/did/www/saipe/data/statecounty/data/ 18
census.gov/housing/census/data/ 18
justice.gov/data/ 18
www2.census.gov/programs-surveys/acs/summary_file/2011/data/ 17
nrc.gov/data/ 16
www2.census.gov/programs-surveys/acs/summary_file/2009/data/ 16
whitehouse.gov/administration/eop/aapi/data/data 15
water.usgs.gov/nawqa/pnsp/data/ 13
commerce.gov/page/data 12
census.gov/people/wealth/data/ 11
mars.nasa.gov/msl/mission/communicationwithearth/data/ 10
opm.gov/data/ 10
www2.census.gov/programs-surveys/acs/summary_file/2005/data/ 10
ams.usda.gov/services/transportation-analysis/data 9
census.gov/did/www/sahie/data/ 9
census.gov/people/disabilityemptab/data/ 8
ssa.gov/open/data/ 8
census.gov/people/childsupport/data/ 7
ozoneaq.gsfc.nasa.gov/data/ 6
rebeccawilliams commented 7 years ago

This can be closed to focus on simplification and automation.

This should perhaps instead be added to pulse.cio.gov cc @konklone