search

GSA / data.gov

Main repository for the data.gov service
https://data.gov
Other
592 stars 91 forks source link

resources on Catalog with HTTP links are not downloadable in Chrome #3476

Closed FuhuXia closed 1 year ago

FuhuXia commented 2 years ago

User complains some dataset resources are not downloadable. Clicking on the download link has no effect in Chrome browser. It turns out a Chrome security feature blocks Mixed Content http resource download but gives no visual indication of the block.

How to reproduce

Use Chrome broser to download resource from datasets:

Example datasets from user email: https://catalog.data.gov/dataset/reca-awards-by-place-of-residence-state-country-as-of-date https://catalog.data.gov/dataset/reca-claims-summary-report https://catalog.data.gov/dataset/reca-program-outreach-log

Expected behavior

Resources can be downloaded.

Actual behavior

Click on download button but nothing happens. Open the Chrome developer console it shows a Mixed Content error:

Mixed Content: The site at '<URL>' was loaded over a secure connection, but the file at '<URL>' was redirected through an insecure connection. This file should be served over HTTPS. This download has been blocked. See <URL> for more details.

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

jbrown-xentity commented 2 years ago

Should investigate how many organizations this affects, and reach out to them directly. Could consider errors/rejection on harvest, gov policy says urls should be https: https://https.cio.gov/guide/

jbrown-xentity commented 1 year ago

Related to https://github.com/GSA/data.gov/issues/3974 and https://github.com/GSA/data.gov/issues/2985

jbrown-xentity commented 1 year ago

I think this should be investigated as to how large of a problem it is first. I would bet that since these links are usually made available publicly similarly by organization that this is isolated to a few organizations...

nickumia-reisys commented 1 year ago

Picking this up to assess the scope of the problem.

nickumia-reisys commented 1 year ago

Will do the full analysis, but from the preliminary data of 30k datasets, this problem seems to be rampant?

http://surveys.ngdc.noaa.gov/mgg/NOS/coast/H08001-H10000/H09980/GEODAS/H09980.a93.gz
http://surveys.ngdc.noaa.gov/mgg/NOS/coast/H08001-H10000/H09980/DR/H09980.pdf
http://gcmd.nasa.gov/MailComments/MailComments.jsf?rcpt=gcmduso
http://gcmd.nasa.gov/learn/keyword_list.html
http://www.getty.edu/research/tools/vocabularies/tgn/index.html
http://dx.doi.org/10.5066/F7DV1H10
URL:http://SoilDataMart.nrcs.usda.gov/
URL:http://SoilDataMart.nrcs.usda.gov/
URL:http://SoilDataMart.nrcs.usda.gov/
URL:http://SoilDataMart.nrcs.usda.gov/
URL:http://SoilDataMart.nrcs.usda.gov/
http://erddap.secoora.org/erddap/tabledap/gov_usgs_waterdata_02307669.html
http://erddap.secoora.org/erddap/tabledap/gov_usgs_waterdata_02307669.graph
http://erddap.secoora.org/erddap/tabledap/gov_usgs_waterdata_02307669
http://erddap.secoora.org/erddap/tabledap/gov_usgs_waterdata_02307669
http://www.response.restoration.noaa.gov
http://www.response.restoration.noaa.gov
http://ccma.nos.noaa.gov/
http://www.ioos.noaa.gov/
http://www.bp.com
http://www.response.restoration.noaa.gov
http://erddap.aoos.org/erddap/tabledap/noaa_nos_co_ops_9451497
http://erddap.aoos.org/erddap/tabledap/noaa_nos_co_ops_9451497
http://dnr.alaska.gov/lrisservices/ls_proxy/email?layerid=31
http://dnr.alaska.gov/lrisservices/ls_proxy/email?layerid=31
ftp://ftp-oceans.ncei.noaa.gov/nodc/archive/arc0001/7600520/

I'll get the number of links per organization as a first pass.

nickumia-reisys commented 1 year ago

Do we care if it's an http link to a webpage vs. a file download? @jbrown-xentity @FuhuXia

FuhuXia commented 1 year ago

Do we care if it's an http link to a webpage vs. a file download? @jbrown-xentity @FuhuXia

no

nickumia-reisys commented 1 year ago

Do we support ftp links?

nickumia-reisys commented 1 year ago

Here's the full list of organizations with non-https links: 

{'352b4532-793d-4075-a03f-05b778a3c43a': 506, '842b8561-cf24-49cf-b901-66c6932a392b': 3, 'a543287f-0731-4645-b100-a29f4f39be97': 33, 'c549d5ce-2b93-4397-ab76-aa2b31d9983a': 236, 'ae156ef5-7869-4748-9ad2-f103cf02a218': 28023, '8dc070a3-41fa-4e78-8680-a0b98d678fb5': 6, '77ade652-dece-4e18-b71b-b5e9be8c86c2': 607, '4d976f1e-ccb5-4dbe-aa98-9d2931fbd215': 179, '29846409-fa62-4648-a3b9-12d071a2a5b2': 11269, 'fb3131aa-ef06-4a00-ad84-67d93a71d7e3': 426, 'f4ca4614-8901-409b-8553-2e994ad10023': 4742, '16980d1c-5e8f-4188-b962-42446f2d3f63': 93910, '5f4f1195-e770-4a2a-8f75-195cd98860ce': 112729, '82b85475-f85d-404a-b95b-89d1a42e9f6b': 218, '6a3f935c-da6f-47f6-a386-1c76afd0b1e0': 66756, '2c2fc21f-21d0-4450-af01-cf8c69b44156': 295, '7f8ee588-32a3-4b6c-b435-d6603c91dbcc': 48, '1f2ebc13-fc03-4bcd-b2c0-dad0bb510b65': 1170, '6cd4f2ba-1583-469c-a6f0-95ba585aeb7e': 27, '5e4d7221-af33-40f5-bbed-78ae1dd8721e': 1, 'a65bf892-ca10-45e5-88f7-9cbaa4bdbfca': 1, '2ea36261-b7ad-4018-92fd-5b1cba9e9fdc': 13, '4455a74e-f5bd-4a7e-941e-9fe5a72eb882': 22, '8148853c-d386-4705-a1c4-41c3212f78e4': 24, '1149ee63-2fff-494e-82e5-9aace9d3b3bf': 72, '9cd95192-4220-45cd-8d96-f720d58643bd': 4, '143529f7-2eef-4a07-b227-93ac9e84fad8': 5513, '0af4eecd-141b-41b3-9367-a5a51ce44b90': 2, 'eca91a93-a8a2-48b9-a949-5003b40b9012': 21, '7c50b56f-e8ac-4bab-bb77-e88ef2637387': 1, '92803bce-3fa9-4dfb-8b70-3baca3835f0c': 120, '2d7e3c15-e0cb-4857-b4d5-40ed6e6d7954': 31, 'bccbad82-abc4-4712-bd29-3e194e7a8042': 18, 'ed01ec43-dcbc-4772-8e20-4c90b457a7a7': 5616, '4af4b741-65f5-4ac6-950d-75e1abc1a7df': 4, '97ae80aa-2507-4920-808f-99a45dc5ef27': 57, 'b98a6f3b-552a-4826-931e-4594420f4596': 598, 'bb10fdb9-d96f-4284-9a5f-9ed269d9dbcb': 4, '1a128ac1-72ce-4027-b46d-a73862031b04': 44, 'a2ea361e-1350-4df1-8d61-4c326b3394ae': 858, 'e9d6fde3-a97c-46c5-ad7d-51579264b7f1': 37, 'fd2d1d4a-6fc9-4010-a5be-154b4419fd1e': 28, '8571f574-7d3d-48f3-8cc7-af44048076a1': 12, '57d66334-c55a-4d71-8c4e-8dd1055f5354': 3, '217e855b-cd64-4ebc-958b-abbbb0f57ac2': 12, '441a7317-0631-4d27-b8bb-dcfaa6be5915': 54, '2cca5b33-6bfa-496a-895f-3301690bbc2c': 5, '176f2a2d-ca9b-41f2-8df3-d93096ebdb85': 27, '76081985-fac4-4ce8-8e1d-43c5cd268ec5': 108, 'ae56c24b-46d3-4688-965e-94bdc208164f': 22, 'dde3fc99-e074-41cf-a843-14aa9c777889': 10, 'c0e9f307-e44b-4de2-bb46-e8045d0990db': 6, '7f3c6c7a-ef5f-4ee0-87ce-a0300ba767d5': 12, 'aa795a1b-5fbb-46d1-b9f2-8b86e1e48bef': 13, '806cc6e7-ca17-42aa-8647-3d270679cfff': 2, 'f8b1a6b3-a9bd-4f01-96ef-154082643353': 6, '7ba1dcf3-0855-491e-bd62-1625f38972b9': 1, '7cff86d0-0d45-4944-a4ba-4511805793bc': 49, 'ad042a79-d405-4cad-92dc-8c5a70028758': 1, 'cace522d-4bc1-47ca-a148-1b0598f8b3fd': 3, 'd00c5cb7-d928-470d-88ff-2239928a3e98': 1, 'a706cca3-8032-4bcb-89c8-6f2d14fa1137': 12, 'd737f173-fa1c-4f41-a363-6e2d5a8e7256': 3, 'aa84d2f5-8a8a-4ae6-a7a6-c9a47ab8dcd5': 4, '7306ea36-16be-415a-99a0-d1a15117b1ad': 2}
FuhuXia commented 1 year ago

Do we support ftp links?

same as http

jbrown-xentity commented 1 year ago

As far as we can tell, this is no longer a problem. See example here: https://catalog.data.gov/dataset/food-price-outlook; the resource link is http but everything works as expected. We may eventually take on this follow up work in a 2 pronged approach:

  1. Announce to the user in some UX way that the link they are using is http, not https
  2. Create a report by organization that allows the data providers to review and clean up their data at the source, and announce to harvest community.
nickumia-reisys commented 1 year ago

Even though there are still issues of http links existing on catalog, since Chrome is not throwing an issue anymore with downloading data, so we are closing this issue.

nickumia-reisys commented 1 year ago

The list of organizations from above for future reference:

ids='352b4532-793d-4075-a03f-05b778a3c43a 842b8561-cf24-49cf-b901-66c6932a392b a543287f-0731-4645-b100-a29f4f39be97 c549d5ce-2b93-4397-ab76-aa2b31d9983a ae156ef5-7869-4748-9ad2-f103cf02a218 8dc070a3-41fa-4e78-8680-a0b98d678fb5 77ade652-dece-4e18-b71b-b5e9be8c86c2 4d976f1e-ccb5-4dbe-aa98-9d2931fbd215 29846409-fa62-4648-a3b9-12d071a2a5b2 fb3131aa-ef06-4a00-ad84-67d93a71d7e3 f4ca4614-8901-409b-8553-2e994ad10023 16980d1c-5e8f-4188-b962-42446f2d3f63 5f4f1195-e770-4a2a-8f75-195cd98860ce 82b85475-f85d-404a-b95b-89d1a42e9f6b 6a3f935c-da6f-47f6-a386-1c76afd0b1e0 2c2fc21f-21d0-4450-af01-cf8c69b44156 7f8ee588-32a3-4b6c-b435-d6603c91dbcc 1f2ebc13-fc03-4bcd-b2c0-dad0bb510b65 6cd4f2ba-1583-469c-a6f0-95ba585aeb7e 5e4d7221-af33-40f5-bbed-78ae1dd8721e a65bf892-ca10-45e5-88f7-9cbaa4bdbfca 2ea36261-b7ad-4018-92fd-5b1cba9e9fdc 4455a74e-f5bd-4a7e-941e-9fe5a72eb882 8148853c-d386-4705-a1c4-41c3212f78e4 1149ee63-2fff-494e-82e5-9aace9d3b3bf 9cd95192-4220-45cd-8d96-f720d58643bd 143529f7-2eef-4a07-b227-93ac9e84fad8 0af4eecd-141b-41b3-9367-a5a51ce44b90 eca91a93-a8a2-48b9-a949-5003b40b9012 7c50b56f-e8ac-4bab-bb77-e88ef2637387 92803bce-3fa9-4dfb-8b70-3baca3835f0c 2d7e3c15-e0cb-4857-b4d5-40ed6e6d7954 bccbad82-abc4-4712-bd29-3e194e7a8042 ed01ec43-dcbc-4772-8e20-4c90b457a7a7 4af4b741-65f5-4ac6-950d-75e1abc1a7df 97ae80aa-2507-4920-808f-99a45dc5ef27 b98a6f3b-552a-4826-931e-4594420f4596 bb10fdb9-d96f-4284-9a5f-9ed269d9dbcb 1a128ac1-72ce-4027-b46d-a73862031b04 a2ea361e-1350-4df1-8d61-4c326b3394ae e9d6fde3-a97c-46c5-ad7d-51579264b7f1 fd2d1d4a-6fc9-4010-a5be-154b4419fd1e 8571f574-7d3d-48f3-8cc7-af44048076a1 57d66334-c55a-4d71-8c4e-8dd1055f5354 217e855b-cd64-4ebc-958b-abbbb0f57ac2 441a7317-0631-4d27-b8bb-dcfaa6be5915 2cca5b33-6bfa-496a-895f-3301690bbc2c 176f2a2d-ca9b-41f2-8df3-d93096ebdb85 76081985-fac4-4ce8-8e1d-43c5cd268ec5 ae56c24b-46d3-4688-965e-94bdc208164f dde3fc99-e074-41cf-a843-14aa9c777889 c0e9f307-e44b-4de2-bb46-e8045d0990db 7f3c6c7a-ef5f-4ee0-87ce-a0300ba767d5 aa795a1b-5fbb-46d1-b9f2-8b86e1e48bef 806cc6e7-ca17-42aa-8647-3d270679cfff f8b1a6b3-a9bd-4f01-96ef-154082643353 7ba1dcf3-0855-491e-bd62-1625f38972b9 7cff86d0-0d45-4944-a4ba-4511805793bc ad042a79-d405-4cad-92dc-8c5a70028758 cace522d-4bc1-47ca-a148-1b0598f8b3fd d00c5cb7-d928-470d-88ff-2239928a3e98 a706cca3-8032-4bcb-89c8-6f2d14fa1137 d737f173-fa1c-4f41-a363-6e2d5a8e7256 aa84d2f5-8a8a-4ae6-a7a6-c9a47ab8dcd5 7306ea36-16be-415a-99a0-d1a15117b1ad'
for id in $ids; do name=$(curl -sL https://catalog.data.gov/api/action/organization_show -d "{\"id\": \"$id\"}" | jq .result.title); echo "$id, $name"; done
352b4532-793d-4075-a03f-05b778a3c43a, "Department of Agriculture"
842b8561-cf24-49cf-b901-66c6932a392b, "Small Business Administration"
a543287f-0731-4645-b100-a29f4f39be97, "Department of the Treasury"
c549d5ce-2b93-4397-ab76-aa2b31d9983a, "Department of Transportation"
ae156ef5-7869-4748-9ad2-f103cf02a218, "Federal Emergency Management Agency, Department of Homeland Security"
8dc070a3-41fa-4e78-8680-a0b98d678fb5, "Federal Housing Finance Agency"
77ade652-dece-4e18-b71b-b5e9be8c86c2, "Federal Geographic Data Committee"
4d976f1e-ccb5-4dbe-aa98-9d2931fbd215, "Department of Defense"
29846409-fa62-4648-a3b9-12d071a2a5b2, "Pennsylvania Geospatial Data Clearinghouse"
fb3131aa-ef06-4a00-ad84-67d93a71d7e3, "US Census Bureau, Department of Commerce"
f4ca4614-8901-409b-8553-2e994ad10023, "National Aeronautics and Space Administration"
16980d1c-5e8f-4188-b962-42446f2d3f63, "Department of Commerce"
5f4f1195-e770-4a2a-8f75-195cd98860ce, "National Oceanic and Atmospheric Administration, Department of Commerce"
82b85475-f85d-404a-b95b-89d1a42e9f6b, "U.S. Environmental Protection Agency"
6a3f935c-da6f-47f6-a386-1c76afd0b1e0, "Earth Data Analysis Center, University of New Mexico"
2c2fc21f-21d0-4450-af01-cf8c69b44156, "U.S. Department of Health & Human Services"
7f8ee588-32a3-4b6c-b435-d6603c91dbcc, "Department of Housing and Urban Development"
1f2ebc13-fc03-4bcd-b2c0-dad0bb510b65, "Department of Energy"
6cd4f2ba-1583-469c-a6f0-95ba585aeb7e, "Department of Labor"
5e4d7221-af33-40f5-bbed-78ae1dd8721e, "Federal Deposit Insurance Corporation"
a65bf892-ca10-45e5-88f7-9cbaa4bdbfca, "Army Corps of Engineers, Department of the Army, Department of Defense"
2ea36261-b7ad-4018-92fd-5b1cba9e9fdc, "U.S. Geological Survey, Department of the Interior"
4455a74e-f5bd-4a7e-941e-9fe5a72eb882, "Department of Homeland Security"
8148853c-d386-4705-a1c4-41c3212f78e4, "National Archives and Records Administration"
1149ee63-2fff-494e-82e5-9aace9d3b3bf, "City of New York"
9cd95192-4220-45cd-8d96-f720d58643bd, "US Consumer Product Safety Commission"
143529f7-2eef-4a07-b227-93ac9e84fad8, "Department of the Interior"
0af4eecd-141b-41b3-9367-a5a51ce44b90, "State of Oregon"
eca91a93-a8a2-48b9-a949-5003b40b9012, "Office of Personnel Management"
7c50b56f-e8ac-4bab-bb77-e88ef2637387, "World Wide Human Geography Data Working Group"
92803bce-3fa9-4dfb-8b70-3baca3835f0c, "State of Minnesota"
2d7e3c15-e0cb-4857-b4d5-40ed6e6d7954, "General Services Administration"
bccbad82-abc4-4712-bd29-3e194e7a8042, "State of Washington"
ed01ec43-dcbc-4772-8e20-4c90b457a7a7, "State of Alaska"
4af4b741-65f5-4ac6-950d-75e1abc1a7df, "U.S. Department of Labor Bureau of Labor Statistics"
97ae80aa-2507-4920-808f-99a45dc5ef27, "Department of Justice"
b98a6f3b-552a-4826-931e-4594420f4596, "State of Maryland"
bb10fdb9-d96f-4284-9a5f-9ed269d9dbcb, "Federal Communications Commission"
1a128ac1-72ce-4027-b46d-a73862031b04, "Department of Veterans Affairs"
a2ea361e-1350-4df1-8d61-4c326b3394ae, "OpenTopography"
e9d6fde3-a97c-46c5-ad7d-51579264b7f1, "WV GIS Technical Center"
fd2d1d4a-6fc9-4010-a5be-154b4419fd1e, "Millennium Challenge Corporation"
8571f574-7d3d-48f3-8cc7-af44048076a1, "City of New Orleans"
57d66334-c55a-4d71-8c4e-8dd1055f5354, "US Agency for International Development"
217e855b-cd64-4ebc-958b-abbbb0f57ac2, "Department of Education"
441a7317-0631-4d27-b8bb-dcfaa6be5915, "Department of State"
2cca5b33-6bfa-496a-895f-3301690bbc2c, "City of Providence"
176f2a2d-ca9b-41f2-8df3-d93096ebdb85, "National Institute of Standards and Technology"
76081985-fac4-4ce8-8e1d-43c5cd268ec5, "Institute of Museum and Library Services"
ae56c24b-46d3-4688-965e-94bdc208164f, "State of California"
dde3fc99-e074-41cf-a843-14aa9c777889, "State of New York"
c0e9f307-e44b-4de2-bb46-e8045d0990db, "State of Connecticut"
7f3c6c7a-ef5f-4ee0-87ce-a0300ba767d5, "City of San Francisco"
aa795a1b-5fbb-46d1-b9f2-8b86e1e48bef, "City of Los Angeles"
806cc6e7-ca17-42aa-8647-3d270679cfff, "University of Idaho"
f8b1a6b3-a9bd-4f01-96ef-154082643353, "City of Austin"
7ba1dcf3-0855-491e-bd62-1625f38972b9, "City of Chicago"
7cff86d0-0d45-4944-a4ba-4511805793bc, "Lake County, Illinois"
ad042a79-d405-4cad-92dc-8c5a70028758, "City of Baton Rouge"
cace522d-4bc1-47ca-a148-1b0598f8b3fd, "Florida Department of Environmental Protection"
d00c5cb7-d928-470d-88ff-2239928a3e98, "Centers for Disease Control and Prevention"
a706cca3-8032-4bcb-89c8-6f2d14fa1137, "City of Seattle"
d737f173-fa1c-4f41-a363-6e2d5a8e7256, "King County, Washington"
aa84d2f5-8a8a-4ae6-a7a6-c9a47ab8dcd5, "Georgia Tech Center for GIS College of Architecture"
7306ea36-16be-415a-99a0-d1a15117b1ad, "Vermont Center for Geographic Information"